Meme'd

From here.

* Pick 15 of your favorite movies.
* Go to IMDb and find one quote from each film.
* Post them here for everyone to guess.
* Strike it out when someone guesses correctly, put who guessed it and the correct answer.
* NO GOOGLING/IMDb search or other search functions.

It was hard for me to pick 15 movies, because I really don't think about them too much.  I just picked stuff that I knew would have quotes.

1) So you should be! You spilled an ocean of blood. You showed no mercy, no pity. We too are children of this age... weaned on strife and chaos. We are your sons, yet you count on our fidelity. In my eyes, that makes you a fool. A senile old fool!

2) Don't know, I don't know such stuff. I just do eyes, ju-, ju-, just eyes... just genetic design, just eyes. You Nexus, huh? I design your eyes.

3) Okay. You people sit tight, hold the fort and keep the home fires burning. And if we're not back by dawn... call the president.

4) Don't ever make trouble here. Or I'll beat you up each time. Careful, mind the step.

5) There are warrants out on you for treason, illegal entry, decadence, pornography... and for being a lazy pig.

6) You seem a decent fellow... I hate to kill you.

7) Has the whole world gone crazy? Am I the only one around here who gives a shit about the rules? Mark it zero!

8) Bugger this for a bunch of bananas.

9) Good. Bad. I'm the guy with the gun.

10) I ain't like that no more. I ain't the same, Ned. Claudia, she straightened me up, cleared me of drinkin' whiskey and all.

11) The way your dad looked at it, this watch was your birthright.

12) You still don't understand what you're dealing with, do you? Perfect organism. Its structural perfection is matched only by its hostility.

13) I disagree, sir. I've carried the Baretta for ten years, and I've never missed with it.

14) Oh, the reason I called... Could you find out who else is in town? I've made two spooks and a ghoul already, so if they've double-booked the job, and/or they're going to kill me, I'd like to know. If you could find that out, that'd be great.

15) Find one in every car. You'll see.

Virtual Server management- open source style

I really want to try the VMWare player for openqrm. It has all kinds of crazy features:

Support for different virtualization technologies

VMware, Xen, KVM and Linux-VServer vms can be managed transparently via openQRM. openQRM seamlessly support P2V (physical to virtual), V2P (virtual to physical) AND V2V (virtual to virtual) migration. This mean server appliances can not only move from physical to virtual (and back) easily but also that they can be migrated from virtualization technology A to virtualization technology B without any hassle.

Fully automatic Nagios configuration (single click) to monitor all systems and services

High-availability : "N to 1" fail-over

F5 sync troubleshooting

Found a secret command for troubleshooting sync on F5 BigIP: csTest.pl

csTest.pl 

It only has a few switches. -q (or --q) for quiet just outputs the bottom line, or -v or --v for verbose (seems similar to the default for me, but may include something else).

Anyway it will tell you what is wrong with sync as configured.  It will report sync errors, time differences, username/password problems, error in ip configuration of sync, etc. etc.

No break in MacOS X minicom? No it is the prolific driver

Just when I thought I was going nuts. Couldn't password recover a Cisco switch, then a router. I used to do it on OS X with a Powerbook G4. Now on the intel 10.5 I can't.

Well it turns out it is the prolific driver:

http://blog.curthread.org/posts/macosx/usb_serial_break.html

Fix is rumored to be this driver. Maybe I'll try it, later. Meanwhile I've got linux boxes that will control break.

Later: 2 seconds in a linux VM and ctrl break worked fine.

The role of flash in storage

EMC has some SSD/Flash for a segment of their storage systems. The MacBook Air famously has an SSD option. Solid State Disk and Flash are everywhere. So why can't I buy a shelf of it for my server room?

Well, SSD isn't quite ready to go in everything yet. As this blog points out: 1) it is in devices as cache/flash/nvram 2) if the storage isn't engineered properly, there are performance walls to run into 3) availability of dependable SSD is a problem (maybe not for iPods, hahah).

There is one thing I want to try out in the Garden:

http://www.fusionio.com/

Works in linux- Check, ridiculous performance profile for certain activities- Check. I just imagine these with linux RAID, LVM and a database that has a random read profile. I thought about making one as an appliance to shop around. Financing?

NAT in my IPv6? It is more likely than you think...

NAT in my IPv6?

For the good: potentially make it easier to speed up IPv6 adoption.

For the bad: probably break end to end connectivity for some situations.

I hope not. I thought 6 to 4 tunnel was good enough. Guess not.

VMWare appliances to Parallels

Ever found a VMWare appliance or VM that you wanted to run under parallels?

well Virtualization Daily has an answer:

Convert the image from VMWare format to a raw hard disk image using Qemu.

qemu-img convert appliance-harddrive-name.vmdk -O raw appliance-harddrive-name-raw.hdd

There are some caveats: not all VMDKs can be converted, IDE disk only, single disk images (can't have multiple VMDK files). I've had pretty good fortune with Linux appliances so far (because it had the kernel drivers to load up even though the hardware had changed slightly).

They did some sample parallels appliances:

RHCE Self Study

I self studied for the RHCE exam for 5.1 and passed. It was difficult, and I have a solid background in Linux administration.

I used the Michael Jang RHCE self study book, which covers the exam topics. It could really use some sample exercises (like some sample test scenarios). Make sure you get the most recent version. If you can afford it, I think it is generally better to take the 1 week course RH300, after seeing some co-workers and friends take that route.

I would concentrate on reviewing the book then finding weak areas and getting the knowledge of the topics. You need to know everything on the exam blueprint. Then the important thing is working on speed. The main things to remember about the RHCE exam is it is performance based and has a time limit. So you must be able to do the relevant tasks in a quick time frame. Speed is of the essence. If you know one topic really well and can do things quickly, but another at a mediocre level and are slow and constantly referring to apropos or man pages, that may be enough to prevent finishing in a timely manner. Some referral to resources is expected, but you need to know your business and be quick about it.

For me, I was weak at SELinux and ACLs, as I use them very sparingly. So I spent extra time on those above and beyond the self study book and practiced tasks with a hands on manner. Topics that I was strong in, I spent the time reading the book and doing the material, but then I moved on.

The RHCE is a great certification and very satisfying to attain. Redhat's lab performance based exam is a great way to demonstrate linux knowledge. I learned some new things and some new ways to do things as I was studying for the exam.

WRT54G Benchmarks

Alright benchmarked the WRT54G v.3 that I have with dd-wrt v.24. 216Mhz default, no overclock. Basically the thing does everything in software except layer 2 switching and runs out of CPU. I needed to know where the cut off is, because broadband keeps getting faster. You can depend on the box for about 20Mb/s throughput. If you approach that limit, you may run into CPU problems. I also found out the default window size on iperf on Windows client is 8K (too small for 100Mb networks). So all my tests use 256k window sizes (default on linux and os x). All tests were performed with iperf.

Full test notes for posterity:
test 1-LAN bandwidth- both ports plugged into lan ports, same vlan: iperf -s on vista server, iperf -c on OS X client 93.7 Mb/s no CPU on WRT54g.
test 2- LAN bandwidth- both ports plugged into lan ports, same vlan: iperf -s on OS X iperf -c on vista, 61Mb/s (too small a windows size 8K on windows), no CPU.
test 3- LAN bandwidth same as 2, but with real window sizes: iperf -s -w 256k on OS X, iperf -c 192.168.100.8 -w 256k- 93.7Mb/s second switched no CPU.
test 4- routing (no SPI/Firewall) from LAN to WAN same iperf settings as above except client is on the other network - 22.7Mb/s CPU pegged.
test 5- reverse client and server traffic now WAN to LAN- 22.7Mb/s CPU pegged.
test6 - enable SPI/firewall, repeat test 5- 18.3Mb/s. CPU pegged.
test7- reverse client and server now LAN to WAN- 18.3Mb/s CPU pegged.
test8- LAN to wireless- 17.3Mb/s no encryption G only, some 2.4Ghz interference. CPU not pegged, but high (fluctuating between 50-65%).
test9- wireless to LAN- same as above- 17.3Mb/s CPU, not pegged but high (fluctuating between 50-65%).

I should try with WPA, but there is so much interference here. I can't be sure what I'm testing, the wireless, the encryption, the interference.

Summary of results: the switch in the Linksys I have is in hardware- it gives 93.7Mb/s throughput. So L2 performance is good.

Just routing from Lan to WAN with no firewall you can get 22.7Mb/s and it is CPU limited. With a firewall enabled from LAN to WAN, you are limited to 18.3Mb/s and it is CPU limited. Linux iptables style firewall (what dd-wrt uses) is a pretty efficient packet filter (about 10% overhead it looks like).

On the wireless from wireless I got 17.3Mb/s throughput (no encryption), and there was still CPU left. That implies the wireless G protocol or interference will probably be the limit not WRT54G. Basically if you are doing more than 17Mb/s, the WRT54G could be a choke point as could wireless 802.11g. If you want more performance overclock or get a higher CPU dd-wrt capable box and don't use 802.11g wireless. We'll call it 17Mb/s is the upper end for an internet connection for these boxes (due to CPU and 802.11g being limiting factors), more features may make the ceiling a little lower.

GNS3 http://www.gns3.net

So there is a Cisco router simulator called dynamips. I was using it for a while on linux. It is kind of hard to setup and a pain in some ways. Now there is a graphical front end to it:

http://www.gns3.net Graphical Cisco Router Topology emulator

It makes it super easy to lab up simple topologies virtually (no clunky router hardware to dig around with you), and to test configurations and syntax. All you need in gns3 and at least one IOS binary. I used it on OS X, and it worked like a dream. Just get the DMG, click the image, copy GNS to the applications directory, then in the config stage I pointed gns at some images I had on my laptop (my tftp library I keep to upload to the lab). It fired up.

Then you can drag and drop some topologies. Console into the routers, etc.

This site had some nice tutorials for gns3 that the gns3 site doesn't cover.

Another soft skills post

Okay so here are some soft skills tips for IT and general professionalism:

1) Read How to Win Friends and Influence People (there is a nice audio book out there too). A friend of mine called it the Necronomicon he found it so powerful. It really is a good basis if you don't get interacting with people or the beauty of a win-win situation. Non-zero sum games are fun to play.

2) Get a safari account. Even the small one: safari.oreilly.com or safari.ciscopress.com. It allows you to burn through IT reference books that would not be worth having on your shelf after you extract the needed details from them. It is totally worth every penny for an IT professional.

3) If you haven't go talk to some one in another group at your company. Maybe become friends with a couple. It will help you. Sometimes you get advance warning of new projects, sometimes you get a political ally if something unpopular needs to be fought, sometimes you get someone fun to hang out with that has a new point of view. It is all beneficial.

RCSP (Riverbed Certified Solutions Professional)

The RCSP exam is an interesting exam to self study for. There is basically the official Riverbed class, and really if you have the option, I recommend that. But if you are going to go your own...

First priority, read the RCSP certification website on preparing (and the rest on their cert in general). I found it useful to even read the class descriptions.

Second priority gather all the free materials you can.
You'll need to get the Certification Guide, you will also need the Management Console User's Guide, the Command-Line Interface Guide, the Steelhead Appliance Deployment Guide and the Steelhead Appliance Installation and Configuration Guide. Those are available from Riverbed's site, but you probably need registration.

Now you can either do the recommended outside reading or skip only to the Riverbed stuff. I think it is much easier to pass the test (but longer) if you do the outside reading.

The outside reading consists of some documents linked in the certification guide. The three of the linked documents I learned the most from are Steven's TCP/IP and Internet Routing Architectures and RFC 2474. I had already read Stevens and Halabi and if you haven't and you are a network jockey, you should. There really is only one network protocol that you can find everywhere, and Stevens covers it like sheet. Halabi follows up with Internet Routing Architectures (which if you've had to do BGP, you should have read anyway). So I was new to RFC 2474 even though I'd done a few QoS implementations, and the RFC was pretty dry, it was good to make it through the material. You can get access to Stevens and Halabi with a free trial of Safari (safari.ciscopress.com or safari.oreilly.com) or you can keep a Safari account like I do, the entry level is pretty inexpensive.

I also found a Cisco section on QoS that was useful and not part of the material recommended by the study guide:
"Implementing Quality of Service Policies with DSCP" when I was studying up on QoS that was informative.

After you mow through the outside material at least once, start learning the Riverbed technologies. If you are skipping the outside reading, I hope you have sound networking fundamentals and have some experience. Anything in the study guide is fair game so read all the study guide and all the Riverbed docs. I had to read the CLI guide a few times, because frankly I don't use the Steelhead CLI as much as the GUI or the CMC. The exam is very particular and don't worry if you have to take the exam more than once. The RCSP exam is close to the Cisco exams in the exact nature of the questions, but has a different feel (no simulations). I think you could implement Steelheads and not pass this exam, so pay attention to all the study material. Do not be over confident.

How to tell if you have a MySQL performance problem

The top three things for mysql performance:

1) run 64bit OS on 64bit hardware with 64bit version of mysql (newer versions of mysql do better with large amounts of RAM if you aren't using 4.1+ you may not be able to use enough RAM).
2) make sure your indexes fit in RAM
3) Make sure you log slow queries and check the logs for problems. This is your number

Number 1 is easy.

Number 2 is a little harder. You can eyeball this if you are running ISAM databases by summing up all your IDX files and then seeing if MySQL is using more RAM than that.

But a really good way is to run a report: http://hackmysql.com/mysqlreport It has really good documentation and can make suggestions. You can use it to tune other options.

Number 3 requires a change in logging. You can also change the threshold- what is slow for you may not be for someone else. The problem with logs is you have to look at them, understand them and then act on them. Get going.

Virtualization for linux takes a step forward

www.enomalism.com

Hopefully it works as advertised. Looks like it has some features sorely missing from the opensource, management of multiple Xen Servers.

The interesting points:

# Create a cloud of abstracted, highly scalable, and managed compute infrastructure capable of hosting end-customer applications and billed by consumption.
# Simple easy to use web based user interface
# Automagically load balance and and monitor operations using a RESTFul API.
# VM Agnostic, migrate to and from various virtual environments including KVM/Qemu, Amazon EC2 and Xen. (OpenVZ, VirtualBox and VMware coming soon)
# Fully Automated platform with easy setup rules for a completely autonomous and self healing virtual environment

Those alone make it worth while. It depends on mysql for its data storage. I might be trying this out in the lab. When I do, I'll post back.

Since Xen is at least where VMWare Server/ESX is, it is interesting that enomalism might give at least a set of the features available in the ESX/virtual infrastructure product. The main weakness of Xen is managing a pool of Xen servers, and if enomalism does what it advertises, it is a huge step forward.

F5 advanced troubleshooting

Say you have a version 9.x F5 Bigip. Nice loadbalancer, does some cool stuff. You add some new config and suddenly you have what looks like a layer 2 problem. So you console the box, and run top and the CPU (and maybe RAM) is pegged. Something called tmm is pegging the box.


TMM is a daemon that does most of the traffic management on F5. It is written by F5 and is opaque... except for:

tmstat

/usr/bin/tmstat from F5 gives you some output about what tmm is doing. So you can kind of figure out where things are going wrong, while you are dialing support.

Output looks vaguely top like:
NAME
CPU: 0% busy 1% idle 99% sleep Thu Apr 17 13:59:00 2008

Memory Allocated New Flow Old Flow Poll
21,081,060 / 1,807,745,024 99,851 25,284 1,228,703 Cycles
[ . : . | . : . ] 1 20 2,035 Total
vnic
Tc4,240b rx Cryplinkps 21,664b txass 27 Timers
[ . : . | . : . ] i8254x:00 (t[ . : . | . : . ]otal) 0 Stats
23,832b rx 1,000 linksa 4,112b txeudo
[ . : . | . : . ]ects 0 fu[ . : . | . : . ]tropy Virtual Class
Wa0b rx 0 linkecord 0b txcure 10,867,070 (total)
[ . : . | . : . ] 0 ci[ . : . | . : . ] 10,485,780 mco db
0 De0b rx 0 linkunseen) 0b tx 168,855 ssl
[ . : . | . : . ] [ . : . | . : . ] 143,727 tcl
0b rx 0 link 0b tx 68,708 (unseen)
[ . : . | . : . ] [ . : . | . : . ]
0b rx 0 link 0b tx Umem Class
[ . : . | . : . ] [ . : . | . : . ] 854 (total)
0b rx 0 link 0b tx 513 xfrag
[ . : . | . : . ] [ . : . | . : . ] 127 connflow
0b rx 0 link 0b tx 117 listener
[ . : . | . : . ] [ . : . | . : . ] 19 poolmbr
0b rx 0 link 0b tx 78 (unseen)

Some good wireless networking training- free

Aruba makes a nice lightweight access point (LWAP). Centrally managed from a central point. Part of their magic is a GRE tunnel from AP to controller, so the controller can do all kinds of nifty tricks with/to the wireless client packets.

Here is some network training that is has some free options:

Free Wireless Training

Some is Aruba specific (hello mounting Aruba hardware). But the networking fundamentals course is vendor agnostic and has some good information.

Ever wondered why 802.11b clients slow down 802.11g networks, then do the networking fundamentals:

http://www.arubanetworks.com/education/networking_fundamentals.php

It does a review of basic networking, covers wireless and security basics. Self paced in wmv and mov formats.

The other training courses are more Aruba-centric.

Xen Virtualization on CentOS

You can use many tools on Centos 5 to manage virtual servers:

CLI tools: xm, install-sh, virsh
GUI tools: virt-manager

Here are some useful links-

How to install a virtual server (after installing CentOS with the Virtualization packages):
http://wiki.centos.org/HowTos/Xen/InstallingCentOSDomU

That is a command line tutorial. But you could use virt-manager if you have X and Xen. It walks you though all the steps in a much easier way, but it is a good idea to know what is going on behind the scenes. Everything in this tutorial can be done by virt-manager without much hassle.

This is how you install a native virtualization DomU like Windows:

http://wiki.centos.org/HowTos/Xen/InstallingHVMDomU

It also has the way to check your processor extensions.

This one has some basics about keyboard mapping and also has how to deal with SELinux (without turning it off people!):

http://wiki.centos.org/TipsAndTricks/Xen

This article has P2V basics. It gives some insight into what the virt-p2v boot iso (linked in the intro post) does behind the scenes:

http://wiki.centos.org/HowTos/Xen/MoveNative2DomU

And if you had to move something yourself if you had to do it manually.

Free Virtualization Introduction

Xen on Linux is usable out of the box now. If you download CentOS 5.1 (or later I would guess), you can setup virtualization during install.

Virtualization lets you run multiple instances of an operating system and applications on one piece of hardware. You can imagine or read on the net why that would be useful.

Some terms to translate for those familiar with VMWare or Parallels style terminology:

Host OS: VMWare Server (not ESX) uses an operating system (windows or linux) as a host, ESX is the host, and Parallels uses OS X. With Xen virtualization this roughly corresponds to Domain0. Domain0 has unlimited privileges.

Guest OS: You can run windows, linux or other operating systems under VMWare or Parallels. On Xen the guest OS is known as a DomU (domain U), which has direct access to hardware. This domain has limited resources (ram and processors as configured), by default it has no access to hardware.

Paravirtualization vs. fully virtualized-

paravirtualization uses software to allow virtualization. Under CentOS 5, the paravirtualizable guest operating systems are basically linux, linux and linux. Native virtualization uses processor extensions so you need a recent Intel processor that supports or a AMD that supports AMD-V extensions. The extensions aren't very common outside of server class chips (Xeon and Opteron). Here is how to tell if you have the extensions:

egrep '^flags.*(vmx|svm)' /proc/cpuinfo

If you don't have a lines of output with vmx or svm in them, you don't have the extensions.

P2V: Physical to Virtual. How to import physical servers (usually old), to a virtual machine, which can save you time and money and get you out of supporting old and busted servers. There are some pay tools for P2V for Xen and Linux. But here is a pretty good free one:

http://et.redhat.com/~rjones/virt-p2v/

That is it for the intro. More tomorrow with specific examples.

system-config-netboot

Redhat, time for your verbal beating. RHEL3 and 4 I love you. RHEL 5 you are obviously rushed out the door. Proper XEN support starts in 5.1. But the real symptom of the this rush is:

PXE boot.

That is right. The Doc guide used to have directions on PXE boot/network install for RHEL5. It mentioned system-config-netboot (which wasn't in RHEL5). So instead of fixing system-config-netboot somebody went and trashed the doc excising references to system-config-netboot, but not the whole section and not the references to the pxeos tool which is part of the package.

Well if you are lazy like me, and like the tool to setup the /tftpboot directory and not to have to remember anything but the /etc/dhcpd.conf commands:

You can use the system-config-netboot from Fedora 8 (which was kind of a pain to find- thanks to all the mirrors that haven't mirrored FC8 as of this article) right on RHEL 5.1 (for sure, tested, it works).

My favorite property of RHEL is the kickstart/PXE boot ease of deployment. An afternoon or two of fun and you can deploy 1000s or 10,000s of boxes with no trouble. Unless of course you can't get system-config-netboot to setup your PXE environment for you... because it isn't on the distribution.

There is a ticket on bugzilla.redhat.com that says it will be in RHEL5.2. Which will be nice.