getting rid of standard error

scp foo server: 2>/dev/null


the descriptor 2 is for standard error.

Redirecting 2 to /dev/null makes the standard error go away.

LVM and a rant on bonding and 802.1q

use LVM if you are using a modern linux.

Really. It will make your life easier.

Bonding and 802.1q configuration under linux suck right now. If I get time to experiment, I will figure out the model config. But really Redhat or Suse needs to come out with a configuration tool so you can bond interfaces (and use static or dhcp addresses) and use 802.1q vlan tagging on those interfaces (or non-bonded interfaces).

Cisco 2500 router IOS upgrade

Bucket of pain. The 2500 series routers can have 16Mb of RAM and 16Mb of flash. It stores the OS, called IOS in the flash. The config goes in NVRAM and the boot stuff goes in the boot rom.

I have two routers with two banks of 8Mb flash and it was a mother to upgrade one of them.

The first router upgraded fine with the classic copy tftp: flash: syntax. It erased the old IOS and put the new one on over and away it went (only had to good with the conf reg once 0x2142 to get rid of a config with a password I forgot).

The second one was pain. The two Flash banks showed up seperate, the copy tftp: flash: spat back READ ONLY FILE SYSTEM... so on and so forth.

Here was the fix:

conf 0x2101 (this boots a rom or cut down IOS).
partition 1 16 (make one big 16Mb partition instead of two 8s)
copy tftp flash
conf 0x2102

The 2500 is great for a lab, but don't use one in production. The new ISR routers are quite nice.

Use DNS- it's good enough for the internet

I just fixed a couple boxes that didn't know what localhost was... actually they did, but it was wrong (pointing to their actual IP address, not 127.0.0.1).

Look don't mess with host files. You don't need to. Use DNS.

If you have more than one host, use DNS with Dynamic DHCP. You can reserve IP addresses so that hosts always get the same IP, you can extend lease times, you can put all kinds of things in DNS. But if you update dynamically you will always have the forward and reverse DNS correct (A record and PTR) and you won't have stupid host file troubles like I just had.

Cisco 3005 VPN concentrator resurrection

I found an unused 3005 going through one site's material. Since the current VPN terminates on PCs, I thought I'd get the 3005 going.

While I had the 3005 on the shelf in the lab, I found a problem. The 3005 has a public and private interface. The private interface would intermittently drop physical connection. I inspected the network jack, it looked good, no bent pins. But everytime I'd wiggle the network cable (or even move the middle of the cable), the connection would drop.

So I tore the out of warranty and service 3005 apart (don't do this, it will void your warranty). I checked the posts and solder on the network jack. It looked good, so I put the 3005 back together. While I had it apart I noticed two little silver tabs on the sides inside of the jack where the pins are. I used a very fine screwdriver and bent these two tabs out on both jacks hoping it would tighten the grip on the network cable.

Sure enough it works. I'll try to get a macro picture up soon.

Silly Juniper...

Just got the Juniper ScreenOS Product Documentation CD Version 5.0 June 2004 Rev. B in some brand new NetScreen 50 boxes.

Either the doc CDs aren't revisioned very often or these NS50's move kinda slow... any way,

The disc is CDFS or whatever, but all the directories are 444 permissions on linux and MacOS. You can't change directories to read the PDFs when anything but root. On a linux box, at least you can be root, on a Mac it is really inconveniant. In either case, why would I want to be root when I'm reading PDFs?

Looks like nobody at Juniper uses the doc CD on different architectures. Maybe they only use windows internally or have all the docs on the webserver internally.

mount loop: cd or dvd iso

if you need to use a dvd or cd iso on your linux box just mount it loop. You can even export or serve out (via http) the mount.

mount -o loop /home/fedora/FC3-i386-DVD.iso /home/fedora/pub/mirrors

and that makes gardening a little easier...

What do you do to emulate a server?

So I get one of those typical gardening compalints, "I can't communicate to my server that is running on port 3000...what is wrong?"

Well, first I "netstat -an | grep 3000", looking for a deamon listening on 3000. Nothing. No server, no communication. But I don't know the server software well enough to start it up (custom, not init.d script). So I whip out netcat (could be nc on some platforms) and start it listening as a deamon:

netcat -l -p 3000

netcat listen on port 3000. Any thing netcat recieves will be output to std out. So now from a host that should be able to connect to the server on 3000, "telnet servername 3000". Connected and things type appear on the terminal. Looks good.

Looks the the problem is the server software. Back to your application, developer.

The benchmarks...

I promised some comparisions of SLES9 and why it has slow I/O. I haven't cleared releasing the application yet (it is a small peice of C code that opens as many files as you throw as an argument and then writes to those files). In place of that code, a nice workabel substitute is substitue a one large file write: "time dd if=/dev/zero of=/tmp/testfile bs=16k count=65536 ". You can also try reads, but that is more divergent based on caching the filesystem, if you bench reads, reboot between benchmarks (or otherwise flush all cache and buffers).

I've benchmarked this on HP servers (and a couple desktops). I've tried different filesystems, different kernel versions and different I/O subsystems (SCSI, SATA, ATA). The numbers pretty much go about the same way (except kernel version as you will see). Apparently Redhat backported the patch or knows about the bug and fixes it in their kernel.

Here are some benchmarks all on the same HP DL140 hardware (2.8 Xeon, 1G ram, sata drive, 11211 Bogomips):

Hardware	OS	FS Type	time a.out 1000	user	sys	Notes
dl140	SLES 9	reiser	5m15.042s	0m38.427s	0m8.303s	unresponsive after a few seconds and well after test ls will hang
dl140	SLES 9	ext3	5m31.042s	0m38.427s	0m8.303s	unresponsive after a few seconds and well after test ls will hang
dl140	SLES 9	reiser	3m53.546s	0m44.687s	0m3.052s	2.6.9 kernel unresponsive
dl140	SLES 9	reiser	2m51.070s	0m44.687s	0m3.052s	2.6.16.1 vanilla kernel responsive.
dl140	FC 5	ext3	1m52.354s	0m44.515s	0m7.124s	responsive.
dl140	FC 5	ext3	1m52.354s	0m44.515s	0m7.124s	run 5 instances still responsive

Centos/RHEL4 perform similarly to FC 5. You can see reiser is slightly faster than ext3 on the SUSE test, but it doesn't matter as they are blown away by a good kernel. The interesting thing is SuSE/Novell didn't really want to hear about this when I tried to open a ticket. I'll be trying again. I have benchmarks from DL380's and a reproduceable method, that doesn't rely on the C program, just dd (you can also produce the bug with sort and some other ways).

The nice thing here is we can double our performance by going to a new distribution.

The dismal thing is a 500$ desktop (1.7 P4 Celeron 512 Mb of ram, ata drive) with FC 5 was able to perform on par with a DL385 (~10,000$) dual Opteron, 8GB of ram, 6 drive SCSI raid array, on the first run. And able to beat the Opteron with multiple runs. That means this silicon garden is poorly optimized and utilized.

Two ways to sync files on servers.

Two ways to synchronize files:

rsync over ssh (superhandy with keys and a key agent):

rsync -av source destination

Where source and destination are ssh style: username@host:/path/to/

Another nice way to sync files is rdist.

rdist allows you to sync files to many nodes from a master (it is easy to set up and configure).
It is very handy. It is probably included with your distribution, it is definately used by several webservice providers and the homepage is here: http://www.magnicomp.com/rdist/

SLES9 kernel compile

I'm still getting to the bottom of why I/O on SLES 9 is so pathetic (future article with benchmarks on various HP server hardware). But I'm testing many kernel compiles and there wasn't a good recipe for SLES9 and vanilla 2.6 kernels. I've tested this with 2.6.16.1, 2.6.9, 2.6.12.6.

Sles 9 kernel build:

make sure you have the kernel source on the machine and gcc.

yast2 -i kernel-source gcc

then get the linux kernel and untar it (replace the kernel.org/foo.kernel stuff with a real path to a kernel):

wget http://kernel.org/foo.kernel.bz
tar -jxvf linux-2.6.6.1.tar.bz2

then move the kernel to /usr/src/

mv kernel-foo /usr/src/

remove old kernel build sim link:
rm /usr/src/linux

make new symlink to the new kernel:
ln -s /usr/src/kernel-foo /usr/src/linux

If you want to build a kernel with similar options to a SLES Kernel (building one with the old config), then you need to get and old .config file.
If you are running a plain SLES kernel:

zcat /proc/config.gz > /usr/src/linux/.config

or you can copy the .config file from a SLES kernel source:
cp /usr/src/linux-2.6.5-7.244/.config /usr/src/linux/.config

Now you have one more interactive part:
make oldconfig

That will ask questions about new options in the kernel that are not covered by the old .config only. You may be able to take all defaults.

I will break down the next steps with commentary, but the rest does not necessarily need intervention, so I usually stack them on a command line (see below):

Clean up configs:
make clean

Make a bootable linux image:
make bzImage

Compile loadable kernel modules (these are the same as drivers usually):
make modules

Install said drivers:
make modules_install

This makes a module dependency map and if it isn't done right you might not boot. The 2.6.16.1 will need to be replaced with the kernel version.
depmod -ae -F System.map 2.6.16.1

Install everything:
make install

So a line to do all of the post interactive stuff with a 2.6.12.6 kernel would look like:
make oldconfig && make clean && make bzImage && make modules && make modules_install && depmod -ae -F System.map 2.6.12.6 && make install

And then you might have to wait for a while. If you use grub as a bootloader, you shouldn't have to do anything else to run your new kernel except reboot. This whole process is so much easier than 2.0, 2.2 or 2.4 kernels. It still isn't as easy as apt-get upgrade kernel or yum update kernel or emerge or whatever, but such is the price you pay for SuSE.

Sending attachments from a script or Linux CLI

mutt -s SUBJECT -a ATTACHMENT user@domain.com

is a nice way. I like mutt as a mail user agent. It can do pgp and stuff too if you need to.

mailx is another way. I prefer mutt, but now you have at least two ways to send attachments from a linux script or command line.


UPDATE:
A third way posted by anonymous in the comments:

uuencode local_file.name remote_file.name | mail -s "file attachment" user@example.com

if you have uuencode installed and mail, it works very well. Thanks anonymous!

SLES 9 IO problem

There are some aphids in the garden right now.

I have an interesting problem with SLES9. I know it isn't my favorite linux distribution either, but it is pretty good for business that like support and it is a great java platform.

The interesting problem is with I/O. On a default install on various platforms, if you run a high I/O job the system becomes unresponsive. I believe it is a problem with the kernel version, because none of the other Linux distributions I've tried have it.

SLES9 is stuck in a heavily patched 2.6.5 kernel, and if you look through the kernel changelog, you'll see plenty of virtual memory and I/O improvements.

Today, I think I'll try a vanilla 2.6.12 or later kernel- because if my suspicions are right, somewhere between 2.6.5 and 2.6.12 the bug is fixed.

Two laws that stifle the growth of technology in the US

These are the two worst laws for a gardener of silicon rust:

the DMCA ( http://en.wikipedia.org/wiki/DMCA )

and

the Patriot Act ( http://en.wikipedia.org/wiki/USA_PATRIOT_Act )

These laws do not encourage the things that brought the great innovations that the US is famous for like the Internet, Unix, ethernet, etc. They would inhibit a Bell Labs or a Xerox PARC or even the children of products from Xerox Parc (like Apple and 3Com and the like).


If you are a citizen and want to see the U.S. keep up with the world in technology, write you congress person and contribute to the EFF ( http://www.eff.org/ ). If you can, do more. If you can't anything will help.