What do you do to emulate a server?

So I get one of those typical gardening compalints, "I can't communicate to my server that is running on port 3000...what is wrong?"

Well, first I "netstat -an | grep 3000", looking for a deamon listening on 3000. Nothing. No server, no communication. But I don't know the server software well enough to start it up (custom, not init.d script). So I whip out netcat (could be nc on some platforms) and start it listening as a deamon:

netcat -l -p 3000

netcat listen on port 3000. Any thing netcat recieves will be output to std out. So now from a host that should be able to connect to the server on 3000, "telnet servername 3000". Connected and things type appear on the terminal. Looks good.

Looks the the problem is the server software. Back to your application, developer.

The benchmarks...

I promised some comparisions of SLES9 and why it has slow I/O. I haven't cleared releasing the application yet (it is a small peice of C code that opens as many files as you throw as an argument and then writes to those files). In place of that code, a nice workabel substitute is substitue a one large file write: "time dd if=/dev/zero of=/tmp/testfile bs=16k count=65536 ". You can also try reads, but that is more divergent based on caching the filesystem, if you bench reads, reboot between benchmarks (or otherwise flush all cache and buffers).

I've benchmarked this on HP servers (and a couple desktops). I've tried different filesystems, different kernel versions and different I/O subsystems (SCSI, SATA, ATA). The numbers pretty much go about the same way (except kernel version as you will see). Apparently Redhat backported the patch or knows about the bug and fixes it in their kernel.

Here are some benchmarks all on the same HP DL140 hardware (2.8 Xeon, 1G ram, sata drive, 11211 Bogomips):

Hardware	OS	FS Type	time a.out 1000	user	sys	Notes
dl140	SLES 9	reiser	5m15.042s	0m38.427s	0m8.303s	unresponsive after a few seconds and well after test ls will hang
dl140	SLES 9	ext3	5m31.042s	0m38.427s	0m8.303s	unresponsive after a few seconds and well after test ls will hang
dl140	SLES 9	reiser	3m53.546s	0m44.687s	0m3.052s	2.6.9 kernel unresponsive
dl140	SLES 9	reiser	2m51.070s	0m44.687s	0m3.052s	2.6.16.1 vanilla kernel responsive.
dl140	FC 5	ext3	1m52.354s	0m44.515s	0m7.124s	responsive.
dl140	FC 5	ext3	1m52.354s	0m44.515s	0m7.124s	run 5 instances still responsive

Centos/RHEL4 perform similarly to FC 5. You can see reiser is slightly faster than ext3 on the SUSE test, but it doesn't matter as they are blown away by a good kernel. The interesting thing is SuSE/Novell didn't really want to hear about this when I tried to open a ticket. I'll be trying again. I have benchmarks from DL380's and a reproduceable method, that doesn't rely on the C program, just dd (you can also produce the bug with sort and some other ways).

The nice thing here is we can double our performance by going to a new distribution.

The dismal thing is a 500$ desktop (1.7 P4 Celeron 512 Mb of ram, ata drive) with FC 5 was able to perform on par with a DL385 (~10,000$) dual Opteron, 8GB of ram, 6 drive SCSI raid array, on the first run. And able to beat the Opteron with multiple runs. That means this silicon garden is poorly optimized and utilized.

Two ways to sync files on servers.

Two ways to synchronize files:

rsync over ssh (superhandy with keys and a key agent):

rsync -av source destination

Where source and destination are ssh style: username@host:/path/to/

Another nice way to sync files is rdist.

rdist allows you to sync files to many nodes from a master (it is easy to set up and configure).
It is very handy. It is probably included with your distribution, it is definately used by several webservice providers and the homepage is here: http://www.magnicomp.com/rdist/

SLES9 kernel compile

I'm still getting to the bottom of why I/O on SLES 9 is so pathetic (future article with benchmarks on various HP server hardware). But I'm testing many kernel compiles and there wasn't a good recipe for SLES9 and vanilla 2.6 kernels. I've tested this with 2.6.16.1, 2.6.9, 2.6.12.6.

Sles 9 kernel build:

make sure you have the kernel source on the machine and gcc.

yast2 -i kernel-source gcc

then get the linux kernel and untar it (replace the kernel.org/foo.kernel stuff with a real path to a kernel):

wget http://kernel.org/foo.kernel.bz
tar -jxvf linux-2.6.6.1.tar.bz2

then move the kernel to /usr/src/

mv kernel-foo /usr/src/

remove old kernel build sim link:
rm /usr/src/linux

make new symlink to the new kernel:
ln -s /usr/src/kernel-foo /usr/src/linux

If you want to build a kernel with similar options to a SLES Kernel (building one with the old config), then you need to get and old .config file.
If you are running a plain SLES kernel:

zcat /proc/config.gz > /usr/src/linux/.config

or you can copy the .config file from a SLES kernel source:
cp /usr/src/linux-2.6.5-7.244/.config /usr/src/linux/.config

Now you have one more interactive part:
make oldconfig

That will ask questions about new options in the kernel that are not covered by the old .config only. You may be able to take all defaults.

I will break down the next steps with commentary, but the rest does not necessarily need intervention, so I usually stack them on a command line (see below):

Clean up configs:
make clean

Make a bootable linux image:
make bzImage

Compile loadable kernel modules (these are the same as drivers usually):
make modules

Install said drivers:
make modules_install

This makes a module dependency map and if it isn't done right you might not boot. The 2.6.16.1 will need to be replaced with the kernel version.
depmod -ae -F System.map 2.6.16.1

Install everything:
make install

So a line to do all of the post interactive stuff with a 2.6.12.6 kernel would look like:
make oldconfig && make clean && make bzImage && make modules && make modules_install && depmod -ae -F System.map 2.6.12.6 && make install

And then you might have to wait for a while. If you use grub as a bootloader, you shouldn't have to do anything else to run your new kernel except reboot. This whole process is so much easier than 2.0, 2.2 or 2.4 kernels. It still isn't as easy as apt-get upgrade kernel or yum update kernel or emerge or whatever, but such is the price you pay for SuSE.

Sending attachments from a script or Linux CLI

mutt -s SUBJECT -a ATTACHMENT user@domain.com

is a nice way. I like mutt as a mail user agent. It can do pgp and stuff too if you need to.

mailx is another way. I prefer mutt, but now you have at least two ways to send attachments from a linux script or command line.


UPDATE:
A third way posted by anonymous in the comments:

uuencode local_file.name remote_file.name | mail -s "file attachment" user@example.com

if you have uuencode installed and mail, it works very well. Thanks anonymous!