The benchmarks...

I promised some comparisions of SLES9 and why it has slow I/O. I haven't cleared releasing the application yet (it is a small peice of C code that opens as many files as you throw as an argument and then writes to those files). In place of that code, a nice workabel substitute is substitue a one large file write: "time dd if=/dev/zero of=/tmp/testfile bs=16k count=65536 ". You can also try reads, but that is more divergent based on caching the filesystem, if you bench reads, reboot between benchmarks (or otherwise flush all cache and buffers).

I've benchmarked this on HP servers (and a couple desktops). I've tried different filesystems, different kernel versions and different I/O subsystems (SCSI, SATA, ATA). The numbers pretty much go about the same way (except kernel version as you will see). Apparently Redhat backported the patch or knows about the bug and fixes it in their kernel.

Here are some benchmarks all on the same HP DL140 hardware (2.8 Xeon, 1G ram, sata drive, 11211 Bogomips):

Hardware	OS	FS Type	time a.out 1000	user	sys	Notes
dl140	SLES 9	reiser	5m15.042s	0m38.427s	0m8.303s	unresponsive after a few seconds and well after test ls will hang
dl140	SLES 9	ext3	5m31.042s	0m38.427s	0m8.303s	unresponsive after a few seconds and well after test ls will hang
dl140	SLES 9	reiser	3m53.546s	0m44.687s	0m3.052s	2.6.9 kernel unresponsive
dl140	SLES 9	reiser	2m51.070s	0m44.687s	0m3.052s	2.6.16.1 vanilla kernel responsive.
dl140	FC 5	ext3	1m52.354s	0m44.515s	0m7.124s	responsive.
dl140	FC 5	ext3	1m52.354s	0m44.515s	0m7.124s	run 5 instances still responsive

Centos/RHEL4 perform similarly to FC 5. You can see reiser is slightly faster than ext3 on the SUSE test, but it doesn't matter as they are blown away by a good kernel. The interesting thing is SuSE/Novell didn't really want to hear about this when I tried to open a ticket. I'll be trying again. I have benchmarks from DL380's and a reproduceable method, that doesn't rely on the C program, just dd (you can also produce the bug with sort and some other ways).

The nice thing here is we can double our performance by going to a new distribution.

The dismal thing is a 500$ desktop (1.7 P4 Celeron 512 Mb of ram, ata drive) with FC 5 was able to perform on par with a DL385 (~10,000$) dual Opteron, 8GB of ram, 6 drive SCSI raid array, on the first run. And able to beat the Opteron with multiple runs. That means this silicon garden is poorly optimized and utilized.