WRT54G Benchmarks

Alright benchmarked the WRT54G v.3 that I have with dd-wrt v.24. 216Mhz default, no overclock. Basically the thing does everything in software except layer 2 switching and runs out of CPU. I needed to know where the cut off is, because broadband keeps getting faster. You can depend on the box for about 20Mb/s throughput. If you approach that limit, you may run into CPU problems. I also found out the default window size on iperf on Windows client is 8K (too small for 100Mb networks). So all my tests use 256k window sizes (default on linux and os x). All tests were performed with iperf.

Full test notes for posterity:
test 1-LAN bandwidth- both ports plugged into lan ports, same vlan: iperf -s on vista server, iperf -c on OS X client 93.7 Mb/s no CPU on WRT54g.
test 2- LAN bandwidth- both ports plugged into lan ports, same vlan: iperf -s on OS X iperf -c on vista, 61Mb/s (too small a windows size 8K on windows), no CPU.
test 3- LAN bandwidth same as 2, but with real window sizes: iperf -s -w 256k on OS X, iperf -c 192.168.100.8 -w 256k- 93.7Mb/s second switched no CPU.
test 4- routing (no SPI/Firewall) from LAN to WAN same iperf settings as above except client is on the other network - 22.7Mb/s CPU pegged.
test 5- reverse client and server traffic now WAN to LAN- 22.7Mb/s CPU pegged.
test6 - enable SPI/firewall, repeat test 5- 18.3Mb/s. CPU pegged.
test7- reverse client and server now LAN to WAN- 18.3Mb/s CPU pegged.
test8- LAN to wireless- 17.3Mb/s no encryption G only, some 2.4Ghz interference. CPU not pegged, but high (fluctuating between 50-65%).
test9- wireless to LAN- same as above- 17.3Mb/s CPU, not pegged but high (fluctuating between 50-65%).

I should try with WPA, but there is so much interference here. I can't be sure what I'm testing, the wireless, the encryption, the interference.

Summary of results: the switch in the Linksys I have is in hardware- it gives 93.7Mb/s throughput. So L2 performance is good.

Just routing from Lan to WAN with no firewall you can get 22.7Mb/s and it is CPU limited. With a firewall enabled from LAN to WAN, you are limited to 18.3Mb/s and it is CPU limited. Linux iptables style firewall (what dd-wrt uses) is a pretty efficient packet filter (about 10% overhead it looks like).

On the wireless from wireless I got 17.3Mb/s throughput (no encryption), and there was still CPU left. That implies the wireless G protocol or interference will probably be the limit not WRT54G. Basically if you are doing more than 17Mb/s, the WRT54G could be a choke point as could wireless 802.11g. If you want more performance overclock or get a higher CPU dd-wrt capable box and don't use 802.11g wireless. We'll call it 17Mb/s is the upper end for an internet connection for these boxes (due to CPU and 802.11g being limiting factors), more features may make the ceiling a little lower.

The benchmarks...

I promised some comparisions of SLES9 and why it has slow I/O. I haven't cleared releasing the application yet (it is a small peice of C code that opens as many files as you throw as an argument and then writes to those files). In place of that code, a nice workabel substitute is substitue a one large file write: "time dd if=/dev/zero of=/tmp/testfile bs=16k count=65536 ". You can also try reads, but that is more divergent based on caching the filesystem, if you bench reads, reboot between benchmarks (or otherwise flush all cache and buffers).

I've benchmarked this on HP servers (and a couple desktops). I've tried different filesystems, different kernel versions and different I/O subsystems (SCSI, SATA, ATA). The numbers pretty much go about the same way (except kernel version as you will see). Apparently Redhat backported the patch or knows about the bug and fixes it in their kernel.

Here are some benchmarks all on the same HP DL140 hardware (2.8 Xeon, 1G ram, sata drive, 11211 Bogomips):

Hardware	OS	FS Type	time a.out 1000	user	sys	Notes
dl140	SLES 9	reiser	5m15.042s	0m38.427s	0m8.303s	unresponsive after a few seconds and well after test ls will hang
dl140	SLES 9	ext3	5m31.042s	0m38.427s	0m8.303s	unresponsive after a few seconds and well after test ls will hang
dl140	SLES 9	reiser	3m53.546s	0m44.687s	0m3.052s	2.6.9 kernel unresponsive
dl140	SLES 9	reiser	2m51.070s	0m44.687s	0m3.052s	2.6.16.1 vanilla kernel responsive.
dl140	FC 5	ext3	1m52.354s	0m44.515s	0m7.124s	responsive.
dl140	FC 5	ext3	1m52.354s	0m44.515s	0m7.124s	run 5 instances still responsive

Centos/RHEL4 perform similarly to FC 5. You can see reiser is slightly faster than ext3 on the SUSE test, but it doesn't matter as they are blown away by a good kernel. The interesting thing is SuSE/Novell didn't really want to hear about this when I tried to open a ticket. I'll be trying again. I have benchmarks from DL380's and a reproduceable method, that doesn't rely on the C program, just dd (you can also produce the bug with sort and some other ways).

The nice thing here is we can double our performance by going to a new distribution.

The dismal thing is a 500$ desktop (1.7 P4 Celeron 512 Mb of ram, ata drive) with FC 5 was able to perform on par with a DL385 (~10,000$) dual Opteron, 8GB of ram, 6 drive SCSI raid array, on the first run. And able to beat the Opteron with multiple runs. That means this silicon garden is poorly optimized and utilized.