When too much RAM hurts- Centos/Redhat 5 and writes

Had an odd troubleshoot on linux.  DB server, over 200G of RAM, having problems every 30 seconds with DB queries.

The symptom was looking like blocking on I/O but not much actual I/O happening.  DB was on NFS without an async mount, so all writes being acknowledged.  Problem not really apparent on local disk that had write cache on the raid.  1G NFS mount made more problems than 10G.  So it kinda looked like write acknowledgement was a problem.  DB on a beefy NetApp so performance should be awesome.  Checked all the best practice stuff (NFS window size, netapp options, mysql .conf options. etc. etc.).  Had not gone to async on the mount yet.

Finally looking at what mysql was waiting for and how much BW was going to NFS mount on the netapp (nmon, iptraf, vmstat, iostat, top, etc), we found it waiting on a write but not really any iowait showing.

Then found this doc:

http://westnet.com/~gsmith/content/linux-pdflush.htm

Centos 5/Redhat 5 sets aside cache as a % of memory- good.  It flushes it after it gets a certain amount of memory or after 30 seconds.  Also good.  Unless you have like 28G of cache because it is taking 10% of physical memory- then  flushing 28G can be a problem.

So with a write instensive load turning that down to something the interface can keep up with makes a huge difference.  We went down to 1% of ram and flush basically every 3 seconds, although I'm considering tighter tuning on the flush interval.  Problem gone.  Writes stream to disk no problem.

system-config-netboot

Redhat, time for your verbal beating. RHEL3 and 4 I love you. RHEL 5 you are obviously rushed out the door. Proper XEN support starts in 5.1. But the real symptom of the this rush is:

PXE boot.

That is right. The Doc guide used to have directions on PXE boot/network install for RHEL5. It mentioned system-config-netboot (which wasn't in RHEL5). So instead of fixing system-config-netboot somebody went and trashed the doc excising references to system-config-netboot, but not the whole section and not the references to the pxeos tool which is part of the package.

Well if you are lazy like me, and like the tool to setup the /tftpboot directory and not to have to remember anything but the /etc/dhcpd.conf commands:

You can use the system-config-netboot from Fedora 8 (which was kind of a pain to find- thanks to all the mirrors that haven't mirrored FC8 as of this article) right on RHEL 5.1 (for sure, tested, it works).

My favorite property of RHEL is the kickstart/PXE boot ease of deployment. An afternoon or two of fun and you can deploy 1000s or 10,000s of boxes with no trouble. Unless of course you can't get system-config-netboot to setup your PXE environment for you... because it isn't on the distribution.

There is a ticket on bugzilla.redhat.com that says it will be in RHEL5.2. Which will be nice.