When too much RAM hurts- Centos/Redhat 5 and writes

Had an odd troubleshoot on linux.  DB server, over 200G of RAM, having problems every 30 seconds with DB queries.

The symptom was looking like blocking on I/O but not much actual I/O happening.  DB was on NFS without an async mount, so all writes being acknowledged.  Problem not really apparent on local disk that had write cache on the raid.  1G NFS mount made more problems than 10G.  So it kinda looked like write acknowledgement was a problem.  DB on a beefy NetApp so performance should be awesome.  Checked all the best practice stuff (NFS window size, netapp options, mysql .conf options. etc. etc.).  Had not gone to async on the mount yet.

Finally looking at what mysql was waiting for and how much BW was going to NFS mount on the netapp (nmon, iptraf, vmstat, iostat, top, etc), we found it waiting on a write but not really any iowait showing.

Then found this doc:

http://westnet.com/~gsmith/content/linux-pdflush.htm

Centos 5/Redhat 5 sets aside cache as a % of memory- good.  It flushes it after it gets a certain amount of memory or after 30 seconds.  Also good.  Unless you have like 28G of cache because it is taking 10% of physical memory- then  flushing 28G can be a problem.

So with a write instensive load turning that down to something the interface can keep up with makes a huge difference.  We went down to 1% of ram and flush basically every 3 seconds, although I'm considering tighter tuning on the flush interval.  Problem gone.  Writes stream to disk no problem.

Platforms for OS based Appliances

Sometimes I need a box for snort sensors or linux appliances, or BSD or even firewall distributions:

http://www.arinfotek.com/  Has some cool security oriented appliances- the Teak series.  

With iBase and Win Enterprise as competition in this sector.  Some have SSL acceleration or other cool security chips.

Here are some really cool boxes either way though:

The first is a full fledged tiny PC- also with bargain price:

Fit-PC2  It is a full PC even though it is small enough to stick to the back of your monitor.  Power consumption is a ridiculously low 5-8W.

The second is Hero Logic-   in competition with Soekris, but seems like a great polished product.  I'd love to try one out.

ESXi- VMWare's lightest and easiest Hypervisor

ESXi is lightweight and has almost all the features of ESX.  It is super easy to setup.

But did you know you can run it or install it from a USB key?  Well not at the same time, but check this out:

http://blog.torh.net/2009/02/22/running-vmware-esxi-hypervisor/

http://www.squishnet.com/?p=17

They are a little windows centric.  If you are on OS X or Linux it is easier (as you have dd and the linux boot tools on there.  But you get the point.

Virtual Server management- open source style

I really want to try the VMWare player for openqrm. It has all kinds of crazy features:

Support for different virtualization technologies

VMware, Xen, KVM and Linux-VServer vms can be managed transparently via openQRM. openQRM seamlessly support P2V (physical to virtual), V2P (virtual to physical) AND V2V (virtual to virtual) migration. This mean server appliances can not only move from physical to virtual (and back) easily but also that they can be migrated from virtualization technology A to virtualization technology B without any hassle.

Fully automatic Nagios configuration (single click) to monitor all systems and services

High-availability : "N to 1" fail-over

The role of flash in storage

EMC has some SSD/Flash for a segment of their storage systems. The MacBook Air famously has an SSD option. Solid State Disk and Flash are everywhere. So why can't I buy a shelf of it for my server room?

Well, SSD isn't quite ready to go in everything yet. As this blog points out: 1) it is in devices as cache/flash/nvram 2) if the storage isn't engineered properly, there are performance walls to run into 3) availability of dependable SSD is a problem (maybe not for iPods, hahah).

There is one thing I want to try out in the Garden:

http://www.fusionio.com/

Works in linux- Check, ridiculous performance profile for certain activities- Check. I just imagine these with linux RAID, LVM and a database that has a random read profile. I thought about making one as an appliance to shop around. Financing?

VMWare appliances to Parallels

Ever found a VMWare appliance or VM that you wanted to run under parallels?

well Virtualization Daily has an answer:

Convert the image from VMWare format to a raw hard disk image using Qemu.

qemu-img convert appliance-harddrive-name.vmdk -O raw appliance-harddrive-name-raw.hdd

There are some caveats: not all VMDKs can be converted, IDE disk only, single disk images (can't have multiple VMDK files). I've had pretty good fortune with Linux appliances so far (because it had the kernel drivers to load up even though the hardware had changed slightly).

They did some sample parallels appliances:

RHCE Self Study

I self studied for the RHCE exam for 5.1 and passed. It was difficult, and I have a solid background in Linux administration.

I used the Michael Jang RHCE self study book, which covers the exam topics. It could really use some sample exercises (like some sample test scenarios). Make sure you get the most recent version. If you can afford it, I think it is generally better to take the 1 week course RH300, after seeing some co-workers and friends take that route.

I would concentrate on reviewing the book then finding weak areas and getting the knowledge of the topics. You need to know everything on the exam blueprint. Then the important thing is working on speed. The main things to remember about the RHCE exam is it is performance based and has a time limit. So you must be able to do the relevant tasks in a quick time frame. Speed is of the essence. If you know one topic really well and can do things quickly, but another at a mediocre level and are slow and constantly referring to apropos or man pages, that may be enough to prevent finishing in a timely manner. Some referral to resources is expected, but you need to know your business and be quick about it.

For me, I was weak at SELinux and ACLs, as I use them very sparingly. So I spent extra time on those above and beyond the self study book and practiced tasks with a hands on manner. Topics that I was strong in, I spent the time reading the book and doing the material, but then I moved on.

The RHCE is a great certification and very satisfying to attain. Redhat's lab performance based exam is a great way to demonstrate linux knowledge. I learned some new things and some new ways to do things as I was studying for the exam.

How to tell if you have a MySQL performance problem

The top three things for mysql performance:

1) run 64bit OS on 64bit hardware with 64bit version of mysql (newer versions of mysql do better with large amounts of RAM if you aren't using 4.1+ you may not be able to use enough RAM).
2) make sure your indexes fit in RAM
3) Make sure you log slow queries and check the logs for problems. This is your number

Number 1 is easy.

Number 2 is a little harder. You can eyeball this if you are running ISAM databases by summing up all your IDX files and then seeing if MySQL is using more RAM than that.

But a really good way is to run a report: http://hackmysql.com/mysqlreport It has really good documentation and can make suggestions. You can use it to tune other options.

Number 3 requires a change in logging. You can also change the threshold- what is slow for you may not be for someone else. The problem with logs is you have to look at them, understand them and then act on them. Get going.

Virtualization for linux takes a step forward

www.enomalism.com

Hopefully it works as advertised. Looks like it has some features sorely missing from the opensource, management of multiple Xen Servers.

The interesting points:

# Create a cloud of abstracted, highly scalable, and managed compute infrastructure capable of hosting end-customer applications and billed by consumption.
# Simple easy to use web based user interface
# Automagically load balance and and monitor operations using a RESTFul API.
# VM Agnostic, migrate to and from various virtual environments including KVM/Qemu, Amazon EC2 and Xen. (OpenVZ, VirtualBox and VMware coming soon)
# Fully Automated platform with easy setup rules for a completely autonomous and self healing virtual environment

Those alone make it worth while. It depends on mysql for its data storage. I might be trying this out in the lab. When I do, I'll post back.

Since Xen is at least where VMWare Server/ESX is, it is interesting that enomalism might give at least a set of the features available in the ESX/virtual infrastructure product. The main weakness of Xen is managing a pool of Xen servers, and if enomalism does what it advertises, it is a huge step forward.

Xen Virtualization on CentOS

You can use many tools on Centos 5 to manage virtual servers:

CLI tools: xm, install-sh, virsh
GUI tools: virt-manager

Here are some useful links-

How to install a virtual server (after installing CentOS with the Virtualization packages):
http://wiki.centos.org/HowTos/Xen/InstallingCentOSDomU

That is a command line tutorial. But you could use virt-manager if you have X and Xen. It walks you though all the steps in a much easier way, but it is a good idea to know what is going on behind the scenes. Everything in this tutorial can be done by virt-manager without much hassle.

This is how you install a native virtualization DomU like Windows:

http://wiki.centos.org/HowTos/Xen/InstallingHVMDomU

It also has the way to check your processor extensions.

This one has some basics about keyboard mapping and also has how to deal with SELinux (without turning it off people!):

http://wiki.centos.org/TipsAndTricks/Xen

This article has P2V basics. It gives some insight into what the virt-p2v boot iso (linked in the intro post) does behind the scenes:

http://wiki.centos.org/HowTos/Xen/MoveNative2DomU

And if you had to move something yourself if you had to do it manually.

Free Virtualization Introduction

Xen on Linux is usable out of the box now. If you download CentOS 5.1 (or later I would guess), you can setup virtualization during install.

Virtualization lets you run multiple instances of an operating system and applications on one piece of hardware. You can imagine or read on the net why that would be useful.

Some terms to translate for those familiar with VMWare or Parallels style terminology:

Host OS: VMWare Server (not ESX) uses an operating system (windows or linux) as a host, ESX is the host, and Parallels uses OS X. With Xen virtualization this roughly corresponds to Domain0. Domain0 has unlimited privileges.

Guest OS: You can run windows, linux or other operating systems under VMWare or Parallels. On Xen the guest OS is known as a DomU (domain U), which has direct access to hardware. This domain has limited resources (ram and processors as configured), by default it has no access to hardware.

Paravirtualization vs. fully virtualized-

paravirtualization uses software to allow virtualization. Under CentOS 5, the paravirtualizable guest operating systems are basically linux, linux and linux. Native virtualization uses processor extensions so you need a recent Intel processor that supports or a AMD that supports AMD-V extensions. The extensions aren't very common outside of server class chips (Xeon and Opteron). Here is how to tell if you have the extensions:

egrep '^flags.*(vmx|svm)' /proc/cpuinfo

If you don't have a lines of output with vmx or svm in them, you don't have the extensions.

P2V: Physical to Virtual. How to import physical servers (usually old), to a virtual machine, which can save you time and money and get you out of supporting old and busted servers. There are some pay tools for P2V for Xen and Linux. But here is a pretty good free one:

http://et.redhat.com/~rjones/virt-p2v/

That is it for the intro. More tomorrow with specific examples.

system-config-netboot

Redhat, time for your verbal beating. RHEL3 and 4 I love you. RHEL 5 you are obviously rushed out the door. Proper XEN support starts in 5.1. But the real symptom of the this rush is:

PXE boot.

That is right. The Doc guide used to have directions on PXE boot/network install for RHEL5. It mentioned system-config-netboot (which wasn't in RHEL5). So instead of fixing system-config-netboot somebody went and trashed the doc excising references to system-config-netboot, but not the whole section and not the references to the pxeos tool which is part of the package.

Well if you are lazy like me, and like the tool to setup the /tftpboot directory and not to have to remember anything but the /etc/dhcpd.conf commands:

You can use the system-config-netboot from Fedora 8 (which was kind of a pain to find- thanks to all the mirrors that haven't mirrored FC8 as of this article) right on RHEL 5.1 (for sure, tested, it works).

My favorite property of RHEL is the kickstart/PXE boot ease of deployment. An afternoon or two of fun and you can deploy 1000s or 10,000s of boxes with no trouble. Unless of course you can't get system-config-netboot to setup your PXE environment for you... because it isn't on the distribution.

There is a ticket on bugzilla.redhat.com that says it will be in RHEL5.2. Which will be nice.

su - on sles 9 doesn't respect limits.

This can end up causing some good security holes.

If you put in a ulimit for higher files for a user named oracle (just an example) in /etc/security/limits.conf:

oracle soft nofile 2048
oracle hard nofile 65535


then run ulimit -a >~limits in cron. Your file limit is 1024 (the default on SLES).

Why? I'll tell you.

Crond does a su - to you user. ON SLES SU - DOESN"T RESPECT THE LIMITS IF THEY ARE HIGHER!!! I haven't tried lower yet.

If you login interactively your limits are fine. If you su - oracle as root, your limits are f-d up (back to the system default 1024).

This doesn't happen on RH/Fedora and as far as I can tell Debian/Ubuntu.

updating one package on sles 9

online_update --url 'my_local_update_server' --force -S patch-10903

or you can do what I do and write script that checks architecture and rpm -ivh http://installserver/sample-i386.rpm from a web server. The script is useable on RH or SLES for onesy twosey patches.

SLES still sucks

but this makes it suck less:

automatic update at your command-

It still doesn't do the right thing with a kernel (it upgrades instead of installs leaving modules broken and your currently running machine in bad need of a reboot), so it is dangerous in some ways.

Just FYI here is the magic:

ssh -n $HOSTNAME "which online_update && online_update -gVu http://servername/YOU/ && online_update -iV"

This executes the command "which online_update" and if that is successful runs online_update to download packages from your You server (yast2 can help you make one, that works) to the local box, then if that is successful online_updates from the packages on the local box. No other combination of switches appears to work update a machine via online_update. SLES needs to download then install.

RHEL/CentOS does the right thing with the kernel and only requires a "which yum && yum -y update" and you can run your own repositories if you use yum, like I do, so it is still better (if you use up2date, that is okay too- but yum does better with the repositories.

The reason I run "which commandname" is to avoid trying to yum a SLES box and online_update a CentOS.

I can feed the script file a list of servers and it will go patch the lot. You can save the output and have a list of patched boxes.

tar and ssh like peanut butter and jelly

Everybody knows the classic command to make a tar file:

tar -cvf file.tar directory_to_tar

and if you use gnu tar you can improve it with the compress in that step:

tar -zxvf file.tar directory_to_tar
or
tar -jxvf file.tar directory_to_tar

for gzip and bzip respectively...

but did you know:

tar zcvf - /directory_to_tar | ssh hostname "cat > file.tgz"

so you can tar gzip on one end and write the file on the other end of an ssh session?

Huh? Did you?

Yes I know I still owe an automounter deal... I promise it is on the way.

Too busy gardening the Silicon Rust...

automounter is awesome

Ever had an nfs client hang because the server rebooted or hiccupped?

Ever had a web of nfs mounts that won't come up cleanly because there isn't a good order? Like Server A mounts Server B that mounts Server C that mounts Server A (not a good practice, but all too common in crufty environments).

Well automounter can solve some of those problems.

I'll post the technical details tomorrow or the next day.

MySQL stupidity

If you want to be able to troubleshoot mysql:

1) mysqlreport is very cool: http://hackmysql.com. The documentation is awesome and for a quick rush it is fabulous. After you get it installed try having it mail you a tab delimited report: mysqlreport --email user@domain.com --pass --all -tab

2) mytop is also very cool:

3) making sure your indexes fit in RAM is very cool, ls *MYI and sum them up. That amount should be less than physical ram (and less than the key_buffer_size variable from the my.cnf--- you did already check the my.cnf).

4) you can make some changes on the fly, check your variables with show variables; then you can set them with the mysql client

5) make sure your mysql install is logging :roll eyes:

getting rid of standard error

scp foo server: 2>/dev/null


the descriptor 2 is for standard error.

Redirecting 2 to /dev/null makes the standard error go away.

LVM and a rant on bonding and 802.1q

use LVM if you are using a modern linux.

Really. It will make your life easier.

Bonding and 802.1q configuration under linux suck right now. If I get time to experiment, I will figure out the model config. But really Redhat or Suse needs to come out with a configuration tool so you can bond interfaces (and use static or dhcp addresses) and use 802.1q vlan tagging on those interfaces (or non-bonded interfaces).