Steve Kemp's Blog

I've been a little quieter than usual recently, having spent more time outdoors putting cute people in front of the camera. However that said I've still been doing some things. Most interestingly I've given away my first ever project.

The collection of small scripts known as xen-tools (which was initially a sleazy hack to go with a small introduction to Xen article) has now got new developers, and a new home where development is continuing.

This isn't the first time I've stopped working on something, but it is the first time I've explicitly "given away" a project. (Mostly on the basis that if I didn't care nobody else did either, or people cared but were too busy/unable to actually do soemthing useful.)

I'll be following new updates with interest, even though these days I'm 100% Xen-free. No need to go into huge details about why, but I'm enjoying KVM.

Having said that I recently got into a huge mess with a combination of LVM, KVM, and ext3. I've written up the details on ServerFault in the optimistic hope that somebody will report having experienced a similar problem. If you have seen something similar I'd love to hear from you.

Otherwise I'm genuinely at a loss to understand what went wrong, and why things failed. I could suspect hardware issues, but that feels like a cop-out, albeit one that has a potential solution (Mad Hatter: All Change!) rather than my current answer and explaination "It broke. I don't know why. It might happen again. It might not. Trust me?".

ObFilm: Alice In Wonderland (1951 version.)

Good people steal ideas, right? On that basis I setup a static domain to host the javascript and icons I use upon a few different sites & projects. This was preempted by the release of a new version of the excellent jQuery library.

I also managed to put together a tremendous hack to solve a pretty annoying problem running multiple distributions from a single external kernel under KVM.

Ubuntu users, in particular, will be well aware of dmesg SPAM coming from the use of CONFIG_SYSFS_DEPRECATED.

In short the way that the kernel presents information beneath the /sys tree has changed over the life of the kernel - and this has a knock-on effect to the userspace supplied by different distributions and releases of GNU/Linux.

Some distributions need an "old" kernel and an "old" udev with "old" udev rules in order to create the appropriate device nodes such that the kernel will boot & mount its filesystems. (i.e. These need CONFIG_SYSFS_DEPRECATED to be set.)

Conversely some distributions mandate a "new" minimum kernel version, and supply a "new" version of udev with "new" udev rules and they absolutely will not function when presented with an "old" kernel. (i.e. They must have kernels without CONFIG_SYSFS_DEPRECATED set.)

I've solved this problem via a kernel patch which is both evil and genius. The details are a little me-specific, but in short:

• devtmpfs is used to setup and mount an initial /dev tree before /sbin/init is launched..
• udev launches later and mounts a tmpfs over /dev such that it can start creating its own nodes.
• At this point evil begins: I've patched the kernel such that any attempt to mount a tmpfs filesystem at /dev is silently changed to mount a devtmpfss filesystem instead.
  • The alternative is that udev creates many nodes, but manages to fail to create the root & swap nodes such that the KVM guests fail to boot.

Ultimately udev doesn't get an empty /dev tree to play with, instead it finds one already pre-populated, such that any devices it cannot create are there regardless - because the devtmpfs implementation has already created them.

Genius. And evil. So very evil.

Meh.

Steal that idea. I dare you .. (I'm impressed at how well devtmpfs works, and how easy I was able to make my "patch of evil"^tm. Just a few lines in fs/namespace.c.)

ObSubject: The Last House On The Left

This week I've been mostly migrating guests from Xen to KVM. This has been a a pretty painless process, and I'm happy with the progress.

The migration process is basically:

• Stop the Xen guest (domU).
• Mount the filesystem (LVM-based) upon the Xen host (dom0).
• Copy those mounted contents over to a new LVM location upon the KVM host using rsync.
• Patch the filesystem once the rsync has been moved:
  • Create /dev nodes for the new root & swap devices.
  • Update /etc/fstab to use those devices.
• Fiddle with routing to ensure traffic for the guest arrives at the KVM host, rather than the Xen host.
• Hardwire static routes on the dom0 so that cross-guest traffic works correctly.
• Boot up the new guest, and hope for the best.

The main delay in the migration comes from the rsync step which can take a while when there are a lot of small files involved. In the future I guess I should ask users to do this themselves in advance, or investigate the patches to rsync that let block devices be transferred - rather than filesystem contents.

Thankfully all of the guests I've moved thus far have worked successfully post-migration, and performance is good. (The KVM host is going to be saturated with I/O when the rsyncing of a new guest is carried out - so I expect performance to dip while that happens, but once everybody is moved it should otherwise perform well.)

So Xen vs. KVM? Its swings and roundabouts really. In terms of what I'm offering to users there isn't too much difference between them. The only significant change this time round is that I'll let users upload their own kernel and one brave soul has already done that!

ObFilm: Pitch Black

Recently I've been spidering the internet, merrily downloading content for the past few days.

The intention behind the spidering is to record, in a database, the following pieces of information for each image it stumbles across:

• The page that contained the link to this image. (i.e. the image parent)
• The image URL.
• The MD5sum of the image itself.
• The dimensions of the image.

I was motivated by seeing an image upon a website and thinking "Hang on I've seen that before - but where?".

Thus far I've got details of about 30,000 images and I can now find duplicates or answer the question "Does this image appear on the internet and if so where?".

Obviously this is going to be foiled trivially via rotations, cropping, or even resizing. But I'm going to let the spider run for the next few days at least to see what interesting things the data can be used for.

In other news I'm a little behind schedule but I'm going to be moving from Xen to KVM over the next week or ten days.

My current plan is to setup the new host on Monday, move myself there that same day. Once that's been demonstrated to work I can move the other users over one by one, probably one a day. That will allow a little bit of freedom for people to choose their downtime window, and will ensure that its not an all-or-nothing thing.

The new management system is pretty good, but I have the advantage here in that I've worked upon about four systems for driving KVM hosting. The system allows people to enable/disable VNC access, use the serial console, and either use one of a number of pre-cooked kernels or upload their own. (Hmmm security you say?)

ObFilm: Chasing Amy

This weekend I mostly fiddled around migrating machines from Xen hosting to KVM hosting. Ultimately it was largely a waste of time, due to various other factors. Still with a bit of luck it will be possible to move the machiens next week.

That aside I spent a while updating my blogspam detection site. As a brief recap this site offers a simple XML-RPC service which allows you to test whether incoming blog comments are spam or not.

Originally this was put together to fight an invasion of comments submited to the Debian Administration website: The site currently shows:

Depressing. But not as depressing as the real live stats which show since I last reset the counters 36,995 spam comments vs. 1,206 non-spam comments. (live updating counters here)

Anyway I updated the service today to add two new plugins, both of which are a little reactionary.

The first new plugin is called "multilink" and is based upon the observation that spammers rarely know the markup of the site they are submitting comments to. This means you can frequently see submitted comments like this:

 <a href="http://spam.com">buy viagra</a>
 [url=http://spam.com]buy viagra[/url]
 [link=http://spam.com]buy me[/link]

Here we have three different styles of links - "a href", "link=", and "url=". I figure this is a clear indicator of a confused mind, or more likely a spammer.

The second new plugin is designed to stop people who enter "<strong>" words. It is a little coarse but actuall zero false positives in the real world so I'm going to leave it live to see how it works out.

In happier news I'm just back from a trip to the beach. Sand rocks. Even if it wasn't windy enough for my kite ..

ObFilm: Dracula ("Bram Stoker's Dracula" - 1992)

Since I already shared it elsewhere here is my KVM-launcher, and the mercurial repository it lives in.

I'll add my kvm-shell program later - the tools I've written so far is mostly standalone, rather than a package.

This is almost a content-free post, but I can pretend it isn't because I'm testing a new theme on my blog. The theme is included in the new release of my chronicle blog compiler which was released yesterday.

ObFilm: Heathers

I think I've made the decision that at some point in the next few months the xen-hosting.org setup I maintain will be going away, and will be replaced with kvm-hosting(.org).

The only issue I need to ponder is handling the migration with the minimum downtime.

The plan would probably involve upgrading the host machine to Lenny, then installing KVM and fiddling with filesystems until the guests boot. I suspect it wouldn't be a huge job, but there are a few issues that will need to be planned.

Most notably I expect that most of the current guests don't have grub installed, etc, so we'd be in the position to use an external kernel + initrd. That's not an insurmountable problem, but I know that externally supplied kernels have caused me problem in the past with KVM.

Perhaps the actual plan would be to wait until September at which point I could order a new machine and cancel the current one. That would mean another increase in spec and the migration process would be a lot simpler - instead of everybody being offline for a few hours I could migrate guests individually from the old host to the new.

Anyway decisions decisions ..

ObFilm: Buffy - But we'll pretend the TV series counts as a film, kthxbye?

Gunnar Wolf made an interesting post about KVM today which is timely.

He points to a simple shell script for managing running instances of KVM which was a big improvement on mine - and so is worth a look if you're doing that stuff yourself.

Once I find time I will document my reasons for changing from Xen to KVM, but barring a few irritations I'm liking it a lot.

I made a new release of the chronicle blog compiler yesterday, mostly to update one of the themes.

That was for purely selfish reasons as I've taken the time to update the antispam protection site I'm maintaining. There have been some nice changes to make it scale more and now it is time for me to make it look prettier.

(A common theme - I'm very bad at doing website design.)

So now the site blog matches the real site.

ObQuote: Resident Evil

Yesterday I was forced to test my backup system in anger, on a large scale, for the first time in months.

A broken package upgrade meant that my anti-spam system lost the contents of all its MySQL databases.

That was a little traumatic, to say the least. But happily I have a good scheme of backups in place, and only a single MX machine was affected.

So, whilst there was approximately an hour of downtime on the primary MX the service as a whole continued to run, and the secondary (+ trial tertiary) MX machines managed to handle the load between them.

I'm almost pleased I had to suffer this downtime, because it did convince me that my split-architecture is stable - and that the loss of the primary MX machine isn't a catastrophic failure.

The main reason for panicing was that I was late for a night in the pub. Thankfully the people I were due to meet believe in flexible approaches to start times - something I personally don't really believe in.

Anyway the mail service is running well, and I've setup "instant activation now", combined with a full month of free service which is helping attract more users.

Apart from that I've continued my plan of migrating away from Xen, and toward KVM. That is going well.

I've got a few guests up and running, and I'm impressed at how stable, fast, and simple the whole process is. :)

ObQuote: Brief Encounter

(That is a great film; and a true classic. Recommended.)