Steve Kemp's Blog

This week I've been mostly migrating guests from Xen to KVM. This has been a a pretty painless process, and I'm happy with the progress.

The migration process is basically:

• Stop the Xen guest (domU).
• Mount the filesystem (LVM-based) upon the Xen host (dom0).
• Copy those mounted contents over to a new LVM location upon the KVM host using rsync.
• Patch the filesystem once the rsync has been moved:
  • Create /dev nodes for the new root & swap devices.
  • Update /etc/fstab to use those devices.
• Fiddle with routing to ensure traffic for the guest arrives at the KVM host, rather than the Xen host.
• Hardwire static routes on the dom0 so that cross-guest traffic works correctly.
• Boot up the new guest, and hope for the best.

The main delay in the migration comes from the rsync step which can take a while when there are a lot of small files involved. In the future I guess I should ask users to do this themselves in advance, or investigate the patches to rsync that let block devices be transferred - rather than filesystem contents.

Thankfully all of the guests I've moved thus far have worked successfully post-migration, and performance is good. (The KVM host is going to be saturated with I/O when the rsyncing of a new guest is carried out - so I expect performance to dip while that happens, but once everybody is moved it should otherwise perform well.)

So Xen vs. KVM? Its swings and roundabouts really. In terms of what I'm offering to users there isn't too much difference between them. The only significant change this time round is that I'll let users upload their own kernel and one brave soul has already done that!

ObFilm: Pitch Black

Recently I've been spidering the internet, merrily downloading content for the past few days.

The intention behind the spidering is to record, in a database, the following pieces of information for each image it stumbles across:

• The page that contained the link to this image. (i.e. the image parent)
• The image URL.
• The MD5sum of the image itself.
• The dimensions of the image.

I was motivated by seeing an image upon a website and thinking "Hang on I've seen that before - but where?".

Thus far I've got details of about 30,000 images and I can now find duplicates or answer the question "Does this image appear on the internet and if so where?".

Obviously this is going to be foiled trivially via rotations, cropping, or even resizing. But I'm going to let the spider run for the next few days at least to see what interesting things the data can be used for.

In other news I'm a little behind schedule but I'm going to be moving from Xen to KVM over the next week or ten days.

My current plan is to setup the new host on Monday, move myself there that same day. Once that's been demonstrated to work I can move the other users over one by one, probably one a day. That will allow a little bit of freedom for people to choose their downtime window, and will ensure that its not an all-or-nothing thing.

The new management system is pretty good, but I have the advantage here in that I've worked upon about four systems for driving KVM hosting. The system allows people to enable/disable VNC access, use the serial console, and either use one of a number of pre-cooked kernels or upload their own. (Hmmm security you say?)

ObFilm: Chasing Amy

This weekend I mostly fiddled around migrating machines from Xen hosting to KVM hosting. Ultimately it was largely a waste of time, due to various other factors. Still with a bit of luck it will be possible to move the machiens next week.

That aside I spent a while updating my blogspam detection site. As a brief recap this site offers a simple XML-RPC service which allows you to test whether incoming blog comments are spam or not.

Originally this was put together to fight an invasion of comments submited to the Debian Administration website: The site currently shows:

Depressing. But not as depressing as the real live stats which show since I last reset the counters 36,995 spam comments vs. 1,206 non-spam comments. (live updating counters here)

Anyway I updated the service today to add two new plugins, both of which are a little reactionary.

The first new plugin is called "multilink" and is based upon the observation that spammers rarely know the markup of the site they are submitting comments to. This means you can frequently see submitted comments like this:

 <a href="http://spam.com">buy viagra</a>
 [url=http://spam.com]buy viagra[/url]
 [link=http://spam.com]buy me[/link]

Here we have three different styles of links - "a href", "link=", and "url=". I figure this is a clear indicator of a confused mind, or more likely a spammer.

The second new plugin is designed to stop people who enter "<strong>" words. It is a little coarse but actuall zero false positives in the real world so I'm going to leave it live to see how it works out.

In happier news I'm just back from a trip to the beach. Sand rocks. Even if it wasn't windy enough for my kite ..

ObFilm: Dracula ("Bram Stoker's Dracula" - 1992)

I think I've made the decision that at some point in the next few months the xen-hosting.org setup I maintain will be going away, and will be replaced with kvm-hosting(.org).

The only issue I need to ponder is handling the migration with the minimum downtime.

The plan would probably involve upgrading the host machine to Lenny, then installing KVM and fiddling with filesystems until the guests boot. I suspect it wouldn't be a huge job, but there are a few issues that will need to be planned.

Most notably I expect that most of the current guests don't have grub installed, etc, so we'd be in the position to use an external kernel + initrd. That's not an insurmountable problem, but I know that externally supplied kernels have caused me problem in the past with KVM.

Perhaps the actual plan would be to wait until September at which point I could order a new machine and cancel the current one. That would mean another increase in spec and the migration process would be a lot simpler - instead of everybody being offline for a few hours I could migrate guests individually from the old host to the new.

Anyway decisions decisions ..

ObFilm: Buffy - But we'll pretend the TV series counts as a film, kthxbye?

Gunnar Wolf made an interesting post about KVM today which is timely.

He points to a simple shell script for managing running instances of KVM which was a big improvement on mine - and so is worth a look if you're doing that stuff yourself.

Once I find time I will document my reasons for changing from Xen to KVM, but barring a few irritations I'm liking it a lot.

I made a new release of the chronicle blog compiler yesterday, mostly to update one of the themes.

That was for purely selfish reasons as I've taken the time to update the antispam protection site I'm maintaining. There have been some nice changes to make it scale more and now it is time for me to make it look prettier.

(A common theme - I'm very bad at doing website design.)

So now the site blog matches the real site.

ObQuote: Resident Evil

Yesterday I was forced to test my backup system in anger, on a large scale, for the first time in months.

A broken package upgrade meant that my anti-spam system lost the contents of all its MySQL databases.

That was a little traumatic, to say the least. But happily I have a good scheme of backups in place, and only a single MX machine was affected.

So, whilst there was approximately an hour of downtime on the primary MX the service as a whole continued to run, and the secondary (+ trial tertiary) MX machines managed to handle the load between them.

I'm almost pleased I had to suffer this downtime, because it did convince me that my split-architecture is stable - and that the loss of the primary MX machine isn't a catastrophic failure.

The main reason for panicing was that I was late for a night in the pub. Thankfully the people I were due to meet believe in flexible approaches to start times - something I personally don't really believe in.

Anyway the mail service is running well, and I've setup "instant activation now", combined with a full month of free service which is helping attract more users.

Apart from that I've continued my plan of migrating away from Xen, and toward KVM. That is going well.

I've got a few guests up and running, and I'm impressed at how stable, fast, and simple the whole process is. :)

ObQuote: Brief Encounter

(That is a great film; and a true classic. Recommended.)

I installed Debian upon a new desktop machine yesterday, via a PXE network boot.

It was painless.

Getting xen up and running, with a 32-bit guest and a 64-bit guest each running XDMCP & VNC was also pretty straightforward.

There is a minor outstanding problem with the 32-bit xen guest though; connecting to it from dom0, via XDMCP, I see only a blank window - no login manager running.

GDM appears painlessly when I connect via VNC.

The relevent configuration file looks like this:

# /etc/gdm/gdm.conf
[security]
AllowRoot=true
AllowRemoteRoot=true

[xdmcp]
Enable=true

The same configuration on the 64-bit guest works OK for both cases.

(I like to use XDMCP for accessing the desktop of Xen guests, since it means that I get it all full-screen, and don't have to worry about shortcuts affecting the host system and not the guest - as is the case if you're connecting via VNC, etc).

Weirdness. Help welcome; I'm not 100% sure where to look

Anyway, once again, a huge thank you to the Debian Developers, bug submitters, and anybody else involved peripherally (such as myself!) with Debian!

I love it when a plan comes together.

SSL

ObRandom: Where is the cheapest place to get an SSL certificate, for two years, which will work with my shiny Apache2 install?

Somebody, rightly, called me for not having SSL available as an option on my mail filtering website.

I've installed a self-signed certificate just now, but I will need to pay the money and buy a "real" one shortly.

So far completessl.com seems to be high in the running:

• 1 year - £26
• 2 years - £49

For double-bonus points they accept Paypal which most of my customers pay with ..

ObQuote: The Princess Bride

Whilst I'm very pleased with my new segmented network setup, and the new machine, I'm extremely annoyed that I cannot get a couple of (graphical) Xen guest desktop guests up and running.

The initial idea was that I would setup a 64-bit installation of Etch and then communicate with it via VNC - xen-tools will do the necessary magic if you create your guest with "--role=gdm". Unfortunately it doesn't work.

When vncserver attempts to start upon an AMD64 host it dies with a segfault - meaning that I cannot create a scratch desktop environment to play with.

All of this works perfectly with a 32-bit guest, and that actually is pretty neat. It lets me create a fully virtualised, restorable, environment for working with flash/java/etc.

The bug was filed over three years ago as #276948, but there doesn't appear to be a solution.

Also, only on the amd64 guest, I'm seeing errors when I try to start X which mention things like "no such file or directory /dev/tty0". I've no idea whats going on there - though it could be a vt (virtual terminal) thing?.

The upshot of all this is that I currenly have fewer guests than I was expecting:

skx@gold:~/blog/data$ xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     3114     2 r-----   1180.6
cfmaster.services.xen                      1      256     1 -b----      1.0
etch32.desktop.xen                         2      256     1 -b----      1.4
etch32.security-build.xen                  3      128     1 -b----      1.4
etch64.security-build.xen                  4      128     1 -b----      1.4
sarge32.security-build.xen                 5      128     1 -b----      1.0

Xen Migration

This afternoon I mostly migrated Xen guests from their old host to their new. (As part of a an upgrade of facilities. Upgrading in place would have been much fiddlier and more annoying!)

The migration took almost three hours, which was longer than anticipated but shorter than I'd feared. In the future I'll know to do it differently, but I managed to script it fairly well after the first couple were done manually.

Everything appears to be working correctly so I will soon nip out for some high quality beer.

One thing that I wanted to do with the new host was track bandwidth usage upon a per-guest basis.

This should be possible with something like vnstat - however solutions counting traffic by interface name are not a good mesh with Xen - since by default a guest will have an interface with a name like 'vif20.0' - and no means of mapping that to a specific guest.

Each of my guests has been allocated three IPs which are defined like this in the Xen configuration file:

vif = [ 'ip=1.2.3.4 1.2.3.5 1.2.3.6' ]

This works prefectly.

This also works:

vif = [ 'ip=1.2.3.4,vifname=foo 1.2.3.5 1.2.3.6' ]

Unfortunately anything else I've tried to give each IP a static interface name fails. I've seen reports of this online but no solutions.

Given a configuration file like this the Xen guest doesn't receive any traffic upon the second + third address:

vif = [ 'ip=1.2.3.4,vifname=foo1',
        'ip=1.2.3.5,vifname=foo2',
        'ip=1.2.3.6,vifname=foo3' ]

Any suggestions welcome.