Yesterday I was forced to test my backup system in anger, on a large scale, for the first time in months.
A broken package upgrade meant that my anti-spam system lost the contents of all its MySQL databases.
That was a little traumatic, to say the least. But happily I have a good scheme of backups in place, and only a single MX machine was affected.
So, whilst there was approximately an hour of downtime on the primary MX the service as a whole continued to run, and the secondary (+ trial tertiary) MX machines managed to handle the load between them.
I'm almost pleased I had to suffer this downtime, because it did convince me that my split-architecture is stable - and that the loss of the primary MX machine isn't a catastrophic failure.
The main reason for panicing was that I was late for a night in the pub. Thankfully the people I were due to meet believe in flexible approaches to start times - something I personally don't really believe in.
Anyway the mail service is running well, and I've setup "instant activation now", combined with a full month of free service which is helping attract more users.
Apart from that I've continued my plan of migrating away from Xen, and toward KVM. That is going well.
I've got a few guests up and running, and I'm impressed at how stable, fast, and simple the whole process is. :)
ObQuote: Brief Encounter
(That is a great film; and a true classic. Recommended.)
