Steve Kemp's Blog

Over the past couple of months the machine which hosts the Debian Administration website has been struggling with two distinct problems:

The dreaded scheduler bug/issue

The machine would frequently hang with the messages of the form:

Task xxx blocked for more than 120 seconds

This would usually require the application of raised elephants to recover from.

OOM-issues

The system would exhaust the generous 2Gb of memory it possessed, and start killing random tasks until the memory usage fell - at which point the server itself stopped functioning in a useful manner.

Hopefully these problems are now over:

• Bytemark have generously increased the memory installed upon this (donated) machine from 2Gb to 4Gb.
• I've cleaned up some of the site code and switched to using a combination of nginx & apache 2 to serve the site. (I've documented this upon the site itself : Speeding up dynamic websites via an nginx proxy.)

The combination of these two changes should resolve the memory issues, and I've installed a home-made 2.6.31.4 kernel which appears to have corrected the task-blocking scheduler issue.

ObTitle: Bridget Jones: The Edge of Reason

All being well the Debian Administration website now fully supports UTF-8.

This change was a long time coming, considering the amount of time the site has been live.

Most of the changes have been present for a while:

• Correctly setting the database to store UTF-8 internally, rather than latin1.
• Correctly setting the charset of the generated pages.

The only missing part was ensuring the at the text input by visitors/users was correctly decoded and treated as UTF-8. This was handled by updating changing the Perl CGI module to explicitly call charset appropriately.

Since the code behind the site masks the database, memcached, and CGI handles behind singletons the change itself was pretty trivial:

• context diff (full file)

I made more changes this evening to tie it all together, and to ensure that my Database connection is always forced to use UTF but I think that wasn't so important.

I hope this is vaguely useful the next time I have to fight with character sets & encodings. It is just all so nasty. Failing that these pages are vaguely useful:

• MySQL UTF settings
• UTF-8 samples

ObFilm: Run Lola Run

Over the past few nights I've managed to successfully migrate the Debian Administration website to the jQuery javascript library

This means that my own javascript library code has been removed, replaced, and improved!

The site itself doesn't use very much javascript - there are a couple of places where focus is set to a couple of elements, but other than that we're only talking about:

• Adding tags to articles, blog entrys, & polls
• Looking at the tag cloud
• Using the messaging system

Still there are a couple of enhancements that I've got planned which will make the site neater and more featureful for those users who've chosen to enable javascript in their browsers.

Here's my list of previous javascript usage - out of date now that I've basically chosen to use jQuery for everything.

ObQuote: Short Circuit.

Recently I have mostly been "behind". I've caught up a little on what I wanted to do though over the past couple of days, so I won't feel too bad.

I've:

made a new release of the chronicle blog compiler, after recieving more great feedback from MJ Ray.

un-stalled the Planet Debian.

updated the weblogs hosted by Debian Administration, after help and suggestions from Daniel Kahn Gillmor.

stripped, cleaned, and tested a new steam engine. Nearly dying in the process.

discovered a beautiful XSS attack against a popular social networking site, then exploited that en masse to collect hundreds of username/password pairs - all because the site admins said "Prove it" when I reported the hole. Decisions decisions .. what to do with the list...

released a couple of woefully late DSAs.

started learning British Sign Language.

Anyway I've been bad and not writing much recently on the Debian Administration site, partly because I'm just sick of the trolling comments that have been building up, and partly due to general lack of time. I know I should ignore them, and I guess by mentioning them here I've kinda already lost, but I find it hard to care when random folk are being snipy.

Still I've remembed that some people are just great to hear from. I know if I see mail from XX they will offer an incisive, valid, criticism or a fully tested and working patch. Sometimes both at the same time.

In conclusion I need my pending holiday in the worst way; and I must find time to write another letter...

ObQuote: Dungeons & Dragons

After mentioning the xml-resume-library package I was reminded that the English translation has been out of date for over a year.

With permission from the maintainer I've made a new upload which fixes this, and a couple of other bugs.

On a different topic it seems that many Debian-related websites are having their designs tweaked.

I'm not redesigning mine, but I'd love other people to have a go.

Here's hoping.

I made a new release of the Chronicle blog compiler the other day, which seems to be getting a suprising number of downloads from my apt repository.

The apt repository will be updated shortly to drop support for Sarge, since in practise I've not uploaded new things there for a while.

In other news I made some new code for the Debian Administration website! The site now has the notion of a "read-only" state. This state forbids new articles from being posted, new votes being cast, and new comments being posted.

The read-only state is mostly designed for emergencies, and for admin work upon the host system (such as when I'm tweaking the newly installed search engine).

In more coding news I've been updating the xen-shell a little recently, so it will shortly have the ability to checksum the filesystem of Xen guests - and later validate them. This isn't a great security feature because it assumes you trust dom0 - and more importantly to checksum files your guest must be shutdown.

However as a small feature I believe the suggestion was an interesting one.

Finally I've been thinking about system exploitation via temporary file abuse. There are a couple of cases that are common:

• Creation of an arbitrary (writeable) file upon a host.
• Creation of an arbitrary (non-writable) file upon a host.
• Truncation of an existing file upon a host.

Exploiting the first to go from user to root access is trivial. But how would you exploit the last two?

Denial Of Service attacks are trivial via the creation/truncation of /etc/nologin, /etc/shadow, (or even /boot/grub/menu.lst! But gaining privileges? I can't quite see how.

Comments welcome!

I was suprised to see Michal Čihař suggesting that memcached wasn't helping his site enough.

I've been a huge fan of Danga's Memcached for the past year or two. I'm using it very heavily upon the Debian Administration website, and also upon my new dating site.

There are, of course, issues with making sure you flush the cache when things are updated, but I've got a good pair of test suites for that now :)

With a nice singleton accessor I can use the code as easily as:

1
2
3
4
5
6
7
8
9
10
11

sub recent_users
{
    my $cache   = Singleton::Memcache->instance();
    my $results = $cache->get( "recent_users" );
    return( $results ) if ( $results );

    # fetch from database
    
    $cache->set( "recent_users", $results );
    return( $results );
}

Obvious once you've done it a few times, but right now I can feel the difference if I disable the cache upon either of the sites. Page loadtime just drops.

I guess the effectiveness depends upon the site you're using, and how often things can be usefully cached. I know that for my uses I tend to have large items which would be expensive to fetch which occur on every page (such as "currently online users", or "recent weblogs".) That probably means I benefit more than others, still I've become so enamoured of the project I feel the need to pimp it a little!

Only three things to say today:

Second anniversary

Today marks the second anniversary of the Debian Administration website.

(I think that this is the first public mention of my intention to setup the site. It was the earliest I could find anyway.)

Edinburgh Trivia

Doing my own small bit for Debconf I went for a walk on Christmas Day.

For two years I’ve lived in the area of Edinburgh known as Leith. (Famous for being a port, and being a port of call for hookers)

During that time I’ve taken buses, and walked, up and down Leith walk hundreds of times. Almost every time I do I notice a large chimney by The Edinburgh Playhouse Theatre.. The top 20/30ft of the chimney can be seen from peeking over the top of local buildings, assuming you’re not blocked by other buildings. And now?

Now I know what it sits upon.

My little adventure in getting to know the city I’ve been living in; I wanted to find the bottom of the chimney and see what it was attached to. So if anybody coming to Debconf next year gets massively lost and curious about a large chimney en route to my house .. I can tell them all about it.

It is the little things in life ..

Xen Shell

I’m not sure that there are many people using this, the freshmeat listing has a few subscribers but I’m never sure how well that translates. (Plus of course nobody rated the software so it might be they hate it?). Anyway there are almost certainly far fewer users of the shell than the main xen-tools package – but as of last night the shell now allows you to control more than one Xen instance.

This is quite a large change, since previously it explicitly only supported one user controlling one Xen instance. Expect a new release shortly once I’ve tested it more, and updated the documentation.