Steve Kemp's Blog

I can be childish too: PHP must die

I’ve been actively migrating services away from PHP for a while now, even at the cost of having to use less mature applications which aren’t as pretty. I’m definitely at the point now where I simply don’t trust it and the applications which (ab)use it.

Tonights job is to look for a non-PHP blogging application which can import my wordpress content from. I even wrote 75% of one myself once upon a time, but I lacked the time to finish it off.

Sure there are good PHP coders out there, I’m just sick of the rest.

Somebody should write a intepretted templating system similar to PHP, but with no bugs. Although maybe thats the problem, giving people the ablity to mix presentation inline with code was a fundamentally bad idea to start with? Since it led to non-programmers producing things that were broken.

I do keep meaning to look at Rails, but I’m usually turned off by the problems of mixing Debian packages with the rubyforge gems – I like to keep things from one source. (e.g. 99% of my perl modules are Debian packages, not CPAN packages.)

Inspired by Joey's wiki compiler I've been toying with a blog compiler.

Very similar idea - you give it a directory of text files, and it creates a static blog complete with tagging support, RSS feeds, and all that good stuff.

Feel free to have a look - probably the demo is the most interesting bit.

The only obvious downside is that people cannot easily leave comments... However that might be a plus for some people, especially those that don't want to touch MySQL / PHP / etc

This week has consisted of fighting registrars and doing a bit of hacking on xen-shell, xen-tools, and the chronicle blog compiler.

CJ has done some good work trying to get the code modularised, and I expect between the pair of us we can make things neater and better generally.

I've also fixed a couple of bugs relating to the hard-wiring of device names (/dev/sda, /dev/tty1, etc). These devices are replaced in newer versions of Xen which wants to use /dev/xvc0 and /dev/xvd[a-z] instead.

There's nothing else happening at the moment; I'm just having a lot of fun laughing at our new kitten sliding around on our polished wooden floor!

Chronicle seems to be getting pretty popular which is ironic because it was a quick hack to allow me to post blog entries on a couple of hosted sites - which I've not yet done. Oops.

In other news I'm loving the Nintendo DS at the moment, Megan brought me one back from America on her recent trip and I think a day hasn't passed where one or both of us has played less than 30 minutes each.

I'm annoyed that Sim City DS only allows us to play with one city - right now it is her turn, and I'll have to wait until she's finished with her creation before I can have a go - because otherwise I'll wipe her city out.. :(

Today mostly consisted of a new release of the chronicle blog compiler. Interestingly this received several random mails today. I wonder what caused that all of a sudden?

The release of the compiler is timely, as it reminds me I've still not managed to find a decent gallery compiler. Although the thought of writing my own, rightly, fills me with depression.

I've been interested in reading more about both Git and SELinux upon Planet Coker Debian recently. I've switched several small projects over to GIT but I've not yet listed them publically. First of all I would like to see if there is a version of trac that I can install which supports git repositories. I guess that's a job to research tomorrow.

I wonder if I would confuse people by hosting GIT projects upon cvsrepository.org? ;)

Felipe Sateler kindly made a Debian package for the chronicle blog compiler, so you can now get it from my apt-get repository.

He suggested it be uploaded to Debian sid, I'm happy to do so if there is any interest. Otherwise I'll keep placing release there when they occur.

(To be honest I don't anticipate any major development unless there are bugs, or people would like to contribute themes ..)

I made a new release of the Chronicle blog compiler the other day, which seems to be getting a suprising number of downloads from my apt repository.

The apt repository will be updated shortly to drop support for Sarge, since in practise I've not uploaded new things there for a while.

In other news I made some new code for the Debian Administration website! The site now has the notion of a "read-only" state. This state forbids new articles from being posted, new votes being cast, and new comments being posted.

The read-only state is mostly designed for emergencies, and for admin work upon the host system (such as when I'm tweaking the newly installed search engine).

In more coding news I've been updating the xen-shell a little recently, so it will shortly have the ability to checksum the filesystem of Xen guests - and later validate them. This isn't a great security feature because it assumes you trust dom0 - and more importantly to checksum files your guest must be shutdown.

However as a small feature I believe the suggestion was an interesting one.

Finally I've been thinking about system exploitation via temporary file abuse. There are a couple of cases that are common:

• Creation of an arbitrary (writeable) file upon a host.
• Creation of an arbitrary (non-writable) file upon a host.
• Truncation of an existing file upon a host.

Exploiting the first to go from user to root access is trivial. But how would you exploit the last two?

Denial Of Service attacks are trivial via the creation/truncation of /etc/nologin, /etc/shadow, (or even /boot/grub/menu.lst! But gaining privileges? I can't quite see how.

Comments welcome!

I'm still in the middle of a quandry with regards to revision control.

90% of my open code is hosted via CVS at a central site.

I wish to migrate away from CVS in the very near future, and having ummed and ahhed for a while I've picked murcurial as my system of choice. There is extensive documentation, and it does everything I believe I need.

The close-runner was git, but on balance I've decided to choose mercurial as it wins in a few respects.

Now the plan. I have two options:

• Leave each project in one central site.
• Migrate project $foo to its own location.

e.g. My xen-tools could be hosted at mercurial.xen-tools.org, my blog compiler could live at mercurial.steve.org.uk.

Alternatively I could just leave the one site in place, ignoring the fact that the domain name is now inappropriate.

The problem? I can't decide which approach to go for. Both have plusses and minuses.

Suggestions or rationales welcome - but no holy wars on why any particular revision control system is best...

I guess ultimately it matters little, and short of mass-editing links its 50/50.

So I run a blog. You might have read bits of it in passing, probably more due to syndication than actual desire.

Initially I started off using Wordpress, but that was later purged once I bit the bullet and decided I'd rather be blogless than install PHP upon any server I cared about.

To replace my blog I looked for different solutions and eventually decided to try something built upon Ruby (on Rails). That lead to me trying Typo. After a little bit of pain I decided this wasn't a great solution.

So in the hunt for something new I moved to Mephisto, but over the past few months I've noticed that my virtual machine has been running low on memory - and the culprit has always been the Mongrel isntance which was running the blog.

So once more into the breach.

This time I had a new plan. I'd actually use a piece of blog software which I wrote for a couple more sites, a static system which outputs RSS & HTML files - with no dynamic-fu at all. That system is the chronicle blog compiler.

In performing the migration I accidentally flooded a couple of Planet-Planet installations. It took me a moment to work out why, and that was because my RSS feeds didn't include valid "pubDate" files for the entries..

So now I've made a new release of chronicle which supports valid RSS feeds, as tested against the feed validator and all is well. Well almost.

Users no longer have access to post comments, and whilst I've successfully migrated 700+ entries through to four different blogging engines I've managed to lost most (all) comments along the way at different points in time which is a real shame.

In the future I'll setup a simple CGI system to write comments to text files, and then allow them to rsync'd and merged into the entries. I just don't have the concentration to manage that right now.

In the meantime hello to Planet Sysadmin - never heard of you before, but thanks for carrying my feed. I'm suprised I qualify, but I guess at the very least I'm spreading interesting lyrics...

(Remind me to post the mod_rewrite rules I'm using now; I've kept all links at the same location through successive blog migrations which means I have numerous mod_rewrite rules which are probably obsolete, but I just don't want to touch them because cool URIs don't change....)

After a lot of hacking I've now got chronicle displaying comments upon entries.

Since my blog is compiled on my home desktop machine and comment submission happens upon a remote machine the process involves a bit of a hack:

Publish Blog

The blog is compiled and uploaded to the live location using rsync.

Wait For Comments

Once the blog is live there are embedded forms which may be used to receive comments.

The CGI script which is the target of the forms will then write each comment out to a text file, located outside the HTTP-root.

Sync Them

Prior to rebuilding the blog the next time I update I rsync the comments directory to my local machine - such that the comments posted are included in the output

This my local tree looks something like this:

~/blog/
|-- comments/
|-- data/
|-- output/
|-- Makefile
`-- chroniclerc

Here I have a Makefile to automate the import of the comments from the live site to the local comments/ directory, rebuild, and finally upload.

All this means that I can rebuild a blog created by a combination of plain text post files and plain text comment files.

It also means that there is a fair lag between comment submission and publication - though I guess there is nothing stopping me from auto-rebuilding and syncing every hour or two via cron...

I'll make a new release with this comment.cgi script and changes once the package hits Debian unstable...

Well I've been in Devon for the past week, and will remain here for another week. During that time I've averaged about ten minutes of online time a day!

So far things are going well, but it will be wierd spending time with another family over Christmas.

Still I've been vaguely productive. I've released a new version of chronicle - which has the CGI support for leaving comments upon my entries.

(TODO: Reinstate some previous comments for Martin Krafft)

One thing I'd not anticipated was the amount of spam I'd start recieving. The peak so far was 40+ comments a day, all with random URLs. I deleted them manually for the first day - but now I've written a shell script or two to classify comments via the spamfilter I use for handling email.

I'm not 100% sure I should have done that. I suspect that over time I will find better results if I actually have a distinct "blog spam" and "blog ham" corpus - rather than risk confusion over "email" and "blog" texts to classify. Still I shall wait and see.

The only thing that I think I want to do now is add a bit more fine control over the comment process, so the choice is not just between comments being globally on, or globally off. A nice middle-ground where I could say "Comments enabled for entries posted within the past two weeks or so".

Anyway thats my quota for today. Now I have to walk the family dog ..

Been a week since I posted. I've not done much, though I did complete most of the migration of my planet-searching code to gluck.debian.org.

This is now logging to a local SQLite database, and available online.

I've updated the blog software so that I can restrict comments to posts made within the past N days - which has helped with spam.

My other comment-spam system is the use of the crm114 mail filter. I have a separate database now for comments (distinct from that I use for email), and after a training on previous comments all is good.

Other than being a little busy over the past week life is good. Especially when I got to tell a recruitment agent that I didn't consider London to be within "Edinburgh & Surrounding Region". Muppets.

The biggest downside of the week was "discovering" a security problem in Java, which had been reported in 2003 and is still unfixed. Grr. (CVE-2003-1156 for those playing along at home).

Heres the code:

#!/bin/sh
#
#  Grep for potentially unsafe /tmp usage in shared libraries
#


find /lib -name '*.so' -type f -print > /tmp/$$
find /usr/lib -name '*.so' -type f -print >> /tmp/$$
for i in $(cat /tmp/$$ ); do
    out=$(strings $i | grep '/tmp')
    if [ ! -z "$out" ]; then
        echo "$i"
        echo "$out"
    fi
done
rm /tmp/$$

Recently I bought a new desktop PC and rotated my existing machines around, leaving me with one spare. Last night I donated that PC to a local friend, (co-incidentally the same woman who had been out cat-sitter when I went to stay at Megan's parents for Christmas).

I said "Its got Debian on it, that funny operating system that you used when you were looking after our kitten, remember?".

She said "Linux ssems to be getting popular these days I might as well try it out".

But seriously she used it. It had GNOME, it had firefox, she seemed happy. It isn't rocket science these days to point a technicalish user at a Debian Desktop and expect them to know what to do with it. The only minor complication was the lack of flash/java since it was etch/amd64 - but thats probably a plus with the amount of flash adverts!

In other news I made a new release of asql last night. This now copes with both Apache's common and combined logformats by default. (Yet another tool of mine which is now being used at work, which motivated the change.)

I've also started pushing commits to xen-tools last night, so there'll be a new release soon. Mostly I'm bored with that code though, it doesn't need significant updates to work any longer. All the interesting things have been done, so it's probably only a matter of time until I drift off. (To the extent that I've unsubscribed myself from xen-users & xen-devel mailing lists)

Virtualisation is increasingly a commodity these days. If xen dies people probably won't care, there are enough other projects out there doing similar things. Being tied to one particular platform will probably come to be regarded as a mistake.

(Talking of which I should try out this lguest thing; I just find it hard to be motivated...)

Today I spent a while fixing some more segfault bugs. I guess that this work qualifies as either fixing RC bugs, or potential security bugs.

Anyway I did an NMU of libpam-tmpdir a while back to fix all but one of the open bugs against it.

I provided a patch for #461625 yelp: segfault while loading info documentation, which fixes the symptoms of bad info-parsing, and avoids the segfault.

I also looked into the #466771 busybox cpio: double free or corruption during cpio extraction of hardlinks - but it turns out that was already fixed in Sid.

Finally I found a segfault bug open against ftp:

• ftp: GET segfaults. Buffer overflow?
• Buffer overflow: comman buffer size is not checked (Almost certainly a duplication bug.)

To reproduce this bug run:

skx@gold:~$ ftp ftp.debian.org
220 saens.debian.org FTP server (vsftpd)
Name (ftp.debian.org:skx): anonymous
331 Please specify the password
Password: foo@bar.com
ftp> cd debian/doc
250 Directory successfully changed.
ftp> get dedication-2.2.cn.txt dedication-2.2.de.txt dedication-2.2.es.txt ..
local: dedication-2.2.de.txt remote: dedication-2.2.cn.txt
Segmentation fault

You need to repeat the arguments about 50 times. But keep adding more and more copies of the three files to the line until you get the crash.

It isn't interesting as a security issue as it is client side only; but as a trivially reproducable issue it becomes fun to solve.

I mailed the maintainer of FTP and said unless I heard differently I'd NMU and cleanup the package in a week.

All being well this entry will be nicely truncated in the RSS feeds as support for the <cut> tag was the main new feature in my previous upload of chronicle - the blog compiler I use/wrote/maintain.

ObQuote: Razor Blade Smile

I've made a new release of the chronicle blog compiler, primarily to allow the "Subject:" header to be used for new blog subjects.

(That allows new entries to be automatically posted via email, with an appropriate procmail setup. I'll add one as an example shortly.)

Whilst on the subject of RSS creation (huh?) I've written a tiny utility which will create an RSS feed from a list of text files. It will also create an index.html file to match.

To see why this is useful you could view my recent changelog.

I think there is a need for a small tool to read files and create feeds from them - like mod_index_rss does, but without messing with Apache.

If there is any interest I'd be happy to release the code, as-is it doesn't use a template..

Online privacy is important. Mostly when this is discussed it is in the context of client-side anonymity.

Looking at it from the other side, though, How do you host a website anonymously?

You could register the domain via a proxy, or with bogus details. But if you host the site yourself the IP address may be traced to the hosting provider, and that may be used to trace back to you.

So, the alternatives? Well you could use a hosted site such as livejournal / wordpress / googlepages / etc. But pretty surely they'll be able to trace content back to you - and if you don't host it there's a high chance they'll just pull it if you talk about "bad things". (I guess you could use TOR for uploading / your connections there.)

So, going back to the question. How can you host something, easily accessible to the world, without risk of your identity/association being discovered?

---

I'm, obviously, ignoring FreeNet. Two reasons for that:

• It's slow, has no search-engine goodness, and is unproven.
• It requires an atypical client. Aunt Milly won't be able to surf Freenet...

I almost think the best way forward would be to write a site which was a proxy for a file-sharing protocol, then link people to items that way. Relying on the swarm to host the files..

The downside is that you'd have to have a convincing argument for when RIAA comes calling, suggesting that you're sharing their stuff too. If it wasn't a general purpose proxy then the deniability is gone, and if it is you're at risk of general copyright infringement claims.

Hard problem. Shame.

ObQuote: HellRaiser

I've made a new release of the chronicle blog compiler.

There are a couple of minor changes to the variables exported to the theme templates, as contributed by MJ Ray, and a new spooling system.

This works in a simple fashion, and allows you to queue up posts. For example If you write a new entry containing the psuedo-header "Publish: 20th April 2008" and you have a crontab entry containing this:

  chronicle-spooler
    --spool-dir=~/blog/spool/  \
    --live-dir=~/blog/data/  \
    --post-move='cd ~/blog && make upload'

It works as expected. When you call this on the 20th April the file will be moved from ~/blog/spool into ~/blog/data, and your blog will be rebuilt & uploaded.

The implementation was different than the original suggestion, but is nice and clean, and can be used to do other things with a little bit of effort.

Anyway if you see this entry the spooling system is working!

ObQuote: 30 Days of Night.

My blog compiler received a bit of love recently, primarily because MJ Ray wanted to use it.

As mentioned before I've added a simple spooling system, and the mercurial repository now contains a simple RSS importer.

In other news I've been working on various Debian packages, here is a brief summery:

bash-completion

After seeing a RFH bug I closed a few bash-completion bugs, and submitted patches for a couple more.

I was intending to do more, but I'm still waiting for the package code to be uploaded to the the alioth project.

javascript work

I've updated the jquery package I uploaded to follow the new "Javascript standard" - in quotes only because it is both minimal and new.

Once the alioth project has been configured I'll upload my sources.

Apache2

I've agreed to work on a couple of SSL-related bugs in the Apache 2.x package(s) - time disappeared but I hope to get that done this weekend.

Initially that was because I was hoping I could trade a little love for getting a minor patch applied to mod_vhost_alias - instead I've now copied that module into libapache2-mod-vhost-bytemark and we'll maintain our own external module.

I've been loaned a Nokia 770 which is very nice. Having used it with vim, ssh & etc I think that I'd rather have a device with a real keyboard.

The Nokia 810 looks pretty ideal for me. I'm going to be asking around to see if I can get a donated/loaned device to play with for a while before I take the plunge and pay for one of my own.

I've got a couple more things on the go at the moment, but mostly being outdoors is more interesting to me than the alternative. Hence the downturn in writing and releasing security advisories.

I'll pick things up more fully over the coming weeks I'm sure.

ObQuote: Shaun of the Dead

I made an emergency release of the chronicle blog compiler yesterday, after noticing that it was truncating titles containing periods.

That was a bit of a mea-culpea moment, but I guess mistakes happen.

The new release is in perfect shape for Lenny, and now includes two new scripts installed into the examples/ directory:

• The RSS-feed importer I mentioned prviously.
• A quick hack to detect duplicate post/entry titles.

The latter was applied to my own blog, and I discovered several duplicates. I guess my film quotes having only a limited source collection to work from could also include duplicates - so I've updated my Makefile to only build and rysnc my blog if there are none detected.

(In many ways that films site is the precursor to this blog; it uses a collection of text files, one per film, and generates a cross-linked HTML output of film entries. Sadly it is out of date, because entering titles is a real pain..)

I'm pleased with the comment process now though, the CGI comment submission script simply archives each submitted comment into a "comments/" directory on the webserver.

There a cron-job passes each one through a bayasian filter and moves the file(s) to either "comments/good/", "comments/bad/" or "comments/unsure/".

When I come to rebuild the blog I rsync the "comments/good" directory to my local machine, rebuild and then rsync the output back to my remote webserver.

(On a single machine this would be much simpler process!)

I've imported my blog source into a mercurial repository, so the client-side is consistent. I have a bad habit of making new postings from wherever I happen to be and having a central repository will make that less prone to diaster.

Just running "make steve" against the Makefile is sufficient to rebuild everything and sync it to my live system.

ObQuote: Kalifornia

Recently I have mostly been "behind". I've caught up a little on what I wanted to do though over the past couple of days, so I won't feel too bad.

I've:

made a new release of the chronicle blog compiler, after recieving more great feedback from MJ Ray.

un-stalled the Planet Debian.

updated the weblogs hosted by Debian Administration, after help and suggestions from Daniel Kahn Gillmor.

stripped, cleaned, and tested a new steam engine. Nearly dying in the process.

discovered a beautiful XSS attack against a popular social networking site, then exploited that en masse to collect hundreds of username/password pairs - all because the site admins said "Prove it" when I reported the hole. Decisions decisions .. what to do with the list...

released a couple of woefully late DSAs.

started learning British Sign Language.

Anyway I've been bad and not writing much recently on the Debian Administration site, partly because I'm just sick of the trolling comments that have been building up, and partly due to general lack of time. I know I should ignore them, and I guess by mentioning them here I've kinda already lost, but I find it hard to care when random folk are being snipy.

Still I've remembed that some people are just great to hear from. I know if I see mail from XX they will offer an incisive, valid, criticism or a fully tested and working patch. Sometimes both at the same time.

In conclusion I need my pending holiday in the worst way; and I must find time to write another letter...

ObQuote: Dungeons & Dragons