Skip to content

My hovercraft is full of eels.

Recently I've been seeing an awful lot more bounced mail addressed to my domains, to the extent that I now wonder whether they are deliberate "attacks".

Over the past four or five years I'd expect to receive one joe-job attack every six months. Over the past two that's risen to once every two months. For the past two months its been once a week.

I run several domains on my Xen guest, and most of those domains rarely have mail received, so there are only a few localparts. (A "localpart" is the bit before the @ sign in an email address.)

My main domain is steve.org.uk and unfortunately this was historically setup with "catchall" behaviour. I used that wildcard expansion pretty seriously so I had localparts such as "slashdot.org", "lwn.net", etc. Over time I've stopped making up new addresses and just stuck with "steve".

Still I'd never quite gotten round to enumerating all valid localparts, instead I tried to mitigate against these rare bounce storms with various simple hacks. For example the following procmail recipe to file away bounces:

#  Bounces
#
:0:
*(Return-Path:).*(<>)
.Automated.bounces/

However this doesn't work as well as it used to - too many idiots people are using challenge/response systems so I'll receive a reply to a mail I didn't send which doesn't look like a bounce (ie. There is a real envelope sender.)

In short blocking bounces by detecting an empty envelope sender is not a complete strategy these days. I started down the heuristic path blocking mail to "unlikely" localparts via patterns such as:

[0-9]@        DENY  Localparts never end in digits
,             DENY  Localparts never contain a comma
|             DENY  Localparts never contain PIPES.
^([^a-zA-Z])  DENY  Localparts start with a-z/A-Z
"             DENY  Quotes are never used in accounts on this system:
'             DENY  Quotes are never used in accounts on this system:

That was actually a simple change to make, via the addition of a new QPSMTPD plugin and it managed to block a lot of the bounceback spam - regardless of the envelope sender. For example:

IP:84.45.254.18    sender:<> Recipient:treacherously9@steve.org.uk
IP:203.202.253.252 sender:<> Recipient:envoyz0@steve.org.uk

Blocking "unlikely" localparts wasn't perfect, but without implementing BATV or enumerating valid localparts there wasn't too much else that I could do. In terms of numbers yesterday I blocked just over 18,500 messages with these six rules.

I also wrote a couple of cronjobs to look at the contents of the Automated.bonces folder so that I could add per-user rejections on the specific addresses being received - with some whitelisting.

(For example if I received 20+ bounces to fluffy32qp@steve.org.uk within the space of ten minutes I'd drop further mails to that address automatically.)

Anyway enough is enough. Today I woke up to just over 40,000 replies to mails I didn't send. I've now scanned my mail directories for all the email addresses I've ever used and will now only accept mail destined to those localparts.

Thankfully it turned out that since 1999 (when steve.org.uk was registered) I've only used about 150 distinct localparts, and many of those are now obsolete. So hopefully I'll now have less of a problem.

It seems to be paying off already:

62.193.234.95   wpc0505.host7x24.com  <>  virtual_rcpt_ok
    901     mail to subtotalingxa@steve.org.uk not accepted here (#5.1.1)

65.99.223.234   cobra.compukey.net    <>  virtual_rcpt_ok
     901     mail to suctionsw@steve.org.uk not accepted here (#5.1.1)

207.44.156.81   box19.fuitadnet.com   <>   virtual_rcpt_ok
     901     mail to reappearcum@steve.org.uk not accepted here (#5.1.1)

In the future this means I could still get flooded with bounces, but there will be two outcomes:

  • The bounces will not hit valid localparts and will be dropped easily, quickly, and cheaply.
  • The bounces will hit valid localparts:
    • Real bounces will end up in Automated.bounces/
    • Challenge/Response things will still reach me. Sigh.

Still this is progress and I can steal some ideas from this great spam filtering service (ahem) to improve the handling of those! (I explicitly chose to use a similar but different system for my personal mails. Even though my support system is on another box I want to avoid problems where failures requiring human intervention are swallowed in the same way that the original one was. Those kind of reasons mandate a similar system but different implementation.)

I guess I could publish some of the qpsmtpd plugins I use locally virtual_rcpt_ok, virtual_badusers, rcpt_pattern_test, etc. Then again most people who do funky things with qpsmtpd will have plenty of choice already.

ObFilm: Monty Python's Flying Circus. (OK technically not a film. Sums up my mood though.)

Nobody touches the second shelf but me.

It seems the IMAP client crash I accidentally discovered in Thunderbird/Icedove was already known.

My report is a duplicate of a bug which was previously reported in 2007. Oops.

ObFilm: The Lost Boys

I'm gonna forget this conversation ever took place.

Recently I mentioned I'd been hacking about with a simple IMAP server.

Yesterday I was working on it some more, because the message store I've been testing against contains about 8 million messages and the damn thing is too slow.

During the course of some tweaking I discovered something interesting, every time a specific IMAP client connected to my server it crashed...

I spent a while fiddling around with backtraces and suchlike, but the upshot is I'm still not sure where the client crashes, but I've mailed some details to a few people to see if we can get it narrowed down.

I guess this counts as an accidental security issue. I wonder if I'll be able to collect a bounty? (Not that I'm bitter about past bounty-worthy reports being ignored ;)

Anyway interesting times, when I least expected them.

Mostly this post is being made to test a new release of the chronicle blog compiler - which now allows gravitars and has improved display of comments as demonstrated here.

ObFilm: Rambo First Blood Part II

I go down with one helluva bang.

Right now I have a lot of music, and I primarily interact with it via playlists.

I have a cronjob that generates, and populates, ~/Playlists/ every night. I generate playlists on multiple criterion:

  • ~/Playlists/Artist/
  • ~/Playlists/Albums/
  • ~/Playlists/Titles/
  • ~/Playlists/Keywords/

Playlists for specific artists & albums are probably self-explanatory, but the others might be interesting.

For every unique songtitle I have a playlist. In most cases that means there is a playlist called "Song Title" having one entry. But, as an explicit example, I have a playlist called "Under The Bridge" with two entries:

All Saints/Under The Bridge.mp3
Red Hot Chili Peppers/Under The Bridge.mp3

Similarly I break each song title into words, and generate one playlist for each distinct word discovered.

As a matter of randomness I have:

TermCount
Girl83
Boy31

(e.g. Songs containing "girl" in their title: "Madonna:Material Girl", "Amy Whitehouse:Hey Little Rich Girl", "Garbage:Stupid Girl"..)

There are times when I want something specific and my playlist approach doesn't work. For example "All songs which are 2 minutes long, and happy". I guess the problem is working out which meta-data is worth searching/storing, and then working out how to jump from that data to a playlist.

Today, whilst walking into town to buy some new pies, I wondered "How many songs do I have that end in a chuckle, or laughter?"

If I wanted an "ends in laughter" playlist right now I'm screwed. Yet no system I've ever seen allows you to add that level of detail. (To be honest I'd probably give up even entering it.)

In conclusion, my music collection is vast and various, and dealing with it is sometimes harder than I'd like.

How do you handle the music on your computer(s)? (When it comes to mobile-music I just use an ipod telling it to play all, randomly. If a song comes on I don't like I just skip it.)

ObFilm: Lolita

Is that the smallest you have?

Between 2002 and 2009 I was an active member of livejournal but recently I removed all my content, deleted my account, and then ceased using the service.

Unfortunately I had to un-delete the account after a week, as I suddenly realised that I'd used the URL skx.livejournal.com as an OpenID identifier.

(A few sites, such as stackoverflow, only use OpenID for handling accounts. So I just entered the livejournaly one without really thinking it through.)

Anyway the upshot of this is that I've hacked up a "quick" Perl OpenID handler, and I can now authenticate as www.steve.org.uk.

"Quick"? Well the thing about OpenID is that it is complex. More complex than you'd expect. (Common theme, here?)

I've now coded an OpenID consumer (again pretty livejournal-specific!) and an OpenID server - and both times the lack of documentation made the process annoying, fiddling, and more difficult than it should be:

So why something for me? Well if you're going to use OpenID then the whole premise is that you'll centralise. That means that you'll use the same identity everywhere - suddenly if your provider goes bust, gets bought out, or is compromised all the sites you use with it are vulnerable.

If I'm going to use OpenID I want the confidence that it is under my control. Completely. (Delegation does solve that to a certain extent; but not exclusively.)

Finally: No, Debian-Administration.org will not accept OpenID logins. Ask me to justify that sometime. In a pub. You're buying ...

ObFilm: Lolita

The doctors say you're going to live, that's the bad news.

It is annoying that some protocols and systems are more complex than you might expect them to be.

Jabber is a protocol that is notionally simple: XML Messages pass back and forth between server(s) and client(s). But if you look at the contents of XML which is passed around you'll soon discover that even logging in is a complex operation and that Jabber is not implemented in a pleasant fashion.

By contrast many other protocols are lovely. I'm sure I'm not alone in using and debugging many common protocols with nothing more than telnet. SMTP, HTTP, POP3, etc, are all pretty easy to drive interactively.

I think 90% of programmers at some point in their lives implement a HTTP server. But I draw the line at that kind of thing these days, client-side applications are useful and simple enough with the right libraries. (e.g. my sift client-side IMAP scripter has replaced procmail on a couple of machines. Watching to see if I get a reply from somebody specific and sending me an SMS on a match..)

But recently I've been flirting with the development of an IMAP server.

Dovecot appears to be the canonincal IMAP/POP3 server these days and it is pretty close to meeting my needs, but it isn't close enough unless I jump through and change the way my mailboxes are organised. (ie. The maildir mailboxes are arranged in such a fashion that dovecot cannot easily handle them, unless I mess about with symlink farms and make them all read-only.)

I guess in conclusion it would be nice if there were a basic IMAP server framework which you could just subclass "login" and "mailbox" sections and then instantiate.

I wrote a quick inetd-driven hack which supports only the bare essentials ("NOOP", "CAPABILITY", "LOGIN", "FETCH", "SELECT" and "LIST") That allows me to connect via IMAP in both mutt and thunderbird, view folders and download messages.

Still I'm strongly suspecting that there are better uses of my time, even if I could use it in several ways..

ObFilm: La Femme Nikita

That's really one of the saddest things I've ever heard.

Today I updated the package of mutt which is stored upon my apt-get repository - the Lenny repository now contains an updated copy of mutt & mutt-ng.

This package is synced from sid and contains the addition of a small patch to update the sidebar handling so that it is possible to show only folders with new mail (Before/After)

I've talked about this patch before, and the mutt sidebar generally, so I'll not repeat myself.

Instead I will share this simple mutt tip:

#
#  Specify which mails to show when changing folder:
#
folder-hook . push '<limit>((~N|~O)!~D)|(~d<1d!~Q)<enter>'

What does this do? When changing folder it limits the display of messages to those which match either pattern:

PatternMeaning
(~N|~O)!~D)

That is "New" or "Old" messages which haven't been deleted.

(~d<1d!~Q)

Messages received in the past day which haven't been replied to.

(The first pattern could be simplified but I like to be explicit and match "N"ew and "O"ld messages directly.)

I also have the following macros setup so I can type ".a" to view all messages in the current folder, ".t" to view only messages received today, ".n" to view only new messages, and ".y" to view all messages received yesterday:

macro index .n "l~N\n"
macro index .a "l~A\n"
macro index .t "l~d<1d\n"
macro index .y "l~d<2d ~d>1d\n"

ObFilm: Dead Like Me

What day is it? What year?

For the past few days Edinburgh has been experiencing a mini-heatwave. This is mostly pleasant, but the downside is that I've become sunburnt and bitten by many insects.

Happily I've discovered that the mint plants I've got growing beneath my bedroom window are antipruritic ("anti-itch") which solves both problems.

Aside from the heat I've had a busy weekend wiring up some Lisp code to serve as an XML-RPC server, so that I can carry out some functions remotely.

I'm almost tempted to package the s-xml-rpc library if only locally, but I'm not confident I know what I'm doing. I mostly unpacked the source beneath /usr/share/common-lisp/source/s-xml-rpc and fiddled until things started working by magic.

In addition to getting more hooked upon Lisp I've spent a while tidying up obsolete pages on my websites, unifying services, and giving a minor overhaul to the appearance of things.

ObFilm: Terminator Salvation

Everything is different, but the same.

I've successfully upgraded my primary web/mail/misc host from Debian Etch to Debian Lenny. There were a few minor problems, but on the whole the upgrade was as painless as I've come to expect.

In the past I'd edited my Exim4 configuration to add quite a few ACL checks, for example rejecting mails based upon spoofed/bogus HELO identifiers, and rejecting messages that didn't contain "Subject" or "Date" headers.

The Debian Exim4 configuration may be split into multiple files (which is how I prefer it on the whole). The idea that you just add new files into the existing hierarchy and they'll magically appear in the correct location when a real configuration file is generated. On the whole this works well, but sometimes editing files in-place is required, and it was these local edits that caused me pain.

Fixing things up was mostly not a challenge, primarily it was a matter of removing ACLs until exim4 started without errors - all my spam checking is handled ahead of exim4 these days, except for the last-ditch spam filtering with a combination of procmail-fu and the crm114 classifier package.

Taking a hint from Bubulle's weblog I decided to nuke my CRM114 spam database to avoid any possible version-mismatch issues so now I'm having to classify a lot of "unsure" messages. Happily my memory of doing this last time round is that the initial training of spam/ham takes a day or so to complete.

Anyway now I can start looking to advantage of the things new to Lenny. But probably not until I'm sure things have calmed down and upgraded correctly.

steve@skx:~$ uptime
 05:00:31 up 260 days, 14:23,  2 users,  load average: 0.95, 0.51, 0.31
steve@skx:~$ cat /etc/issue
Debian GNU/Linux 5.0 \n \l

ObFilm: Bill & Ted's Excellent Adventure

Don't you want to hear my excuse?

Etch -> Lenny

This Saturday I'll be upgrading my main box to lenny.

Mostly this should be painless, as the primary services aren't going to change too much.

I've tested the upgrade of the virtual hosting configuration which I use for exim4 on lenny and that works, as-is. I also have a local version of qpsmtpd which I'll be deploying and that works on lenny with my custom plugins.

A new version of Apache 2.x shouldn't cause any problem, although I will need to test each site I have to make sure that Perl module upgrades don't cause any breakage.

I expect random readers will neither notice nor care if my sites go down for an hour or two, but for local people consider this notice ;)

In other news I put together some javascript plugins for jquery recently:

dltoggle

This allows dl/dt/dd/definition lists to have their contents collapsed easily.

Currently I use some custom code to do that (e.g. as used here) but this jquery plugin is far neater, even if the plugin code isn't perhaps the best.

autoajax

This plugin converts plain links to things that make AJAX requests. In theory this allows graceful enhancements.

e.g. <a href="foo.html#bar">link</a> becomes an AJAX request that loads the contents of "foo.html" into the div with ID bar.

It seems this is a cheap clone of ajaxify, but I didn't know that when I put it together.

ObFilm: The Breakfast Club