Steve Kemp's Blog

I've updated the blogspam site a little. Now:

• There are wordpress & drupal plugins available
• Theres a plugin to test against stopforumspam.com

Still has a sucky site / layout...

ObFilm: Ghostbusters II

Recently the Debian Administration website has been besieged by spammers submitting bogus comments.

I also run a couple of blogs which are constantly having random gibberish submitted to them - although it is less of a problem there because I approve them manually due to the offline nature of my blog, and the blog compiler I use.

Regardless I figured it was time to do something about it, in part because Chris Searle suggested that I should be able to query arbitrary text for spammyness due to my experience with fighting email spam. (As it turns out that is not so useful experience. Comments and emails are not the same. The same header-trickery you see in bogus email, false EHLOs, etc, are not useful concepts to carry accross.)

Anyway we were saying, I was going to do something? So I did. My solution? An XML-RPC server which you submit your comments to. This will return either:

The server itself is very simple, and it comes with a collection of simple plugins which do the job of testing different things. More plugins will almost certainly be added over time - so far I've seen the lotsaurls plugin capturing most of the SPAM, although the DNSRBL lookup is also working nicely having captured live spam without my touching it. Huzzah. & etc..

If there is any interest feel free to let me know, although as the server is open, and the client side usage is documented I imagine other people could run with it easily enough too - avoiding me becoming a single point of failure.

For reference my initial change to hook this up live on the site was only a few lines of code - although later I did need to make some additional changes to make it cope with failures, & etc.

ObFilm: Ghostbusters

Back a while I created an updated mutt package which builds upon the previous work with mutt-ng.

The package which I had constructed had two new features, well if you're generous. I guess there was only technically one new feature:

• The ability to cause the sidebar to only display folders with new mail in them.
• + Updated navigation to make next/prev work correctly with only new mail-containing folders.

I still hate quilt with a passion, but I've synced my patches with the current unstable Mutt - and there are now source & binary packages (x86 + amd64) in my apt-get repository for Etch. I can rebuild for Lenny if there is any interest. But it might take me a while to setup lenny chroots.

As a graphical example this is what you will end up with:

• The sidebar with all folders visible (the default).
• The sidebar listing only folders with unread messages.

OK I've tweaked the settings a little, but only a little:

# set up the sidebar, default to being visible
set sidebar_width=45
set sidebar_visible=yes
set sidebar_delim='|'
color sidebar_new red default

# ctrl-n, ctrl-p to select next, prev folder
# ctrl-o to open selected folder
bind index \CJ sidebar-scroll-down
bind index \CK sidebar-scroll-up
bind index \CP sidebar-prev
bind index \CN sidebar-next
bind pager \CO sidebar-open
bind index \CO sidebar-open

# Remap bounce-message function to "B"
bind index B bounce-message

#
#  Toggle visability of the sidebar.
#
macro index b '<enter-command>toggle sidebar_visible<enter><refresh>'
macro pager b '<enter-command>toggle sidebar_visible<enter><redraw-screen>'

#
#  Show folders with new mail only
#
macro index E '<enter-command>toggle sidebar_newmail_only<enter>'
macro pager E '<enter-command>toggle sidebar_newmail_only<enter>'

ObFilm: Joan of Arc

Since I've ranted a little recently lets do another public bugfix. The last few times people seemed to like them, and writing things down helps me keep track of where I am, what I'm doing, and how soon it will be "beer o'clock".

So I looked over the release critical bug list, looking for things that might be easy to fix.

One bug jumped out at me:

• gnomad2: segfault on start up / vorbis comment headers
• (Original report)

I installed the package:

skx@gold:~$ apt-get install gnomad2
..
The following NEW packages will be installed
  gnomad2 libmtp7 libnjb5 libtagc0
..

Once I copied an .mp3 file to my home directory, with the .ogg suffix I got a segfault on startup:

skx@gold:~$ cp /home/music/Audio/RedDwarf-back-to-reality.mp3 ~/foo.ogg
skx@gold:~$ gnomad2
..
LIBMTP_Get_First_Device: No Devices Attached
PDE device NULL.
TagLib: Ogg::File::packet() -- Could not find the requested packet.
TagLib: Vorbis::File::read() - Could not find the Vorbis comment header.
[segfault]
skx@gold:~$

So I downloaded the source ("apt-get source gnomad2"), and the dependencies for rebuilding ("apt-get build-dep gnomad2"). This allowed me to rebuild it locally:

skx@gold:/tmp/gnomad2-2.9.1$ ./configure --enable-debug && make
skx@gold:/tmp/gnomad2-2.9.1$ cd src/

And now it can be run under GDB.

skx@gold:/tmp/gnomad2-2.9.1/src$ gdb ./gnomad2
GNU gdb 6.8-debian
...
(gdb) run
Starting program: /tmp/gnomad2-2.9.1/src/gnomad2
...
LIBMTP_Get_First_Device: No Devices Attached
PDE device NULL.
[New Thread 0x41dcd950 (LWP 23593)]
TagLib: Ogg::File::packet() -- Could not find the requested packet.
TagLib: Vorbis::File::read() - Could not find the Vorbis comment header.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x41dcd950 (LWP 23593)]
0x00007fc89e0340b0 in taglib_tag_artist () from /usr/lib/libtag_c.so.0
(gdb)

Interestingly the crash comes from a library, libtag_c.so.0. So either:

• The bug is realy in the libtag module.
• The gnomad2 package doesn't handle a failure case it should.

Time to guess which that is. Charitably we'll assume any library that segfaults will be quickly fixed, because it will have more users than a single program..

Moving on we can look at the gnomad2 source for mentions of taglib. Several mentions are found, but src/tagfile.c is clearly the correct place to look. That file contains the following code:

void
get_tag_for_file (metadata_t *meta)
{
  gchar *tmppath = filename_fromutf8(meta->path);
  TagLib_Tag *tag;
  const TagLib_AudioProperties *properties;
  TagLib_File *file = taglib_file_new(tmppath);

  if(file == NULL) {
    g_printf("could not open file %s", tmppath);
    g_free(tmppath);
    return;
  }
  g_free(tmppath);

  tag = taglib_file_tag(file);
  properties = taglib_file_audioproperties(file);

  gchar* artist = taglib_tag_artist(tag);
  ..

This looks like a great place to explore because opening a file to read the tags is most likely where the crash is going to be coming from.

Interestingly we see the code:

• Calls taglib_file_new to get a handle of some kind relating to the file.
• Tests for errors with that operation.
• Calls taglib_file_tag to fetch tag information using the handle, and indirectly the file.
• But then uses taglib_tag_artist to fetch the artist (?I guess?) without testing the handle previously obtained was valid.

Let us guess that the file opening is succeeding but that the tag structure fetched via taglib_file_tag is NULL - and this causes the crash.

A quick edit:

  g_free(tmppath);

  tag = taglib_file_tag(file);
  if(tag == NULL) {
    g_printf("tag was null");
    return;
  }

  properties = taglib_file_audioproperties(file);
  gchar* artist = taglib_tag_artist(tag);

Rebuild, and the segfault is gone. We have a winner. Now we just need to file a patch...

ObFilm: 300

Previously I've posted a couple of running commentries describing how I've examined and gone about fixing a couple of bugs in Debian packages I use, enjoy, or have stumbled upon by accident.

Each of these commentries has resulted in a change or two to the affected software to make the bug vanish.

Fixing the bug is usually the hard part, but obviously it isn't the only thing that you need to do. While it is fun to have a locally fixed piece of software if you don't share it then the very next release will have the same bug again - and you'll spend your life doing nothing more than fixing the same bug again and again.

Generally the way that I report my fixes is by sending email to the Debian bug tracker - because I usually only try to fix bugs that I see reported already. Specificially I tend to only care about:

• Bugs in packages that I maintain.
• Bugs that affect me personally. I'm selfish.
• Bugs in packages I use, even if they don't affect me.
• Segfaults. Because segfaults and security issues often go hand in hand.

ObFilm: Ghostbusters 2

Russ Allbery recently commented that it is really nice to receive patches for trivial scripts posted online.

I agree.

More than once I've posted a trivial script and had it be improved by people, or later included elsewhere.

So in the spirit of sharing here is my latest toy script:

• make-whitelist

This is a trivial script which searches a Maildir hierarchy and outputs a list of each email address which you've ever sent mail to.

Why would you want that? In my case my (personal) spam filtering makes use of whitelisting, and the assumption is that if I've ever mailed you in the past then I want to see your replies, and you get a break.

These days my (personal) mail filtering has a couple of broad rules:

• If your mail is HTML it is junk. Unless I'm bored.
• If your mail is GPG signed/encrypted I will see it.
• If your mail address is on my whitelist then I want to see it.

After that then I see your message only if CRM119 decides I should.

#
# remove potentially spoofed header
#
:0 fhw
* ^X-whitelist:
| $FORMAIL -I "X-whitelist"

#
#  GPG-signed messages are OK and will be whitelisted
#
:0fW
* < 1024000
|/home/steve/bin/isgpged

:0e
| $FORMAIL -A "X-whitelist: yes" -A "X-GPG-Signed: Yes"

#
#  Get the sender of the message.
#
FROM=`formail -x From:| sed 's/^\([^@]*[ <]\)//' | sed 's/\([ >]\).*$//'`

#
# Add a whitelist tag if appropriate
#
:0 fhw
* !^X-whitelist: yes
* ? test -s $HOME/.procmail_whitelist
* ? echo $FROM| fgrep -qisf $HOME/.procmail_whitelist
| $FORMAIL -A "X-whitelist: yes" -A "X-Whitelist-Test: $FROM"

The net result of these tests is that I can now run the spam filter on non-whitelisted mails:

#
# Run CRM114 mailreaver
#
:0fw: .msgid.lock
* !^X-whitelist: yes
| /usr/bin/crm -u /home/steve/.crm /usr/share/crm114/mailreaver.crm

#
#  Spam.
#
:0:
* ^X-CRM114-Status: SPAM.*
* !^X-whitelist: yes
.CRM.Spam/

#
#  Unsure.
#
:0
* ^X-CRM114-Status:.*UNSURE
* !^X-whitelist: yes
.CRM.Unsure/

There is more to my setup than that, but that's the minimum you'd need to see.

Of course this is a reminder, once more, that the kind of filtering that you carry out for yourself is different from that that other people will do.

ObFilm: The NeverEnding Story

Is Debian slowly tearing itself apart, or am I just unduly dramatic?

ObFilm: Kill Bill (volume two)

So I finally got round to vote on the Debian Lenny Firmware issue.

I can't help thinking that most of the discussion so far has been a waste of time, on the grounds that developers either

• Care a lot about the subject and already have an opinion.
• Don't care and get bored by the discussion.

Either way the discussion has two, or more, sides talking past each other rather than any convergence upon a consensus. (Me? I'm in the middle. On the fence. Getting bored. Hitting "Delete thread" a lot. Who knows, maybe I could have my vote influenced, but right now I'm just not caring enough either way.)

Anyway enough on that subject. Tonight I have mostly been looking around to come up with a simple way to give an "overview" of mail traffic. Something like:

• Most prolific poster.
• Most popular thread.
• etc etc.

Individually these jobs are easy. Making it look pretty is hard. (Also I'd like to have archives of the lists in question, which are searchable.)

Seems like nothing out there does everything I want.

ObFilm: Ferris Bueller's Day Off.

"I've got chills
They're multiplying"

I guess technically I could have used that as a subject, but ugh.

If ever you're a bit shiverry, and a bit unwell, don't shave your head. It'll take three times as long, and you'll cut yourself.

ObQuote: Red Dwarf.