Steve Kemp's Blog

Since I'm I'm using real titles I guess I should make a real post, in which real things are mentioned. Unfortunately recently most of my time has been spent offline, doing things in and around Edinburgh.

However I have done a few things which are possibly worthy of mention. My Lenny repository has been updated a little:

The Gimp

There's a slightly newer version of The Gimp available now, corresponding to a recent upload to unstable.

gtk-gnutella

Once again I was forced to update the backported gtk-gnutella package, as my previous one was too old to connect to the network.

itag

Finally I added a Lenny package for the itag software which is now essentially complete.

Of those things I had a lot of fun with the itag software. Partly because it allows me to horde my images in a way that I appreciate, but also because it made me go over some older images and be pleasantly suprised.

My personal archive, ~/Images, is now just over 80Gb, and goes back about ten years. (Of course the older images were taken with random point and shoot digital cameras and each images is only a few hundred k in size. The newer images, saved at full-resolution, may be 5Mb each.)

Otherwise I've been slowly deploying OpenLDAP in anger, which has been educational. I've got a minor problem to solve which is that (posix)group definitions don't seem to be working reliably, but otherwise I've got Apache authenticating against groups, SSH logins working, and the little brother database using the LDAP server as an address book. (Mail clients? mutt is the one true mail client. notmuchmail.org will be interesting when further developed, but everything else I'm going to ignore with my stubborn Yorkshire nature ;)

ObQuote: "Oh no no no, dead broad OFF THE TABLE!", from Shrek.

Today I updated the package of mutt which is stored upon my apt-get repository - the Lenny repository now contains an updated copy of mutt & mutt-ng.

This package is synced from sid and contains the addition of a small patch to update the sidebar handling so that it is possible to show only folders with new mail (Before/After)

I've talked about this patch before, and the mutt sidebar generally, so I'll not repeat myself.

Instead I will share this simple mutt tip:

#
#  Specify which mails to show when changing folder:
#
folder-hook . push '<limit>((~N|~O)!~D)|(~d<1d!~Q)<enter>'

What does this do? When changing folder it limits the display of messages to those which match either pattern:

(The first pattern could be simplified but I like to be explicit and match "N"ew and "O"ld messages directly.)

I also have the following macros setup so I can type ".a" to view all messages in the current folder, ".t" to view only messages received today, ".n" to view only new messages, and ".y" to view all messages received yesterday:

macro index .n "l~N\n"
macro index .a "l~A\n"
macro index .t "l~d<1d\n"
macro index .y "l~d<2d ~d>1d\n"

ObFilm: Dead Like Me

Back a while I created an updated mutt package which builds upon the previous work with mutt-ng.

The package which I had constructed had two new features, well if you're generous. I guess there was only technically one new feature:

• The ability to cause the sidebar to only display folders with new mail in them.
• + Updated navigation to make next/prev work correctly with only new mail-containing folders.

I still hate quilt with a passion, but I've synced my patches with the current unstable Mutt - and there are now source & binary packages (x86 + amd64) in my apt-get repository for Etch. I can rebuild for Lenny if there is any interest. But it might take me a while to setup lenny chroots.

As a graphical example this is what you will end up with:

• The sidebar with all folders visible (the default).
• The sidebar listing only folders with unread messages.

OK I've tweaked the settings a little, but only a little:

# set up the sidebar, default to being visible
set sidebar_width=45
set sidebar_visible=yes
set sidebar_delim='|'
color sidebar_new red default

# ctrl-n, ctrl-p to select next, prev folder
# ctrl-o to open selected folder
bind index \CJ sidebar-scroll-down
bind index \CK sidebar-scroll-up
bind index \CP sidebar-prev
bind index \CN sidebar-next
bind pager \CO sidebar-open
bind index \CO sidebar-open

# Remap bounce-message function to "B"
bind index B bounce-message

#
#  Toggle visability of the sidebar.
#
macro index b '<enter-command>toggle sidebar_visible<enter><refresh>'
macro pager b '<enter-command>toggle sidebar_visible<enter><redraw-screen>'

#
#  Show folders with new mail only
#
macro index E '<enter-command>toggle sidebar_newmail_only<enter>'
macro pager E '<enter-command>toggle sidebar_newmail_only<enter>'

ObFilm: Joan of Arc

François Marier has recently been posting some interesting entries about the Mutt mail client upon his blog.

His tips are pretty basic, but that doesn't make them less useful. So here's my tip of the day: reply_regexp.

When you're viewing a mail, and you choose to reply it the subject of that mail is the basis of your message's subject. For example given a message with "Subject: hello" your reply will typically have the subject "Subject: Re: hello".

This is real rocket science here, people.

Imagine you're using SPAM filtering which tags messages it isn't sure about with a prefix "UNS:". Suddenly things don't look so hot, as you might end up with a mail with the subject:

Subject: UNS: Re: Hello

Reply to that mail, whilst being half-asleep, and what do you get? You get this:

Subject: Re: UNS: Re: Hello

Bad. Ugly. Wrong.

The following snippet in my ~/.muttrc file correctly deals with this case:

set reply_regexp="^(((UNS:[ \t])|[rR][eE]:[ \t])*)+"

Neat. Cool. Nice.

I've paid for some advertising upon the LWN.net site. (No link, I'm not trying to game things here.)

I didn't know what to expect, but I was willing to risk the expenditure as a way of saying thanks to them for their great content. (Because of my Debian Project membership I get a free LWN subscription, as do all developers. see here for details if you're a developer without a subscription.)

I've paid $30 and that has given me a months run of my advert, with clickthroughs hovering around 1%. Not horribly bad at all!

I probably won't repeat the experiment for the forseeable future, but I'm glad that I did it at least once.

If you have something appropriate for a Debian-based audience don't forget you're welcome to advertise upon the Debian Administration website for free - See the Advert FAQ for details.

ObQuote: The Craft

Update: jquery was accepted yesterday. Today I uploaded a new version to more closely align it with the javascript-policy.

So early this evening I looked for more bugs to fix in the Debian packages I use the most, instead of doing security work. (Because I'm waiting for buildds..)

Anyway I figured I'd take a peak at the mutt-patched package - since I have my own patched backport for Etch these days. I'd like to get into the habit of making sure it stays current, but honestly I'll probably forget in a few months.

One bug #464189 caught my eye. It is asking for a couple of patches, neither of which I'd heard of. One of them is obviously extremely useful though - it is designed to change the sidebar in such a way that it only displays mailboxes with unread messages in them.

I hunted high and low for the patch and had to admit defeat.

So I wrote my own, and now my backported package contains a suitable patch.

In other news I made a new release of the chronicle blog compiler, which I'm now using in a few more places.

All in all it has been a busy day and I got a fair amount of hacking done!

The other thing, that I can speak about, which I did today was package the Perl Lingua::Identify module so that my spam filtering service can offer users the opportunity to reject mails written in, say, Russian, or Italian.

I'm tempted to give them up to Sarah for adoption, as she was making noises at the weekend about joining Debian (then again so was Jacob Appelbaum). The only issue is that I think she should use the Debian Perl group to get involved, and I've not seen any sign of activity there from her since she first mentioned joining it. Sarah: Consider this a reminder!

Update: Feel free to nominate any *crash* bugs you'd like me to attempt to fix. Providing they're not in video or audio playback code! Could be a fun challenge..

ObQuote: Cruel Intentions.

One final post then I'm done discussing mutt. (Primarily because too many people failed to understand the problem, I guess I wasn't being as good at explaining as I could have been.)

So the goal was to be able to tag individual messages within mutt, such that a short while later a new folder would be created containing all messages of that tag, regardless of which folder the message(s) are in.

Actually finding the tags and creating the virtual folders is easy to do. I wrote a simple indexer which scans for tags in messages and creates hardlinks as necessary.

The problem with this is that many operations on those hardlinked messags would create a new copy of the message and operate on that - trashing the hardlink. For some operations that's just fine. But I did specifically want to be able to untag a tagged message and not have to hunt around for the original. (e.g. remove the "todo" tag from messages where I'd done the relevant action.)

So after a bit of trial and error I came up with a patch which allows the editing of a message in-place, in the hardlinked folder, without trashing the hardlink. This patch just invokes "$EDITINPLACE <filename>" where "filename" is the name actual file the message I'm working on is stored on disk. (i.e. ~/Maildir/.people.thewomanmeg/cur/1.2.3.txt).

By setting the EDITINPLACE variable to point to my ~/bin/editlabel script I may easily remove any tags from the hardlinked-message and have it apply to the original. Job done!

There are some limitations with this approach that are worth mentioning though:

• IMAP? Ha! Nope. We only work on Maildirs.
• The header-cache facility of Mutt breaks my "find the filename of the Maildir message" function. Not sure why, so I just disabled it.
• Many operations on the virtual mailbox will still break the symlink; because they don't edit in place.

I've documented things in a rather random fashion, and I've made a backported package of mutt with my patch available online too. Primarily so I dont lose the source like I did with my mutt-ng backport.

So, as I previously mentioned I want to be able to tag messages in Mutt.

There exist folder-based solutions already, using the X-Label header. There doesn't appear to be any existing solution allowing you to view all messages with a given tag across mailboxes.

So I wrote a simple shell script to create virtual mailboxes, such as ~/Maildir/tags-debian for all messages with a debian tag, using hardlinks.

My conclusion is that this solution will not work properly in practise, primarily because of deficiencies in mutt.

The simple case works just fine. I add a tag to a message, and later when the indexing job runs the virtual folder is created. I can open it and work on it just fine.

So where's the problem? Well in my case I tend to tag messages with a label such as "todo". Once I've done whatever I was supposed to I can remove the tag.

Using this hardlinking scheme I cannot remove the tag(s) in the virtual folder - I have to remove it in the original message which is a real pain.

Why? Well quite simply mutt will not let me work on my virtual message without destroying the hardlink.. If I use the edit function, for example, I am presented with a copy of the mail for editing - and the hardlink is replaced when that copy is saved.

Even the edit-label patch which allows you to edit the X-Label header from within mutt ends up replacing the hardlink with a new file!

So whats the solution? Well I guess I want to be able to run an external command against a message in mutt - passing the filename of the Maildir message as an argument. That way I can edit the live file.

Right now I don't believe that is possible, but I'd love to be told different.

If anybody has any solutions of editing, or even just deleting, a header from a message within mutt - in such a way that the hardlink isn't destroyed please do let me know.

Simple reproducer:

mkdir -p ~/Maildir/.foo/cur
mkdir    ~/Maildir/.foo/new
mkdir    ~/Maildir/.foo/tmp
cp 'validmessage' ~/Maildir/
ln validmessage ~/Maildir/.foo/cur

Now edit the message - start mutt open the message in the index and press 'e' - the hardlink is now gone. Replaced by a new file with the contents, so the original mail message is unchanged.

Update: I've got an "edit-inplace" primitive working, via the very hacky header-fu patch. It is not complete, but it demonstrates that it can be done. My world is now complete.

I've been re-reading RFC 2822 again over the weekend, for obvious reasons, and I'm amused I've not noticed this section in the past:

3.6.5. Informational fields

The informational fields are all optional. The "Keywords:" field contains a comma-separated list of one or more words or quoted-strings. The "Subject:" and "Comments:" fields are unstructured fields as defined in section 2.2.1, and therefore may contain text or folding white space.

subject  = "Subject:" unstructured CRLF

comments = "Comments:" unstructured CRLF

keywords = "Keywords:" phrase *("," phrase) CRLF

Now we all know that emails have subjects, but how many people have ever used the Keywords: header, or the Comments: one?

It'd be nice if we could use these fields in mails - I can immediately think of "keywords" as tags, and I'm sure I'm not alone.

I've looked at multiple "tags for mutt" systems, but all of them fall down for the same reason. I can add tags to a mail, and limit a folder to those mails that contain a given tag. But I cannot do that for multiple folders and that makes them useless :(

Has anybody worked on a multi-folder tag system for Mutt? If so pointers welcome. If not I'd be tempted to create one.

I guess implementation would be very simple. There are three caeses:

• Adding a tag
• Deleting a tag
• Finding all messages with agiven tag.

The first two are easy. The second could be done by writing a cronjob to scan messages for Keyword: headers, and writing a simple index. That could then be used to populate an "~/Maildir/.tag-results" folder, via hardlinks, of all matching messages.

Better yet you could pre-populate ~/Maildir/.tag-$foo containing each message with a given tag. Then theres no searching required! (Although your cronjob would need to run often enough when the tag were added to a message it would appear there within a reasonable timeframe.

Update: I've written the indexer now. It works pretty quickly after the initial run, and is quite neat! tagging messages with mutt.

Thanks to everybody who told me that there wasn’t a simple way to persuade Mutt to work with mailboxes containing unread messages.

I’ve now discovered the mutt-ng project, packaged in Debian’s experimental branch. This contains the infamous mutt sidebar patch which caters to my needs fully:

• I can toggle the display of the sidebar by pressing “b”.
• From there I can jump to the name of the next/previous mailbox with unread mail.
• Once on a mailbox name I can open it easily.

Job done.

Expect a more involved writeup shortly ..

Assume I’m using mutt for my mailclient and that I have all my mail stored beneath ~/Maildir.

Is there a simple way of limiting an index to only those folders containing New messages?

I know that I can use the index_format to display something like “N” next to folders containing new messages – but I cannot seem to limit the display to just those ones.

(At any given time I’ll have something like 30+ mailboxes with unread messages, if not more.)

My current mutt looks something like this, which shows that I have 24 folders with unread messages – but it doesnt make it easy to tell which they are:

..
269       4096 May 11 22:49   ~/Maildir/.spam/
270       4096 May 11 23:36   ~/Maildir/.spam.bayes/
271       4096 May 11 22:49 N ~/Maildir/.spam.image/
272       4096 May 11 22:49   ~/Maildir/.spam.pyzor/
273       4096 May 11 22:49   ~/Maildir/.spam.razor/
274       4096 May 11 22:49   ~/Maildir/.spam.request/
275       4096 May 11 23:27   ~/Maildir/.spam.unsure/
276       4096 May 11 22:54   ~/Maildir/.steve_org_uk/
277       4096 May 11 22:49   ~/Maildir/.syswear_com/
..
-- Mutt: Mailboxes [24]