Steve Kemp's Blog

Via Axel Beckert I learned today that GNU Screen is 25 years old, and although development is slow it has not ceased.

Back in 2008 I started to post about some annoyances with GNU Screen. At the time I posted a simple patch to implement the unbindall primitive. I posted some other patches and fixed a couple of bugs, but although there was some positive feedback initially over time that ceased completely. Regretably I didn't have the feeling there was the need to maintain a fork properly, so I quietly sighed, cried, and ceased.

In 2009 my code was moved upstream into the GNU Screen repository (+documentation update).

We're now in 2012. It looks like there might be a stable release of GNU Screen in the near future, which makes my code live "for real", but in the meantime the recent snapshot upload to Debian Experimental makes it available to the brave.

2008 - 2012. Four years to make my change visible to end-users. If I didn't use screen every day, and still have my own local version, I'd have forgotten about that entirely.

Still I guess this makes today a happy day!

Wheee!

ObQuote: "Thanks. For a while there I thought you were keeping it a secret. " - Escape To Victory

This evening I sat down and migrated my personal virtual machine from a 32-bit installation of Debian GNU/Linux to a 64-bit installation.

I've been meaning to make this change for a good few months, but it took me until this evening until I decided it was as good a time as any.

Mostly the process is painless:

• Ensure you have a 64-bit kernel, with support for 32-bit binaries too.
• Install the 32-bit compatibility libraries, such that your old binaries work.
• Overwrite your binaries and libraries in-place so you have a 64-bit base system.
• Patch it up afterwards.

I overwrote a lot of the libraries and binaries on the system such that I had a working 64-bit apt-get, dpkg, sash, etc, and associated libraries. Then once I had that I could use those tools to pull the resto of the system up to date.

One thing I hadn't counted on is that I needed to have a 64-bit version of bzip such that "apt-get update" didn't complain about errors. I suspect I could have fixed that by re-configuring my system to disable compression. Still it was easily solved.

Along the way I also shot myself in the foot by having a local caching DNS resolver, listening on 127.0.0.1, which broke. With no DNS I couldn't use apt-get - but once the problem was identified it was trivial to fix.

Anyway all seems OK now. My websites are up, email is flowing and I guess anything else can wait until the morning.

ObQuote: "Somebody's coming up. Somebody serious." - Leon

I'd like to setup some auto-builders for some projects - and theese projects must be built upon Lenny, Squeeze, Lucid, and multiple other distros. (i386 and amd64 obviously.)

Looking around I figure it should be simple. There are a lot of continuous integration tools out there - but when looking at them in depth it seems like they all work in temporary directories and are a little different to how I'd expect them to be.

Ultimately I want to point a tool at a repository (mercurial), and receive a status report and a bunch of .deb packages for a number of distributions.

The alternative seems to be to write a simple queue submission system, then for each job popped from the queue run:

• Creates a new debootstrap-based chroot.
• Installs build-essential, mercurial, etc.
• Fetches the shource.
• Runs make.
• Copies the files produced in ./binary-out/ to a safe location.
• Cleans up.

Surely this wheel must already exist? I guess its a given that we have to find build-dependencies, and that we cannot just run "pbuilder *.dsc" - as the dsc doesn't exist in advance. We really need to run "make dependencies test build", or similar.

Hudson looked promising, but it builds things into /var/lib/hudson, and doesn't seem to support the use of either chroots or schroots.

ObQuote: "I feel like I should get you another sweater." - "Friends"

Today I resigned from the Debian project. The following packages are up for adoption:

• asql
• chronicle
• rinse
  • Thomas Lange is going to take this one by the looks of things.

I'll remove myself from Planet Debian tomorrow, assuming the keyring revokation isn't swift and merciless.

ObQuote: This space is intentionally blank.

Rather than waiting for a few months, as I typically do, I decided to be brave and upgrade my main virtual machine from Lenny to Squeeze. That host runs QPSMTPD, Apache, thttpd, and my blogspam server; nothing too complex or atypical.

The upgrade was mostly painless; I was interrupted several times by debconf asking me if I wished to replace configuration files I'd modified, but otherwise there were only two significant messages in the process:

crm114

crm114 warned me that its spam database and/or configuration files had changed and would most likely result in brokenness, post-upgrade, and I should do something to stop avoiding lost mail.

Happily this was expected.

sysv-rc

It transpired I had a couple of local init scripts which didn't have dependency information succesfully encoded into them; so I couldn't migrate to dependency-based bootup.

Given that this server gets a reboot maybe once every six months that wasn't really worth telling me about; but nevermind. No harm done.

That aside there were no major surprises; all services seemed to start normally and my use of locally-compiled backports meant that custom services largely upgraded in a clean fashion. The only exception was my patched copy of mutt which was replaced unexpectedly. That meant my lovely mutt-sidebar was horribly full of mailboxes, rather than showing only new messages. I created a hasty backported mutt package for Squeeze and made it available. (This patch a) enables the side-bar, and b) allows you to toggle between the display of all mailboxes and those with only new mail in them. It is buggy if you're using IMAP; but works for me. I would not choose to live without it.)

Now that I've had a quick scan over the machine the only other significant change was an upgrade of the mercurial revision control system, the updated templates broke my custom look & feel and also required some Apache mod_rewrite updates to allow simple clones via HTTP. (e.g. "hg clone http://asql.repository.steve.org.uk/").

So in conclusion:

• The upgrade from Lenny to Squeeze (i386) worked well.
• Before you begin running "iptables -I INPUT -p tcp --dport 25 -j REJECT" will avoid some potential surprises
  • There are probably other services worth neutering, but I tend to only do this for SMTP.
• Keeping notes of updated template files will be useful if you make such system-wide changes. (e.g. hgwebdir templates)

ObQuote - "Hmm, upgrades " - The Matrix Reloaded (shudder).

This week in brief:

I've rejoined the Debian Security Team

My first (recent) DSA was released earlier today, with assistance from various team members. (My memory of the process was poor, and some things have changed in my absence.)

BlogSpam gains a new user

The BlogSpam API is now available for users of Trac.

Finally, before I go, I've noticed several people on Planet Debian report their photo-challenges; either a picture a day or one a week. I too take pictures, and I'm happy if I get one session a month.

I suspect some of my content might be a bit too racy for publication here. If you're not avoiding friendface-style sites you can follow "highlights" easily enough - or just look at the site.

ObQuote: "Be strong and you will be renewed. Identify. " - Logan's Run (1976)

It would be really neat if there were some tool which recorded which dotfiles an application read, used, or created.

As an example emacs uses .emacs, but won't create it. However firefox will create and fill ~/.mozilla if it isn't present, and links will create ~/.links2.

What would we do with that data? I'm not sure off the top of my head, but I think it is interesting to collect regardless. Perhaps a simple tool such as apt-file to download the data and let you search:

who-creates ~/.covers
who-creates ~/.dia

Obviously the simple use is to purge user-data when matching packages are removed - e.g. dpkg-posttrigger hook. But that's a potentially dangerous thing to do.

Anyway I'm just pondering - I expect that over time applications will start switching to using "centralised" settings such as ~/.gconf2 etc.

In the menatime I've started cleaning up ~/ on my own machines - things like ~/.spectemurc, ~/.grip, etc.

ObQuote: What a long sword. I like that in a man - Blood of the Samurai (Don't be tempted; awful film.)

I miss the days when Debian was about making software work well together.

These days the user mailing lists are full of posts from users asking for help on Ubuntu [*1*], people suggesting that we copy what Ubuntu has done, and people asking for "howtos" because documentation is too scary or too absent for them to read.

Yesterday the whole commercial spam on Planet debate started. Its yet another example of how in-fighting[*2*] seems to be the most fun part of Debian for too many.

Me? I started and folded a company. I got Debian help and users. Some threw money at me.

Joey Hess? Started making the nice-looking ikiwiki-powered branchable.com.

Commercial? Yes. Spam? No.

I guess there is little I can do. I could quit - as over time I've become less patient dealing with the project as a whole, but simultaneously more interested in dealing with a few specific people. But I suspect the net result would be no change. Either people would say "Ok , bye" or worse still offer my flttry: "Don't go - we lurve you".

Meh.

I shouldn't write when I'm annoyed, but living in a hotel will do that to you.

Footy-Mc-Foot-notes. Cos HTML is hard. Lets go shopping Eat Cake.

1

The Ubuntu forums are largely full of the blind leading the blind. Or equally often the blind being ignored.

I do believe that an Ubuntu stackoverflow site would be more useful than forums. But that's perhaps naive. People will still most often say "My computer doesn't work" missing all useful the details.

The only obvious gain is you can avoid "me too!!!" comments, and "fix this now or I'm gonna go .. use gentoo?".

2

Back a few years when people were less civil some mailing lists and irc channesl were unpleasant places to be.

These days we've solved half the problem: People mostly don't swear at each other.

The distractions, the threads that don't die, even if you ignore them and don't join in the "hilarity" they still have a devisively negative effect.

It is unfortunate that most server-packages don't seperate out their init scripts into separate packages:

• foo
  • Contains the server binary, associated config files, and libraries.
• foo-run or foo-server
  • Contains the init script(s).

Right now its a real pain to have to modify things like /etc/init.d/ssh to launch two daemons, running on two different ports, with two different configuration files.

Running multiple copies of SMTP daemons, databases, and similar things is basically more complex than it has to be, because our packages aren't setup for it.

If you maintain a daemon please do consider this, failing that honoring a flag such as "DISABLED=true" in /etc/default/foo would allow people to use their own /etc/init.d/foo.local initscript. (That's not perfect, but it is a step in the right direction.)

ObFilm: Little Miss Sunshine.

Ubuntu always gets a lot of coverage in blogs, and the recent controvesy realisation that it isn't a 100% community-made distribution has triggered yet another round of this.

A lot of the controvesy, coverage, and attention can be laid at the feet of Canonical themselves; I think it is fair to say that the visibility, hype, advertising, and the goal of trying to be all things to all men means that even relatively trivial issues can easily get blown out of proportion, and to a certain extent this is a self-inflicted. Live by the sword media & etc ...

I think it is fair to say that Ubuntu has attracted a huge swathe of non-technical users. They want something "easy", "free", and "sexy", but more than that they want to use their computer, not develop the operating system.

When a particular bug report, with 400+ comments, hits the press we're primarily seeing a marketing-fail rather than technical-fail. The realisation that yes bugs are reported, but no the community (of users) doesn't get input into every single thing is as it should be. If you look back over "controvesy" in the past you'll see comments from the non-technical users which are tantamoint to blackmail:

This should be fixed ... or I'm gonna .. install .. gentoo. yeah. really.

(Similarly you see many comments of the form "I agree", "oh noes", or "Please revert ASAP", rather than technical arguments.)

This non-technical nature of the userbase is also readily apparent if you browse through the answers to problems posted in forums for example "Delete this file, I don't know why it works but it fixes it for me!!2!". (I've seen some truly horrific advice upon Ubuntu forums, even so far as chmodding various parts of the system to allow users to write binaries to /bin.)

Similarly you'll see that the launchpad is full of generic linux misunderstandings and bugs that aren't "real". The unfortunate fact is that the Ubuntu bug tracker is a wasteland in many places:

• Lots and lots and lots of users reporting bugs.
• Those bugs being ignored for huge periods of time
  • Except for "Hey is this still present in $pending-release?"
• The issue isn't that Ubuntu developers don't care, the issue is one of manpower.

The tight timescale of releases combined with the sheer number of incoming bug reports means that often issues are overlooked. (For example one bug that bit a colleague is #402188 - on the one hand it is a trivial bug, on the other hand its readily apparent to users. If something like that can be missed it makes you wonder ..?) ObRandom: Ubuntu has 100 bugs open against its Vim package, some of which have been marked as NEW since 2009 (ie. untouched, ignored). By contrast the Debian vim package has way fewer bugs. I'm sure there are packages where the situation is reversed but I think is not an unusual comparison.

Finally in addition to sheer numbers of bugs, and tight timescales, it has to be noted that the relative number of developers to users is miniscule, and this in turn has lead to some interesting solutions. The Ubuntu PPA system (personal package archive) should be a good thing. It should allow people to submit new packages for testing, for bugfixes, and for more visibility. Instead downloading a PPA file is no different than going to download.com and downloading a random binary - sure it might be legit, but there's no oversight, no quality control, and most likely no future updates.

Ubuntu as a distribution is interesting, and I'm not trying to be overly critical - A year or two ago had somebody thrown money at me I might have been inclined to accept it.

I think most of the perceived problems stem from a single common source, which is largely the issue of scale. (e.g. bug reports to bug handlers. developer numbers to user numbers.)

There are many good things to be said about Ubuntu (& Canonical) in addition to the negative ones that we see in the press or that I've perceived and mentioned above. The truth is it works for a lot of people, and the growing pains will continue until it either dies or both its audience and itself matures.

Either way I don't hate Ubuntu, in the same way that I don't hate Microsoft, Oracle, Fedora, Gentoo, or other mass-entities. There are pros and cons to be made for most of them, (and of course Debian itself is no different).

However I will say that every time I see people write "If you want a sexy/shiny/easy to use Linux desktop then install Ubuntu" I glance over at my Debian Lenny desktop, marvel at how sexy, shiny and easy to use it is, and get a little bit disappointed at our own marketing failure(s).

ObFilm: Day of the Woman

Recently I received a small flurry of patches to my blog compiler, from Chris Frey. These patches significantly speedup rebuilding a static blog when using Danga's memcached.

The speedup is sufficiently fast that my prior SQLite based approach is no longer required - and (re)building my blog now takes on the order of 5 seconds.

On the topic of other people's blogs I've been enjoying David Watson's recent photo challenge. I was almost tempted to join in, but I'm not sure I could manage one every day - Although I can pretend I recently carried out my my first real photoshoot.

I'm still taking pictures of "things/places" but I'm starting to enjoy "people" more. With a bit of luck I'll get some more people to pose in the near future, even if I have to rely upon posting to gumtree for local bodies!

ObFilm: Love Actually

Last week I resigned from my position as member of the Debian Security Team.

Historically several Debian teams have had members inactive for months and years at a time, and I'd rather be removed of my own volition than end up listed but inactive like that.

It's been a pleasure working with all members of the team, past and current (especially Joey), and who knows I might return in the future.

If you're interested in security work then getting involved isn't difficult. It just takes time, patience, and practise.

ObFilm: The Goonies

Debian project leader elections are coming up soon. My vote will go to the candidate that:

• Makes no promises for significant projectwide change.
• Provides at least one mail every 8 weeks to summerise "stuff".

Thats all.

I guess most people don't really care about being a DPL per-se, instead they stand for election to accomplish a pet project, or two. (Not that there is anything wrong with that. If your platform says "Elect me and I'll do $foo" then you've got implicit support.)

A recurring theme in platforms has been communication, and I think that over time it has got a lot better. Regardless, if theres one thing I want to see from the DPL in 2009 it is even more communication.

In other news, depending on your timezone, todays date is 3/3/9 - enjoy it while it lasts. The next similar date is 4/4/16 - which is seven years in the future.

ObTitle: Blood Ties

I'm not sure how you can pre-announce something in a way that cannot be later faked.

The best I can imagine is you write it in a text file and post the size / hash of the file.

steve@skx:~$ ls -l 10-march-2009
-rw-r--r-- 1 steve users 234 Jan 12 21:40 10-march-2009
steve@skx:~$ sha1sum 10-march-2009
99d1b6d625ed4c15a3be2be5fec63c17941c370d  10-march-2009
steve@skx:~$ md5sum 10-march-2009
1a0e68b8fbb3b0fe30e5b4a9413ceeec  10-march-2009

I don't need anybody to keep me honest, but I welcome interesting suggestions on more neat ways to pre-confirm you have content that hasn't been changed between being written and being released...?

I guess you could use GPG and a disposible key-pair, and then post the secret key afterward, but that feels kinda wrong too.

Update of course you could post the detached signature. D'oh.

Shamir's Secret Sharing could be another option - posting just enough pieces of the secret to make recovery possible with the addition of one piece that was witheld until the later date. Jake wrote a nice introduction to secret sharing a couple of years ago.

ObFilm: Léon

Since I've ranted a little recently lets do another public bugfix. The last few times people seemed to like them, and writing things down helps me keep track of where I am, what I'm doing, and how soon it will be "beer o'clock".

So I looked over the release critical bug list, looking for things that might be easy to fix.

One bug jumped out at me:

• gnomad2: segfault on start up / vorbis comment headers
• (Original report)

I installed the package:

skx@gold:~$ apt-get install gnomad2
..
The following NEW packages will be installed
  gnomad2 libmtp7 libnjb5 libtagc0
..

Once I copied an .mp3 file to my home directory, with the .ogg suffix I got a segfault on startup:

skx@gold:~$ cp /home/music/Audio/RedDwarf-back-to-reality.mp3 ~/foo.ogg
skx@gold:~$ gnomad2
..
LIBMTP_Get_First_Device: No Devices Attached
PDE device NULL.
TagLib: Ogg::File::packet() -- Could not find the requested packet.
TagLib: Vorbis::File::read() - Could not find the Vorbis comment header.
[segfault]
skx@gold:~$

So I downloaded the source ("apt-get source gnomad2"), and the dependencies for rebuilding ("apt-get build-dep gnomad2"). This allowed me to rebuild it locally:

skx@gold:/tmp/gnomad2-2.9.1$ ./configure --enable-debug && make
skx@gold:/tmp/gnomad2-2.9.1$ cd src/

And now it can be run under GDB.

skx@gold:/tmp/gnomad2-2.9.1/src$ gdb ./gnomad2
GNU gdb 6.8-debian
...
(gdb) run
Starting program: /tmp/gnomad2-2.9.1/src/gnomad2
...
LIBMTP_Get_First_Device: No Devices Attached
PDE device NULL.
[New Thread 0x41dcd950 (LWP 23593)]
TagLib: Ogg::File::packet() -- Could not find the requested packet.
TagLib: Vorbis::File::read() - Could not find the Vorbis comment header.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x41dcd950 (LWP 23593)]
0x00007fc89e0340b0 in taglib_tag_artist () from /usr/lib/libtag_c.so.0
(gdb)

Interestingly the crash comes from a library, libtag_c.so.0. So either:

• The bug is realy in the libtag module.
• The gnomad2 package doesn't handle a failure case it should.

Time to guess which that is. Charitably we'll assume any library that segfaults will be quickly fixed, because it will have more users than a single program..

Moving on we can look at the gnomad2 source for mentions of taglib. Several mentions are found, but src/tagfile.c is clearly the correct place to look. That file contains the following code:

void
get_tag_for_file (metadata_t *meta)
{
  gchar *tmppath = filename_fromutf8(meta->path);
  TagLib_Tag *tag;
  const TagLib_AudioProperties *properties;
  TagLib_File *file = taglib_file_new(tmppath);

  if(file == NULL) {
    g_printf("could not open file %s", tmppath);
    g_free(tmppath);
    return;
  }
  g_free(tmppath);

  tag = taglib_file_tag(file);
  properties = taglib_file_audioproperties(file);

  gchar* artist = taglib_tag_artist(tag);
  ..

This looks like a great place to explore because opening a file to read the tags is most likely where the crash is going to be coming from.

Interestingly we see the code:

• Calls taglib_file_new to get a handle of some kind relating to the file.
• Tests for errors with that operation.
• Calls taglib_file_tag to fetch tag information using the handle, and indirectly the file.
• But then uses taglib_tag_artist to fetch the artist (?I guess?) without testing the handle previously obtained was valid.

Let us guess that the file opening is succeeding but that the tag structure fetched via taglib_file_tag is NULL - and this causes the crash.

A quick edit:

  g_free(tmppath);

  tag = taglib_file_tag(file);
  if(tag == NULL) {
    g_printf("tag was null");
    return;
  }

  properties = taglib_file_audioproperties(file);
  gchar* artist = taglib_tag_artist(tag);

Rebuild, and the segfault is gone. We have a winner. Now we just need to file a patch...

ObFilm: 300

Previously I've posted a couple of running commentries describing how I've examined and gone about fixing a couple of bugs in Debian packages I use, enjoy, or have stumbled upon by accident.

Each of these commentries has resulted in a change or two to the affected software to make the bug vanish.

Fixing the bug is usually the hard part, but obviously it isn't the only thing that you need to do. While it is fun to have a locally fixed piece of software if you don't share it then the very next release will have the same bug again - and you'll spend your life doing nothing more than fixing the same bug again and again.

Generally the way that I report my fixes is by sending email to the Debian bug tracker - because I usually only try to fix bugs that I see reported already. Specificially I tend to only care about:

• Bugs in packages that I maintain.
• Bugs that affect me personally. I'm selfish.
• Bugs in packages I use, even if they don't affect me.
• Segfaults. Because segfaults and security issues often go hand in hand.

ObFilm: Ghostbusters 2

Is Debian slowly tearing itself apart, or am I just unduly dramatic?

ObFilm: Kill Bill (volume two)

So I finally got round to vote on the Debian Lenny Firmware issue.

I can't help thinking that most of the discussion so far has been a waste of time, on the grounds that developers either

• Care a lot about the subject and already have an opinion.
• Don't care and get bored by the discussion.

Either way the discussion has two, or more, sides talking past each other rather than any convergence upon a consensus. (Me? I'm in the middle. On the fence. Getting bored. Hitting "Delete thread" a lot. Who knows, maybe I could have my vote influenced, but right now I'm just not caring enough either way.)

Anyway enough on that subject. Tonight I have mostly been looking around to come up with a simple way to give an "overview" of mail traffic. Something like:

• Most prolific poster.
• Most popular thread.
• etc etc.

Individually these jobs are easy. Making it look pretty is hard. (Also I'd like to have archives of the lists in question, which are searchable.)

Seems like nothing out there does everything I want.

ObFilm: Ferris Bueller's Day Off.

So, I've just got a portable machine. I've configured it to be a pretty minimal installation of Debian Lenny, but one thing that makes me unhappy is mail handling.

By default it came with several exim4- packages. Now in general Exim4 rocks. But it is a deamon running all the time, and overhead I could live without.

I looked around to find a mail transport agent that would be more suited to the machine and was suprised to find nothing suitable.

Basically I figure the machine will never generate "real" emails. Instead it will only receive mails from cron, etc. The machine will never have a real fixed IP, and so relaying mail externally is a waste of time. The mail should just go somewhere predictable and local.

There are a couple of lightweight agents which will forward to another system, but nothing seems to exist which will queue mail locally only.

So I've hacked a simple script which will do the job.

Given the spool director /var/spool/skxmail the following command:

skxmail root < /etc/passwd

Produces this:

/var/spool/skxmail/
`-- root
    |-- cur
    |-- new
    |   `-- 1227702470.P8218M243303Q22.vain.my.flat
    `-- tmp

4 directories, 1 file

That seems to be sufficient for my needs. (I support the flag which says "read the receipient from the body).

Of course to do this properly I'd be setgid(mailgroup). Instead I assume that local means everybody can see it and /var/spool/skxmail is mode 777. Ooops.

Still happy to share if it sounds interesting.

ObFilm: Dogma

To solve performance problems I've now started to switch my SMTP servers from using the "forkserver" version of qpsmtpd to using the "prefork" version.

Under testing qpsmtpd-prefork performed significantly better than the qpsmtpd-forkserver for handling incoming SMTP connections.

The loadavg of one machine has dropped from a constant 2.xx to 0.4x!

I'd love to see what the asynchronous server would behave like, but that would require re-writing all my plugins to work in an asynchronous manner which would be a significant undertaking.

(It would be nice if the qpsmtpd package available to Debian would allow you to choose between the two version of the server - I will file a wishlist bug.)

ObQuote: Dune

I've setup several repositories for apt-get in the past, usually using reprepro as the backend. Each time I've come up with a different scheme to maintain them.

Time to make things consistent with a helper tool:

skx@gold:~/hg/rapt$ ls input/
spambayes-threaded_0.1-1_all.deb        spambayes-threaded_0.1-1.dsc
spambayes-threaded_0.1-1_amd64.build    spambayes-threaded_0.1-1.dsc.asc
spambayes-threaded_0.1-1_amd64.changes  spambayes-threaded_0.1-1.tar.gz

So we have an input directory containing just the package(s) we want to be in the repository.

We have an (empty) output directory:

skx@gold:~/hg/rapt$ ls output/
skx@gold:~/hg/rapt$

Now lets run the magic:

skx@gold:~/hg/rapt$ ./bin/rapt --input=./input/ --output=./output/
Data seems not to be signed trying to use directly...
Data seems not to be signed trying to use directly...
Exporting indices...

What do we have now?

skx@gold:~/hg/rapt$ tree output/
output/
|-- dists
|   `-- etch
|       |-- Release
|       |-- main
|           |-- binary-amd64
|           |   |-- Packages
|           |   |-- Packages.bz2
|           |   |-- Packages.gz
|           |   `-- Release
|           `-- source
|               |-- Release
|               |-- Sources
|               |-- Sources.bz2
|               `-- Sources.gz
|-- index.html
`-- pool
    `-- main
        `-- s
            `-- spambayes-threaded
                |-- spambayes-threaded_0.1-1.dsc
                |-- spambayes-threaded_0.1-1.tar.gz
                `-- spambayes-threaded_0.1-1_all.deb

neat.

Every time you run the rapt tool the output pool and dists directories are removed and then rebuilt to contain only the packages located in the incoming/ directory. (More correctly only *.changes are processed. Not *.deb.)

This mode of operation might strike some people as odd - but I guess it depends on whether you view "incoming" to mean "packages to be added to the exiting pool", or "packages to take as the incoming input to the pool generation process".

Anyway if it is useful to others feel free to clone it from the mercurial repository. There is no homepage yet, but it should be readable code and there is a minimum of supplied documentation in the script itself ..

ObQuote: Buffy. Again.

Well I'm back and ready to do some fun work.

In the meantime it seems that at least one of my crash-fixes, from the prior public bugfixing, has been uploaded:

• yelp: segfault while loading info documentation

I'm still a little bit frustrated that some of the other patches I made (to different packages) were ignored, but I guess I shouldn't be too worried. They'll get fixed sooner or later regardless of whether it was "my" fix.

In other news I've been stalling a little on the Debian Administration website.

There are a couple of reasonable articles in the submissions queue - but nothing really special. I can't help thinking that the next article being a nice round number of 600 deserves something good/special/unique? hard to quantify, but definitely something I'm thinking. I guess I leave it while the weekend and if nothing presents itself I'll just go dequeue the pending pieces.

In other news I've managed to migrate the mail scanning service into a nicely split architecture - with minimal downtime.

I'm pleased that:

• The architecture was changed massively from a single-machine orientated service to a trivially scalable one - and that this was essentially seamless.
• My test cases really worked.
• I've switched from being "toy" to being "small".
• I've even pulled in a couple of new users.

Probably more changes to come once I've had a rest (but I guess I write about that elsewhere; because otherwise people get bored!).

The most obvious change to consider is to allow almost "instant-activation". I dislike having to manually approve and setup new domains, even if it does boil down to clicking a button on a webpage - so I'm thinking I should have a system in place such that you can sign up, add your domain, and be good to go without manual involvement. (Once DNS has propogated, obviously!)

Anyway enough writing. Ice-cream calls, and then I must see if more bugs have been reported against my packages...

ObQuote: Run Lola Run.

Well a brief post about what I've been up to over the past few days.

An alioth project was created for the maintainance of the bash-completion package. I spent about 40 minutes yesterday committing fixes to some of the low-lying fruit.

I suspect I'll do a little more of that, and then back off. I only started looking at the package because there was a request-for-help bug filed against it. It works well enough for me with some small local additions

The big decision for the bash-completion project is how to go forwards from the current situation where the project is basically a large monolithic script. Ideally the openssh-client package should contain the completion for ssh, scp, etc..

Making that transition will be hard. But interesting.

In other news I submitted a couple of "make-work" patches to the QPSMTPD SMTP proxy - just tidying up a minor cosmetic issues. I'm starting to get to the point where I understand the internals pretty well now, which is a good thing!

I love working on QPSMTPD. It rocks. It is basically the core of my antispam service and a real delight to code for. I cannot overemphasise that enough - some projects are just so obviously coded properly. Hard to replicate, easy to recognise...

I've been working on my own pre-connection system which is a little more specialied; making use of the Class::Pluggable library - packaged for Debian by Sarah.

(The world -> Pre-Connection/Load-Balancing Proxy -> QPSMTPD -> Exim4. No fragility there then ;)

Finally I made a tweak to the Debian Planet configuration. If you have Javascript disabled you'll no longer see the "Show Author"/"Hide Author" links. This is great for people who use Lynx, Links, or other minimal browsers.

TODO:

I'm still waiting for the creation of the javascript project to be setup so that I can work on importing my jQuery package.

I still need to sit down and work through the Apache2 bugs I identified as being simple to fix. I've got it building from SVN now though; so progress is being made!

Finally this weekend I need to sit down and find the time to answer Steve's "Team Questionnaire". Leave it any longer and it'll never get answered. Sigh.

ObQuote: Shooting Fish

I installed Debian upon a new desktop machine yesterday, via a PXE network boot.

It was painless.

Getting xen up and running, with a 32-bit guest and a 64-bit guest each running XDMCP & VNC was also pretty straightforward.

There is a minor outstanding problem with the 32-bit xen guest though; connecting to it from dom0, via XDMCP, I see only a blank window - no login manager running.

GDM appears painlessly when I connect via VNC.

The relevent configuration file looks like this:

# /etc/gdm/gdm.conf
[security]
AllowRoot=true
AllowRemoteRoot=true

[xdmcp]
Enable=true

The same configuration on the 64-bit guest works OK for both cases.

(I like to use XDMCP for accessing the desktop of Xen guests, since it means that I get it all full-screen, and don't have to worry about shortcuts affecting the host system and not the guest - as is the case if you're connecting via VNC, etc).

Weirdness. Help welcome; I'm not 100% sure where to look

Anyway, once again, a huge thank you to the Debian Developers, bug submitters, and anybody else involved peripherally (such as myself!) with Debian!

I love it when a plan comes together.

SSL

ObRandom: Where is the cheapest place to get an SSL certificate, for two years, which will work with my shiny Apache2 install?

Somebody, rightly, called me for not having SSL available as an option on my mail filtering website.

I've installed a self-signed certificate just now, but I will need to pay the money and buy a "real" one shortly.

So far completessl.com seems to be high in the running:

• 1 year - £26
• 2 years - £49

For double-bonus points they accept Paypal which most of my customers pay with ..

ObQuote: The Princess Bride

My blog compiler received a bit of love recently, primarily because MJ Ray wanted to use it.

As mentioned before I've added a simple spooling system, and the mercurial repository now contains a simple RSS importer.

In other news I've been working on various Debian packages, here is a brief summery:

bash-completion

After seeing a RFH bug I closed a few bash-completion bugs, and submitted patches for a couple more.

I was intending to do more, but I'm still waiting for the package code to be uploaded to the the alioth project.

javascript work

I've updated the jquery package I uploaded to follow the new "Javascript standard" - in quotes only because it is both minimal and new.

Once the alioth project has been configured I'll upload my sources.

Apache2

I've agreed to work on a couple of SSL-related bugs in the Apache 2.x package(s) - time disappeared but I hope to get that done this weekend.

Initially that was because I was hoping I could trade a little love for getting a minor patch applied to mod_vhost_alias - instead I've now copied that module into libapache2-mod-vhost-bytemark and we'll maintain our own external module.

I've been loaned a Nokia 770 which is very nice. Having used it with vim, ssh & etc I think that I'd rather have a device with a real keyboard.

The Nokia 810 looks pretty ideal for me. I'm going to be asking around to see if I can get a donated/loaned device to play with for a while before I take the plunge and pay for one of my own.

I've got a couple more things on the go at the moment, but mostly being outdoors is more interesting to me than the alternative. Hence the downturn in writing and releasing security advisories.

I'll pick things up more fully over the coming weeks I'm sure.

ObQuote: Shaun of the Dead

http://10.print.debian.rocks.twentygototen.org/

ObQuote: Aliens

One thing I do a lot is select a region of text, then have it replaced with the output of a command.

The most common job is sorting a number of lines, such as "use XX:YY;" lines in perl scripts.

Finally having gotten annoyed enough about how clunky shell-command-on-region was I wrote my own lisp function:

• replace-region-command-output

Only after that did I discover M-x sort-lines. D'oh. Still I guess my solution is more general, and less difficult to use. (I find the use of the Emacs prefix troublesome to type; since you have to do it in advance - I almost always forget.)

I also learnt of M-x list-matching-lines yesterday. Thats a new discovery which really rocks. (I can use "^sub " to find a list of subroutines, etc.)

This could be improved, and fleshed out a lot if there were any interest.

But its neat as-is:

#!/bin/sh
#
#  Dump packages in the NEW queue.
#
#  This could be improved, perhaps:
#
#  --show-names --show-dates, etc.  Or just show all info in a table.
#
wget --quiet -O-  http://ftp-master.debian.org/new.html | \
 perl -ne 'print $1 . "\n" if ( $_ =~ /^<td valign="top" class="sid">([^<]+)<\/td>$/ )'

I thought there was something similar in the devscripts package, or contained within debian-goodies but apparently not.

ObQuote:Battle Royale

Today I spent a while fixing some more segfault bugs. I guess that this work qualifies as either fixing RC bugs, or potential security bugs.

Anyway I did an NMU of libpam-tmpdir a while back to fix all but one of the open bugs against it.

I provided a patch for #461625 yelp: segfault while loading info documentation, which fixes the symptoms of bad info-parsing, and avoids the segfault.

I also looked into the #466771 busybox cpio: double free or corruption during cpio extraction of hardlinks - but it turns out that was already fixed in Sid.

Finally I found a segfault bug open against ftp:

• ftp: GET segfaults. Buffer overflow?
• Buffer overflow: comman buffer size is not checked (Almost certainly a duplication bug.)

To reproduce this bug run:

skx@gold:~$ ftp ftp.debian.org
220 saens.debian.org FTP server (vsftpd)
Name (ftp.debian.org:skx): anonymous
331 Please specify the password
Password: foo@bar.com
ftp> cd debian/doc
250 Directory successfully changed.
ftp> get dedication-2.2.cn.txt dedication-2.2.de.txt dedication-2.2.es.txt ..
local: dedication-2.2.de.txt remote: dedication-2.2.cn.txt
Segmentation fault

You need to repeat the arguments about 50 times. But keep adding more and more copies of the three files to the line until you get the crash.

It isn't interesting as a security issue as it is client side only; but as a trivially reproducable issue it becomes fun to solve.

I mailed the maintainer of FTP and said unless I heard differently I'd NMU and cleanup the package in a week.

All being well this entry will be nicely truncated in the RSS feeds as support for the <cut> tag was the main new feature in my previous upload of chronicle - the blog compiler I use/wrote/maintain.

ObQuote: Razor Blade Smile

Well this has been a busy week, and no mistake.

Still I've advocated a new individual who wishes to become a Debian Developer. I guess now I get to watch second-hand to see how long the process takes!

(I messed up though; the first sponsored upload for her has the wrong mail address. It'll get REJECTed, and then we'll try again. D'oh.)

In more optimistic news this weekend I'm going to attempt to finish painting my front room. The painting of this room was started n the 3rd of February. So we're coming close to two months. A new record!

Also this weekend I must write some letters ...

Tonight will involve some Balvenie and a copy of The Godfather (part 1).

ObQuote: Eight Legged Freaks.

I've just finished reading "Don't You Have Time to Think", a collection of letters written to and from Richard P. Feynman. A birthday present from my wishlist.

Previously I've read the collection of letters to/from Tolkien. (Several times actually. Very nice collection!)

It suddenly struck me that over my lifetime I've probably written <200 letters to people.

When I was young I had a couple of pen pals, and when I was entering university I was involved with a couple of play by mail games which involved writing random letters involving strategy & etc.

Personal letters though? I've written very few, and I think they've mostly consisted of letters to my partner/partners of the time.

(For example Megan went home for a few months at the end of a university year about two months after I initially met her. So there were many letters back and forth. Recently she spent two months working in the USA; counting eggs and avoiding alligators so again there was a flurry of written letters, maybe 20 total during the duration of her trip.)

I guess that most of my (hand)written messages to people have been in the form of postcards whilst on holiday.

A long time ago I offered to mail postcards to Debian developers. I know I sent at least two, and I received at least one back - but the thing I remember most was exchanging addresses with Amayita and getting into character set issues. Her emails, containing her Spanish address, were difficult to understand as my mutt/console refused to display the foreign character set properly.

I can't recall whether she did ultimately receive a card from me, but I'm sure she'll remind me if she did.

Anyway I have no resolution, intention, or expectation that I will suddenly start writing more physical mails to people. But I think it almost counts as something we do less of these days. The telephone and internet have become the norm.

In some ways this is fantastic. In others it less good.

On the left my handwriting is so bad that maybe this isn't necessarily a problem.

ObQuote: E.T.

Since my last example of fixing a bug received some interesting feedback (although I notice no upload of the package in question ..) we'll have another go.

Looking over my ~/.bash_history file one command I use multiple times a day is make. Happily GNU make has at least one interesting bug open:

• #419190 - make: double free or corruption

I verified this bug by saving the Makefile in the report and running make:

skx@gold:~$ make
make: file.c:84: lookup_file: Assertion `*name != '\0'' failed.
Aborted

(OK so this isn't a segfault; but an assertion failure is just as bad. Honest!)

So I downloaded the source to make, and rebuilt it. This left me with a binary with debugging symbols. The execution was much more interesting this time round:

skx@gold:~$ ./make
*** glibc detected ***
  /home/skx/./make: double free or corruption (fasttop): 0x00000000006327b0 ***
======= Backtrace: =========
/lib/libc.so.6[0x2b273dbdd8a8]
/lib/libc.so.6(cfree+0x76)[0x2b273dbdf9b6]
/home/skx/./make[0x4120a5]
/home/skx/./make[0x4068ee]
/home/skx/./make[0x406fb2]
...
[snip mucho texto]

And once I'd allowed core-file creation ("ulimit -c 9999999") I found I had a core file to help debugging.

Running the unstripped version under gdb showed this:

(gdb) up
#5  0x00000000004120a5 in multi_glob (chain=0x1c, size=40) at read.c:3106
3106			    free (memname);

So it seems likely that this free is causing the abort. There are two simple things to do here:

• Comment out the free() call - to see if the crash goes away (!)
• Understand the code to see why this pointer might be causing us pain.

To get started I did the first of these: Commenting out the free() call did indeed fix the problem, or at least mask it (at the cost of a memory leak):

skx@gold:~$ ./make
make: *** No rule to make target `Erreur_Lexicale.o', needed by `compilateur'.  Stop.

So, now we need to go back to read.c and see why that free was causing problems.

The function containing the free() is "multi_glob". It has scary pointer magic in it, and it took me a lot of tracing to determine the source of the bug. In short we need to change this:

free (memname);

To this:

free (memname);
memname = 0;

Otherwise the memory is freed multiple times, (once each time through the loop in that function. See the source for details).

Patch mailed.

So today I'm a little bit lazy and I've got the day off work. As my previous plan suggested I wanted to spend at least some of the day tackling semi-random bugs. Earlier I picked a victim: less.

less rocks, and I use it daily. I even wrote an introduction to less once upon a time.

So lets take a look at two bugs from the long-neglected pile. These two issues are basically the same:

• gzip: zless seems spew raw gzip to output for empty files
• zless thinks gzipped empty file is binary

They seem like simple ones to fix, with the same root cause. Here's an example if you want to play along at home:

 cp /dev/null testing
 gzip testing
 zless testing.gz

What do you see? I see this:

"testing.gz" may be a binary file.  See it anyway?

When I select "y" I see the raw binary of the compressed file.

So, we can reproduce it. Now to see why it happens. /bin/zless comes from the gzip package and is a simple shell script:

#!/bin/sh
# snipped a lot of text
LESSOPEN="|gzip -cdfq -- %s"; export LESSOPEN
exec less "$@"

So what happens if we run that?

$ LESSOPEN="|gzip -cdfq -- ~/testing.gz" /usr/bin/less ~/testing.gz
"/home/skx/testing.gz" may be a binary file.  See it anyway?

i.e. it fails in the same way. Interestingly this works just fine:

gzip -cdfq -- ~/testing.gz | less

So we've learnt something interesting and useful. We've learnt that when LESSOPEN is involved we get the bug. Which suggests we should "apt-get source less" and then "rgrep LESSOPEN ~/less-*/".

Doing so reveals the following function in filename.c:

	public char *
open_altfile(filename, pf, pfd)
	char *filename;
	int *pf;
	void **pfd;
{

/* code to test whether $LESSOPEN is set, and attempt to run the
   command if it is */

		/*
		 * Read one char to see if the pipe will produce any data.
		 * If it does, push the char back on the pipe.
		 */
		f = fileno(fd);
		SET_BINARY(f);

		if (read(f, &c, 1) != 1)
		{
			/*
			 * Pipe is empty.  This means there is no alt file.
			 */
			pclose(fd);
			return (NULL);
		}
		ch_ungetchar(c);
		*pfd = (void *) fd;
		*pf = f;
		return (save("-"));

That might not be digestible, but basically less runs the command specified in $LESSOPEN. If it may read a single character of output from that command it replaces the file it was going to read with the output of the command instead!

(i.e. Here less failed to read a single character, because our gzipped file was zero-bytes long! So instead it reverted to showing the binary gzipped file.)

So we have a solution: If we want this to work we merely remove the "read a single character test". I can't think of circumstance in which that would do the wrong thing, so I've submitted a patch to do that.

Bug(s) fixed.

Incidentally if you like these kind of "debuggin by example" posts, or hate them, do let me know. So I'll know whether to take notes next time or not..

The webpages for the Debian Security Audit Project have been outdated for quite some time. Specifically because they contained two static pages comprised of large lists which were annoying to update:

• The list of security advisories we've been responsible for.
• The list of security-sensitive bug reports we'd made which didn't require a security advisory. (ie. packages in sid)

Last week, with the help of Rhonda, and a couple of other people, I removed the static lists and replaced them with simple data files, and a perl script to convert those data files into HTML content.

Now the advisories which have been released are ordered by date, and broken down into years. For example 2002, 2003, & etc. We also have a list of all the people who've been credited with at least one advisory.

There are some outstanding advisories still to be included, but otherwise I'm much happier with the process (and feel only guilt at the breakage of the translations).

There isn't much actual auditing happening at the moment, with only four advisories released in 2007 compared to many more at the peak. But I guess that is a separate problem, and one that I can do less about - short of finding more time to look at source code.

I'm in a position where I need to rebuild a Linux kernel for a number of distributions and architectures. Currently the distributions are:

• Debian Etch
• Ubuntu Dapper
• Ubuntu Edgy
• Ubuntu Feisty
• Ubuntu Gutsy

(For each distribution I need a collection of packages for both i386 and amd64.)

I've written a couple of scripts to automate the process - first of all running "make menuconfig" within a debootstrap-derived chroot of each arch & distribution pair. Then later using those stored .config files to actually produce the packages via make-kpkg.

This process, as you could imagine, takes several hours to complete. Then there's the testing ...

I'm sure there must be other people with this kind of need but I was suprised to see nothing in my search attempts.

ObRandom: I'm tempted to switch from song-lyrics to film names as post titles. Undecided as yet. I guess it doesn't really matter, just gives me a small amount of amusement. Even now.

It is interesting that there have been posts about archive tools appearing upon the Planet Debian.

Recently I setup an instance of rebuildd which worked nicely once I'd installed the required dependencies manually.

I also run three instances of reprepro, but there life is not such a happy picture.

I might be using reprepro incorrectly, but despite fighting with it for some time I cannot coerce the software into allowing me to upload the same version of a binary package for amd64 & i386 architectures - something I frequently want to do.

On the face of it importing packages into a small database doesn't seem terribly difficult, but it is a problem I've not spent much time looking at yet.

Bits from the Security Team

• We get tons of spam. If your issue isn't replied to at least once wait a day and resend.
• Frequently advisories are delayed because our buildd machines are broken. We can't fix them.
• People reporting bugs with the 'security' tag help us.
• People reporting bugs with patches help us more.
• People reporting bugs with patches and pointers to fixed packages they have build help us best.
• I like pies.

I am happy to look over patches, built packages, and generally encourage people to be involved. Our team isn't huge but historically we've only added people who've done a fair bit of work first. That is both good and bad.

I could write more, and probably should, but I'll stop there for now because I'm frustrated by the HPPA build machine. Again.

ObRelated: Moritz is trying to get the archive rebuilt with security features from our compilers (eg. -fstack-protector) included. This would be a fantastic achievement. People interested in tested kernel patches, donating buildd machines, etc, etc should give him a ping.

Here's a quick question - does there exist a stable and reliable caching proxy for APT?

Both apt-proxy and approx cause me problems on a regular basis - with MD5 sum mismatches on Release files. And general hangs, stalls and timeouts.

I've just installed and configured apt-cacher but #437166 doesn't fill me with confidence..

The source-searching system I was talking about previously is progressing slowly.

So far I've synced the source to Etch to my local machine, total size 29Gb, and this evening I've started unpacking all the source.

I'm still in the "a" section at the moment, but thanks to caching I should be able to re-sync the source archive and unpack newer revisions pretty speedily.

The big problem at the moment is that the unpacking of all the archives is incredibly slow. Still I do have one new bug to report aatv: Buffer overflow in handling environmental variables..

That was found with:

rgrep getenv /mnt/mirror/unpacked | grep sprintf

(A very very very slow pair of greps. Hopefully once the unpacking has finished it will become faster. ha!)

The only issue I see at the moment is that I might not have the disk space to store an unpacked tree. I've got 100Gb allocated, with 29Gb comprised of the source. I'll just have to hope that the source is less than 70Gb unpacked or do this in stages.)

I've been working on a list of patterns and processes to run, I think pscan, rats, and its should be the first tools to run on the archive. Then after that some directed use of grep.

If anybody else with more disk space and connectivity than myself be interested I can post the script(s) I'm using to sync and unpack .. Failing that I'll shut up now.