Steve Kemp's Blog Writings relating to Debian & Free Software

Entries tagged ruby

So I failed at writing some clustered code in Perl

Mon, 24 Mar 2014 12:32:38 GMT

Until this time next month I'll be posting code-based discussions only.

Recently I've been wanting to explore creating clustered services, because clusters are definitely things I use professionally.

My initial attempt was to write an auto-clustering version of memcached, because that's a useful tool. Writing the core of the service took an hour or so:

  • Simple KeyVal.pm implementation.
  • Give it the obvious methods get, set, delete.
  • Make it more interesting by creating a read-only append-log.
  • The logfile will be replayed for clustering.

At the point I was done the following code worked:

use KeyVal;

# Create an object, and set some values
my $obj = KeyVal->new( logfile => "/tmp/foo.log" );
$obj->incr( "steve" );
$obj->incr( "steve" );

print $obj->get( "steve" ) # prints 2.

# Now replay the append-only log
my $replay = KeyVal->new( logfile => "/tmp/foo.log" );
$replay->replay();

print $replay->get( "steve" ) # prints 2.

In the first case we used the primitives to increment a value twice, and then fetch it. In the second case we used the logfile the first object created to replay all prior transactions, then output the value.

Neat. The next step was to make it work over a network. Trivial.

Finally I wanted to autodetect peers, and deploy replication. Each host would send out regular messages along the lines of "Do you have updates made since $time?". Any that did would replay the logfile from the given unixtime offset.

However here I ran into problems. Peer discovery was supposed to be basic, and I figured I'd write something that did leader election by magic. Unfortunately Perls threading code is .. unpleasant:

  • I wanted to store all known-peers in a singleton.
  • Then I wanted to create threads that would announce and receive updates.

This failed. Majorly. Because you cannot launch the implementation of a class-method as a thread. Equally you cannot make a variable which is "complex" shared across threads.

I wrote some demo code which works without packages and a shared singleton:

The Ruby version, by contrast, is much more OO and neater. Meh.

I've now shelved the project.

My next, big, task was to make the network service utterly memcached compatible. That would have been fiddly, but not impossible. Right now I just use a simple line-based network protocol.

I suspect I could have got what I wanted using EventMachine, or similar, but that's a path I've not yet explored, and I'm happy enough with that decision.

| 2 comments.

 

I understand volunterering is hard

Sat, 5 Oct 2013 12:32:38 GMT

The tail end of this week was mostly spoiled by the discovery that libbeanstalkclient-ruby was not included in Wheezy.

Apparently it was removed because the maintainer had no time, and there were no reverse dependencies - #650308.

Debian maintainers really need to appreciate that no official dependencies doesn't mean a package is unused.

Last year I needed to redesign our companies monitoring software, because we ran out of options that scaled well. I came up with the (obvious) solution:

  • Have a central queue containing jobs to process.
    • e.g. Run a ping-test on host1.example.com
    • e.g. Run an SSH-probe on host99.example.com
    • e.g. Fetch a web-page from https://example3.net/ and test it has some text or a given HTTP status code.
    • (About 15 different test-types are available).
  • Have N workers each pull one job from the queue, execute it, and send the results somewhere.

I chose beanstalkd for my central queue precisely because it was packaged for Debian, had a client library I could use, and seemed to be a good fit. It was a good fit, a year on and we're still running around 5000 tests every minute with 10 workers.

The monitoring tool is called Custodian Custodian, and I think I've mentioned it before here and on the company blog.

It looks like we'll need to re-package the Ruby beanstalk client, and distribute it alongside our project now. That's not ideal, but also not a huge amount of work.

In summary? Debian you're awesome. But libraries shouldn't be removed unless it can't be helped, because you have more users than you know.

| 4 comments.

 

Tonight I've mostly been using Sinatra

Fri, 6 Jan 2012 12:32:39 GMT

This evening I've mostly been using Sinatra to build a little file storage service which uses a REST API.

That means I can upload a file:

skx@birthday:~/hg/sinatra$ curl -X PUT -F file=@/etc/fstab http://localhost:4567/
{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","path":"/"}

Download that same file:

skx@birthday:~/hg/sinatra$ curl -X GET -F id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a  \
   http://localhost:4567/
# /etc/fstab: static file system information.
..
/dev/cdrom        /media/cdrom0   udf,iso9660 user,noauto     0       0

Get an index of files:

skx@birthday:~/hg/sinatra$ curl http://localhost:4567/
[{"id":"dbd1bdc11b5a1a8e80588a135648b4c2edffb49a","type":"file"}]

And finally we can delete a file:

skx@birthday:~/hg/sinatra$ curl -X DELETE -F "id=dbd1bdc11b5a1a8e80588a135648b4c2edffb49a" \
  http://localhost:4567/
Removed

We can also upload to different paths so we can replicate a file-system if we wanted to. (I added in "type" to hold either "file" or "directory", though I guess if we were to code up a FUSE client we'd want to store things like ctime, UID, GID, etc. THe list operation will show both files and sub-directories)

The code was trivial once I got the hang of Sinatra, and I'm pretty pleased with it so far. I don't yet need to use it for anything, but I'm thinking of unifying the way that I store images on a couple of sites - and fetching them via JSON and Javascript might be an option this was an experiment in that direction. (Though I'd probably want to hook in rsync so we replicated the eventual upload location for safety.)

In other news I've been all organized and upgraded the kernel on my guest:

steve@steve:~$ uptime
 22:00:28 up  4:18,  1 user,  load average: 0.14, 0.05, 0.05
steve@steve:~$ uname -r
3.2.0-kvm-hosting.org-i386-20120106

So for once I'm up to date with a cutting edge kernel. Happy times.

ObQuote: "How you expect to run with the wolves come night when you spend all day sparring with the puppies? " - The Wire (Omar)

| 2 comments.

 

It is an army bred for a single purpose

Mon, 9 Feb 2009 12:32:38 GMT

It is funny the way things work out when you're looking for help.

Recently I was working on a Ruby + FUSE based filesystem and as part of the development I added simple diagnostic output via trivial code such as this:

@debug && puts "called foo(#{param});"

That was adequate for minimal interactive use, but not so good for real live use. In real live use I started outputing messages to a dedicated logfile, but in practise became overwhelmed by thousands of lines of output describing everything ever applied to the filesystem.

I figured the natural solution was to have a ring-buffer. (Everybody knows what a ringbuffer is, right?) It could keep the last 500 messages and newer debug information would just replace older entreis. That'd be just enough to be useful if I had a problem, but not so overwhelming it would get ignored.

In Perl I found a nice ringbuffer library, but for Ruby nothing. Locking a region of shared memory via shmget, shmset and keeping an array of a few hunded strings would be simple, but it seems odd I have to code this myself.

I started searching around and I accidentally stumbled upon the unrelated IPC::DirQueue perl module. Not useful for my ringbuffer logging problem, but beautifully useful.

There is no package for Debian but that was easily created:

dh-make-perl --build --cpan IPC::DirQueue

Already I have a million and one uses for it - not least to solve my problem of maintaining a centralised quarantine for all the spam mail rejected by N MX machines. (Which currently uses a combination of rsync and lockfiles.)

This is the reason why sites like Perl Advent Calendar are useful - they introduce a useful module every day or two, and introduce you to thinks that you can use in the future.

Of course keeping a sustainable site like that up and running is hard which is why sites like debaday struggle to attract contributors, for example.

Anyway random happyness.

ObFilm: Lord of the rings: Two Towers

| 3 comments.

 

When the light begins to change

Tue, 30 Jan 2007 12:32:38 GMT

All being well I now have a blog working using Typo, instead of Wordpress 2 – and no broken links!

We’ll see.

Now to make sure that my server load doesn’t rise through the roof this time.

For future reference importing entries from Wordpress2 -> Typo will almost certainly give you this error:

1
2
3
4
5
const_missing': uninitialized constant TestRequest (NameError)
        from ./db/converters/../../config/../app/models/blog.rb
...
..
.

This can be fixed by changing:

1
2
3
4
5
def really_send_pings(serverurl = blog.server_url, articleurl = nil)
   return unless blog.send_outbound_pings
   ...
   ...
end

To this:

1
2
3
def really_send_pings(serverurl = blog.server_url, articleurl = nil) 
   return
end

| No comments

 

Spiral Logo

Search

Recent Posts

Recent Tags

Links

RSS Feed

  • Subscribe to feed