Skip to content

Entries tagged "redis".

Goodbye mysql ..

22nd November 2011

Yesterday evening I updated my server to remove MySQL:

steve:~# dpkg --purge mysql-client-5.1 \
                      mysql-common     \
                      mysql-server-5.1 \
                      mysql-server-core-5.1 \
                      python-mysqldb        \
                      libdbd-mysql-perl     \
                      libdatetime-format-mysql-perl

Until last month I had two database in use, one each for a pair of web-applications. As of now one is using redis - which I'm already using for my image hosting - and the other application is using SQLite.

Until recently I had a high opinion of SQLite, although that has now been downgraded a little, it is still a thoroughly excellent piece of software. I was just surprised at little things it was missing, to the extent I had to rewrite my applications SQL.

Still one less service is a good thing, and the migration wasn't so painful..

In more productive news I recently acquired a nice external flash - the Yongnuo YN-460 II is (very) cheap and cheerful, it can be fired remotely with my triggers so I've had a lot of fun with opportunistically taking pictures and experimenting with lighting.

Most of the results are NSFW, but there are some other examples lurking around including the first time I managed to successfully capture a falling water-drop. (Not the best picture, not the most explicit effect, but fun regardless. I both can and will do better next time!)

Somebody recently asked me to write about "camera stuff under linux" and happily I declined.

Why decline? Because there are so many good tools, applications, and utilities. (I use local tools for organisation and duplicate detection, rawtherapee for RAW conversion and GIMP for touchups). Having many available options is fantastic though, and something hard to appreciate for "newcomers" to Linux.

(Yeah I waited 90 seconds - if I remembered to add -nojava - for Netscape Navigator to start, under X10, with 8Mb of RAM. Happier days are here. Sure DRM is bad, secure boot .. an open question, but damn we have it good compared to almost any previous point in time!)

ObQuote: "Yeah, obviously it is only a tactical party. I'm only having a party to eventually get sex." - Peep Show

Tags: camera-toys, lighting, mysql, random, redis. | 2 comments.

 

I updated my redis-based filesystem

2nd March 2011

In July last year I made a brief post about a simple filesystem I'd put together which used Redis for the storage.

At that time I thought it was a cute hack, and didn't spend too much time with it. But recently I found a use for it so I cleaned it up, synced up the C client for Redis which I used and generally started to care again.

If it is useful you can now find it online:

The basic idea is the same as it was before, except I did eventually move to an INODE-like system. Each file/directory entry receives a unique identifier (integer) - and then I store the meta-data in a key based off that name.

This means for a file I might have keys, and values,like this:

KeyValue
INODE:1:NAMEThe name of the file (e.g. "passwd").
INODE:1:SIZEThe size of the file (e.g. "1661" )
INODE:1:GIDThe group ID of the file's owner (e.g. "0")
INODE:1:UIDThe user ID of the file's owner (e.g. "0")
INODE:1:MODEThe mode of the file (e.g. 0755)

To store these things I use a Redis "SET" which allows me to easily iterate over all the entries in each directory.

ObQuote: "They fuck up, they get beat. We fuck up, they give us pensions. " - The Wire

Tags: fuse, hacks, random, redis, redisfs. | 3 comments.

 

I've accidentally written a replication-friendly filesystem

29th July 2010

This evening I was mulling over a recurring desire for a simple, scalable, and robust replication filesystem. These days there are several out there, including Gluster.

For the past year I've personally been using chironfs for my replication needs - I have /shared mounted upon a number of machines and a write to it on any will be almost immediately reflected in the others.

This evening, when mulling over a performance problem with Gluster I was suddenly struck by the idea "Hey, Redis is fast. Right? How hard could it be?".

Although Redis is another one of those new-fangled key/value stores it has several other useful primitives, such as "SETS" and "LISTS". Imagine a filesystem which looks like this:

 /
 /srv
 /tmp
 /var/spool/tmp/

Couldn't we store those entries as members of a set? So we'd have:

  SET ENTRIES:/              -> srv, tmp, var
  SET ENTRIES:/var/spool     -> tmp
  SET ENTRIES:/var/spool/tmp -> (nil)

If you do that "readdir(path):" becomes merely "SMEMBERS entries:$path" ("SMEMBERS foo" being "members of the set named foo"). At this point you can add and remove directories with ease.

The next step, given an entry in a directory "/tmp", called "bob", is working out the most important things:

  • Is /tmp/bob a directory?
    • Read the key DIRECTORIES:/tmp/bob - if that contains a value it is.
  • What is the owner of /tmp/bob?
    • Read the key FILES:/tmp/bob:UID.
  • If this is a file what is the size? What are the contents?
    • Read the key FILES:/tmp/bob:size for the size.
    • Read the key FILES:/tmp/bob:data for the contents.

So with a little creative thought you end up with a filesystem which is entirely stored in Redis. At this point you're thinking "Oooh shiny. Fast and shiny". But then you think "Redis has built in replication support..."

Not bad.

My code is a little rough and ready, using libfus2 & the hiredis C API for Redis. If there's interest I'll share it somewhere.

It should be noted that currently there are two limitations with Redis:

  • All data must fit inside RAM.
  • Master<->Slave replication is trivial, and is the only kind of replication you get.

In real terms the second limitation is the killer. You could connect to the Redis server on HostA from many locations - so you get a true replicated server. Given that the binary protocol is simple this might actually be practical in the real-world. My testing so far seems "fine", but I'll need to stress it some more to be sure.

Alternatively you could bind the filesystem to the redis server running upon localhost on multiple machines - one redis server would be the master, and the rest would be slaves. That gives you a filesystem which is read-only on all but one host, but if that master host is updated the slaves see it "immediately". (Does that setup even have a name? I'm thinking of master-write, slave-read, and that gets cumbersome.)

ObQuote: Please, please, please drive faster! -- Wanted

Tags: filesystems, fuse, redis. | 6 comments.

 

I'm a CPAN author.

23rd July 2010

As of this morning I'm a published author on CPAN!

Thus far I have only a single module to my name, but that will most likely change in the future:

CGI::Session::Driver::redis

A module for storing (CGI) session data within a Redis database.

A while back I setup a dynamic website which was 100% redis backed, using my redis backports for lenny, and realised I needed somewhere to store the session data too. Hence this module.

I'll create a .deb package of the module, and stick it alongside the redis server.

ObQuote:I like to keep this handy... for close encounters.

Aliens

Tags: cpan, perl, redis. | No comments

 

Cover your heart! Cover your heart!

13th January 2010

I've long been a fan of Danga's Memcached, back in 2005 I wrote about it for the first time.

Recently I've been looking at the persistant version, memcachedb which is essentially a Berkeley Database using the memcached protocol as the transport layer. This allows you to get persistance for free, and means you can rely upon the content being present, rather than just hoping it will be there as an opportunistic cache.

However in this brave new world of nosql I've been mulling over different options for new projects / new toys. I have to say that memcachedb lost out for me, and that Redis is my preferred flavour of new hot.

Redis has a rich API allowing you to store interesting datatypes such as lists, and it allows both replication and distribution (neither of which I need just yet, but both are useful to have as options for the future).

The documentation is great including a well-documented Twitter clone, and the facilities that are available are a step up from memcached, albeit obvious extensions which you'd expect if you were doing interesting things.

The redis-server package is already part of Debian GNU/Linux, and creating .deb package of the Redis module was trivial.

So now that I'm familiar with it, what to use it for? Well I think the first thing will be the new 2010 Valentines site.

Over the years I've hosted several Livejournal Valentine sites, and this year is no different. (Why? I like matching people, and I like social stuff. This is why I did ctrl-alt-delete in the past, even if that failed it was a great learning experience and I'd do it again if I had the time/patience/more optimism).

So far the site only has a couple of users but I fully expect it to ramp up considerably before the magical day (ha!). In the past I've limited the lifetime of the site(s) from Feb 1st to Feb 14th and still managed to achieve 5000+ users. Right now I'm using flat files for storage specifically to avoid database-load issues (which caused me considerable pain one year).

I suspect the backend will be Redis-based in the very very near future, but in the meantime if you're open to amusement and use Livejournal then feel free to poke at it: ljvalentine.com.

ObFilm: Indiana Jones and the Temple of Doom

Tags: memcached, memcachedb, redis. | 7 comments.