Steve Kemp's Blog Writings relating to Debian & Free Software

Blogspam moved, redis alternatives being examined

Thu, 10 Jul 2014 12:32:38 GMT

As my previous post suggested I'd been running a service for a few years, using Redis as a key-value store.

Redis is lovely. If your dataset will fit in RAM. Otherwise it dies hard.

Inspired by Memcached, which is a simple key=value store, redis allows for more operations: using sets, using hashes, etc, etc.

As it transpires I mostly set keys to values, so it crossed my mind last night an alternative to rewriting the service to use a non-RAM-constrained service might be to juggle redis out and replace it with something else.

If it were possible to have a redis-compatible API which secretly stored the data in leveldb, sqlite, or even Berkley DB, then that would solve my problem of RAM-constraints, and also be useful.

Looking around there are a few projects in this area nds fork of redis, ssdb, etc.

I was hoping to find a Perl Redis::Server module, but sadly nothing exists. I should look at the various node.js stub-servers which exist as they might be easy to hack too.

Anyway the short version is that this might be a way forward, the real solution might be to use sqlite or postgres, but that would take a few days work. For the moment the service has been moved to a donated guest and has 2Gb of RAM instead of the paltry 512Mb it was running on previously.

Happily the server is installed/maintained by my slaughter tool so reinstalling took about ten minutes - the only hard part was migrating the Redis-contents, and that's trivial thanks to the integrated "slave of" support. (I should write that up regardless though.)

| 5 comments.

 

Comments On This Entry

[gravitar] antirez

Submitted at 09:28:30 on 10 July 2014

Hello Steve, did you evaluated different data layouts to save memory? Sometimes it is possible to do wonders.

[gravitar] Steve Kemp

Submitted at 09:31:06 on 10 July 2014

Honestly no.

The priority was getting the system back up and running when RAM-exhaustion caused it to die. (That was achieved by taking down another KVM-guest on the same host, and assigning the freed memory to the blogspam guest).

It did cross my mind that I could shave back some RAM by renaming keys - e.g. changing a key from "blacklist-1.2.3.4" to "b-1.2.3.4", etc, but I figured that if that solved the problem it wouldn't do so indefinitely.

I am going to reassess the RAM-usage and see if I can be more efficient, but it'll probably take me until the weekend to find the time to do anything significant.


[gravitar] postgres_sqlite

Submitted at 12:11:12 on 10 July 2014

Some links that migth interest you
- www.slideshare.net/amdunstan/pgcon-redis
- http://www.pgxn.org/dist/redis_fdw/
Postgres is not so difficult to setup,
it justs sounds difficult (try to pronounce it :)
Sqlite is even easier, but postgres handles total system crashes much better, as this seemed to be important for you?


[gravitar] Tuyan Ozipek

Submitted at 12:07:53 on 13 July 2014

Hi Steve,

Redis should already provide data persistence. Did i miss something in your blogpost , or did you already try it with no joy?

http://redis.io/topics/persistence

Cheers.

[gravitar] Steve Kemp

Submitted at 12:47:46 on 13 July 2014

By persistence here I meant "cope with datasets larger than the RAM on the host". i.e. Persistence on-disk, or similar.

Redis does provide persistence, and it continues to serve me very very well on several hosts. Just one particular service has so much data that it wouldn't fit in RAM any more.


 

Comments are closed on posts which are more than ten days old.

Spiral Logo

Search

Recent Posts

Recent Tags

Links

RSS Feed

  • Subscribe to feed