Skip to content

That is it, I'm going to do it

That's it, I'm going to do it: I have now committed myself to writing a scalable, caching, reverse HTTP proxy.

The biggest question right now is implementation language; obviously "threading" of some kind is required so it is a choice between Perl's anyevent, Python's twisted, Rubys event machine, or node.js.

I'm absolutely, definitely, not going to use C, or C++.

Writing a a reverse proxy in node.js is almost trivial, the hard part will be working out which language to express the caching behaviour, on a per type, and per-resource basis.

I will ponder.

Comments On This Entry

  1. [gravitar] Anonymous

    Might I recommend Warp as a starting point? Quite a nice framework for building anything HTTP based, and absurdly fast.

  2. [gravitar] Sam

    Might I ask what has proven Nginx to be insufficient for your needs? I'm really happy with it as a caching proxy. Very interested in your outcome though - I've considered similar or related projects. Python's gevent seemed a better fit for me though.

  3. [gravitar] Charles Darke

    I guess it's a good opportunity to learn Go language ;)

  4. [author] Steve Kemp

    The rules in nginx are basic, and you cannot do things like expire by prefix unless you pay for the commercial fork.

    It should be possible to have flexible rules on caching. But not so flexible that you tie yourself in knots (e.g. the utter freedom and pain of using varnish)

  5. [gravitar] Gopher

    This really sounds like a perfect match for golang.org … since it's a language to write servers – as long as one isn't afraid of pointers ;).

  6. [author] Steve Kemp

    I assumed the configuration would be in a DSL, or some other scripting language, so that made me ignore Go.

    That said it isn't an unreasonable idea, because I've been interested in exploring Go, and it is threaded and "cool".

    I'll take a look at Warp later. For the moment I'm putting the finishing touches to a Key-Value store, which uses an append-only logfile, and is written in .. perl.

  7. [gravitar] Paul Tötterman

    Is tengine any better than nginx wrt configurability? Apparently it's a free fork with many of the features found in the commercial offering.

  8. [author] Steve Kemp

    tengine is something I've not explored. Perhaps I should, but I guess that the configuration will be lacking.

    I assume I want to specify caching rules based on:

    • Client-request.
    • Server-response.
    • Conditional on cookies, http-code, etc, etc.

    To me that implies some "scripting language", or DSL, rather than a static nginx config. (Which might allow some flexibility, but not enough.)

  9. [gravitar] Abe Hassan

    I was going to comment with either Nginx or Varnish but I suppose a better thing would be to ask you ... which ones have you looked at and why have you ruled them all out? (I ask that, understaind the answer might be "it sounds like a fun problem to try to tackle" which is legitimate too.)

  10. [author] Steve

    nginx is too simple, as mentioned above there are invalidation problems, but equally there is little flexibility.

    varnish is something that works really really well, but it is a pain to configure because it is too complex.

    I've seen people struggle with it for years, and I don't like it.

    So this is a fun problem to have.


  11. [gravitar] Anonymous Coward

    Have you considered using a more exotic language like Erlang or Haskell?

  12. [author] Steve Kemp

    I'd not - just because I want some "scripting" configuration - I'm not sure if Haskell/Erlang would allow that.

    Although I wish to avoid using C/C++ it might be that they're the most sane, since I could embed javascript/lua/similar to configure things.