Steve Kemp's Blog Writings relating to Debian & Free Software

That is it, I'm going to do it

Sat, 22 Mar 2014 12:32:38 GMT

That's it, I'm going to do it: I have now committed myself to writing a scalable, caching, reverse HTTP proxy.

The biggest question right now is implementation language; obviously "threading" of some kind is required so it is a choice between Perl's anyevent, Python's twisted, Rubys event machine, or node.js.

I'm absolutely, definitely, not going to use C, or C++.

Writing a a reverse proxy in node.js is almost trivial, the hard part will be working out which language to express the caching behaviour, on a per type, and per-resource basis.

I will ponder.

| 12 comments.

 

Comments On This Entry

[gravitar] Anonymous

Submitted at 15:53:12 on 22 March 2014

Might I recommend Warp as a starting point? Quite a nice framework for building anything HTTP based, and absurdly fast.

[gravitar] Sam

Submitted at 16:16:15 on 22 March 2014

Might I ask what has proven Nginx to be insufficient for your needs? I'm really happy with it as a caching proxy. Very interested in your outcome though - I've considered similar or related projects. Python's gevent seemed a better fit for me though.

[gravitar] Charles Darke

Submitted at 17:01:12 on 22 March 2014

I guess it's a good opportunity to learn Go language ;)

[gravitar] Steve Kemp

Submitted at 17:09:27 on 22 March 2014

The rules in nginx are basic, and you cannot do things like expire by prefix unless you pay for the commercial fork.

It should be possible to have flexible rules on caching. But not so flexible that you tie yourself in knots (e.g. the utter freedom and pain of using varnish)

[gravitar] Gopher

Submitted at 17:26:32 on 22 March 2014

This really sounds like a perfect match for golang.org … since it's a language to write servers – as long as one isn't afraid of pointers ;).

[gravitar] Steve Kemp

Submitted at 17:45:23 on 22 March 2014

I assumed the configuration would be in a DSL, or some other scripting language, so that made me ignore Go.

That said it isn't an unreasonable idea, because I've been interested in exploring Go, and it is threaded and "cool".

I'll take a look at Warp later. For the moment I'm putting the finishing touches to a Key-Value store, which uses an append-only logfile, and is written in .. perl.

[gravitar] Paul Tötterman

Submitted at 21:48:50 on 22 March 2014

Is tengine any better than nginx wrt configurability? Apparently it's a free fork with many of the features found in the commercial offering.

[gravitar] Steve Kemp

Submitted at 22:23:56 on 22 March 2014

tengine is something I've not explored. Perhaps I should, but I guess that the configuration will be lacking.

I assume I want to specify caching rules based on:

  • Client-request.
  • Server-response.
  • Conditional on cookies, http-code, etc, etc.

To me that implies some "scripting language", or DSL, rather than a static nginx config. (Which might allow some flexibility, but not enough.)

[gravitar] Abe Hassan

Submitted at 04:03:27 on 23 March 2014

I was going to comment with either Nginx or Varnish but I suppose a better thing would be to ask you ... which ones have you looked at and why have you ruled them all out? (I ask that, understaind the answer might be "it sounds like a fun problem to try to tackle" which is legitimate too.)

[gravitar] Steve

Submitted at 11:13:34 on 23 March 2014

nginx is too simple, as mentioned above there are invalidation problems, but equally there is little flexibility.

varnish is something that works really really well, but it is a pain to configure because it is too complex.

I've seen people struggle with it for years, and I don't like it.

So this is a fun problem to have.


[gravitar] Anonymous Coward

Submitted at 06:07:52 on 24 March 2014

Have you considered using a more exotic language like Erlang or Haskell?

[gravitar] Steve Kemp

Submitted at 09:39:40 on 24 March 2014

I'd not - just because I want some "scripting" configuration - I'm not sure if Haskell/Erlang would allow that.

Although I wish to avoid using C/C++ it might be that they're the most sane, since I could embed javascript/lua/similar to configure things.

 

Comments are closed on posts which are more than ten days old.

Spiral Logo

Recent Posts

Recent Tags

Links

Search

Search prior entries:

RSS Feed

  • Subscribe to feed