Drag Racing Against THTTPD Using Siege

This is a simple and informal test of Wasp, comparing the speed of its HTTP server against that of THTTPD, considered to be one of the fastest trivial webservers in the world. 

Test Methodology:

All tests are run on my personal PC, in Ubuntu 8.10, running 32-bit, with a 2.4 GHz dual core Athlon and 2 GB of RAM.  In short, a typical end-user machine as of 2007.  The version of Wasp Lisp used is the bleeding edge, as of 2008/08/04, compiled with FAST=1, and the version of thttpd is 2.25b, also the latest and greatest.

The test server for Wasp is:

(import "lib/http-file-server")
(offer-http-file 2080 "/test" "text/plain" "Hello world!"))

And the command line for THTTPD is:

thttpd -d /tmp/thttpd/ -p 1080

As you can see, I am not spending a lot of effort tweaking either of these servers; THTTPD, written in C, should provide an idea of the upper limit of speed for these tests.  The test application is Siege 2.66, which is not terribly complicated and works fairly well in my experience, running ten iterations of fetching http://localhost:1080/test using 20, 100 and 1000 concurrent connections.  (The latter being more a torture test of Siege and the Kernel than our webservers..)

20 Concurrent Connections, Ten Iterations:

    WaspIO:
        Elapsed time:                9.83 secs
        Transaction rate:           20.35 trans/sec
        Longest transaction:         3.00
   
    THTTPD:
        Elapsed time:                9.03 secs
        Transaction rate:           22.15 trans/sec
        Longest transaction:         0.01

Neither server is really showing any load at this point; things are ticking over very nicely for WaspVM, because most of its GC is going on in the idles.

100 Concurrent Connections, Ten Iterations:

    WaspIO:
        Elapsed time:               26.05 secs
        Transaction rate:           38.39 trans/sec
        Longest transaction:        21.01

    THTTPD:
        Elapsed time:                8.10 secs
        Transaction rate:          123.46 trans/sec
        Longest transaction:         0.08

As you can see, here, the GC is starting to punish occasional transactions.  WaspVM is currently using a Stop the World GC -- completely non-generational, with no tuning since the fork from Mosquito.  It is, in short, very inefficient.

250 Concurrent Connections, Ten Iterations:

    WaspIO:
        Elapsed time:               35.30 secs
        Transaction rate:           70.71 trans/sec
        Longest transaction:        21.22

    THTTPD:
        Elapsed time:                9.65 secs
        Transaction rate:          259.07 trans/sec
        Longest transaction:         0.06

Same sort of results.  Wasp is still scaling up pretty nicely, handling the load.  At this point, Siege is the bottleneck, it cannot allocate enough memory to drive its state machines for more than 250 concurrent sessions.  THTTPD, of course, is still not really breaking a sweat, due to its use of the same asynchronous network event loop used by Wasp.

Summary:

Wasp is holding its own right now under load, doing well except for periodic bouts of high latency when its garbage collector has to work overtime to catch up with the VM.  A possible improvement for low-load situations is forcing a garbage collection cycle whenever Wasp is about to go into deep sleep waiting on I/O events.

Unfortunately, any deep tuning of the Garbage Collector is scheduled after the 1.0 release.