Benchmarks

[Chris Davies]

On Thu, 2004-04-29 at 14:38, Miles Elam wrote:
> Has anyone made any speed/CPU usage comparisons between TUX and Apache 
> 2.x?  I am especially curious about comparative flat file performance 
> with the 2.6 kernel, TUX, and Apache with the worker MPM.  The last 
> comparisons I've seen were between Apache 1.3.x on the 2.4 kernel.  
> Since Apache 2.x uses sendfile(...) these days, I guess what I'm really 
> asking about is the overhead of the context switch to userspace.

I'm not a fan of benchmarks, however, I have been running
apache2/mpm-worker behind tux and when the machine does <10mb/sec, there
is a negligable difference between tux and apache serving.  Get above
15mb/sec and you start to see tux pull ahead.  At 20-30mb/sec, tux
really seems to do the trick and is considerably faster than apache. 
Apache was not engineered for speed, but mpm-worker is indeed the
fastest model that they have.

Some odd statistics:

at 8mb/sec, we were at 20% CPU utilization
at 12mb/sec, we were at 18% CPU 
at 16mb/sec, we were ~15%
at 30mb/sec, we generally sit around 12%

I don't know why.  Sites are heavily biased towards two IP addresses out
of 16 on that machine.  This is running 2.4+tux.

For 2.6 tux, there are some improvements, apache seems to pick up some
speed, probably through some of the 2.6 improvements.  I've not served
enough data with 2.6 to have any real impressions.

I don't have enough large files (and I have tux set not to touch
anything >10mb anyhow) to really see what would happen over a long
download.

Since you're using mpm-worker, I guess you don't need php since php4
requires apache2-prefork.  Even so, we haven't found
apache2-prefork/php4 to be a stable combination anyhow.

Not that tux isn't extremely fast. We are using it to squeeze extra
performance out of existing hardware.  About the only userspace
webservers that I have seen that come close are X15 by Chromium, Zeus,
and perhaps thttpd (which does support php4 sapi).  There's been mention
of another one, fthttpd or something, but we haven't worked with it.

http://www.spec.org/web99/results/web99.html

Again, spec charges some money for people to buy the test suite, and I
doubt many open source projects have the cash to throw there for
bragging rights.  I think Chromium is in there, IBM may have tested
apache2, but IIS, Tux and Zeus seem to dominate the list.  Even Tux 1.0
has some pretty respectible numbers.  The other problem I have with Spec
is that it is very difficult to get a decent comparision of two
machines.  Almost noone runs the same test on the same hardware with
different web servers.

Besides, the test suite is supposed to simulate average traffic.  I
average traffic on the machines we deal with are 25 images + 1 html per
pageload.  Images range from 2-8k, html is generally 12-14k and
processed through a module we wrote.  I have another machine that we are
experimenting with that has much larger images and less html.  Even with
that one, there is quite an improvement over apache.

Yes, we have heavily tweaked apache's config, so, its not like we're
going with an unconfigured apache versus tux.  If you want more out of
apache, also consider the 10K patches by that ex-SGI guy.  Although I
think most of those patches are for 1.3 only.

For a true test, run mpm-worker, get it set up and running.  Then,
change the Port to 81 (or 8080 or whatever), stop apache, turn tux on,
restart apache and see.  It is very easy to turn tux on and off without
much interruption.