Joe’s Blog!

Update 1: Retest data (using different machine and Erlang kernel polling) added near bottom of post.

Update 2: More details and testing on the weird MochiWeb kernel polling results, bottom of post.

Almost a year ago I did some Apache and Nginx performance testing. Apparently I have the bug again and have done some performance testing on Nginx, Yaws and MochiWeb. The latter two being Erlang based. Again deathmatch may be an overstatement but this is my attempt at gleaning some interesting performance data from some high performance web servers. Also, I attempted to improve the graphs this time around since they were a bit hard to read the last time.

The Setup:

I was not able to use the same server and setup as the last time, so comparing between this and my last deathmatch probably isn’t very accurate. For this test I used a Intel Dual Core 2.2GHz, 4GB RAM machine running Ubuntu 8.10 (64bit) and for the test server. Erlang (R12B-3), Yaws (1.77) and Nginx (0.6.32) are installed from the standard repository and mochiweb from subversion (rev 88). All are using the default configurations outside of adjusting listening port numbers. The test is again against a basic robots.txt file. The tests were done using a consumer grade 100mb switch and all tests originated from an old laptop I had laying around. I think that about covers the test bed, if you have any questions let me know.

For the tests I used autobench (httperf under the hood) with the following command, each test ran ten minutes apart. The order of the tests were done in was MochiWeb then Yaws and lastly Nginx.

autobench –single_host –host1 HOST –port1 PORT –uri1 /robots.txt –low_rate 10 –high_rate 200 –rate_step 10 –num_call 10 –num_conn 5000 –timeout 5 –file SERVER-results-`date +%F-%H:%M:%S`.tsv

The Results:

There are a few results from httperf/autobench that I would like to show, errors, network I/O, reply rate (and it’s standard deviation) and response time. (click on the graphs for a larger view)

MochiWeb and Yaws both seem to be the most consistent here. Nginx had a couple of funky spikes, I do not know if this was an issue with Nginx or with my tests and/or test bed. Take from it what you will.

Nginx seems to use a bit more network I/O consistently through the lower ranges of this test and then again as some spikes. MochiWeb and Yaws seem to have some inconsistencies as well.

The reply rate and network I/O graphs certainly seem to be tied, which would make sense. Edit: Average reply rate is average replies per second.

In the higher reaches of the tests Yaws seems to be most consistent.

MochiWeb seems to have consistently the highest response times with Nginx has the lowest. This also follows the data from the first deathmatch. Nginx had consistently low response times against Apache. Edit: Response time is how quickly replies are sent in milliseconds.

Next up are the system graphs, I have CPU usage (both cores combined), context switches, interrupts and load. To help read these please note recall that each test ran ten minutes apart and the order of the tests was MochiWeb then Yaws and lastly Nginx. The data was gathered using sar at five minute intervals and graphed using ksar.

It seems Nginx is the clear winner here. Kernel polling may be the answer here, a retest may be in order to see if it makes a difference.

MochiWeb and Nginx seem pretty even on context switches with Yaws a little higher. I suppose turning on kernel polling might make this a bit more even, since Erlang and Nginx both use epoll. This may also account for the CPU usage difference above.

Interrupts are fairly even across all of them.

Again Nginx takes it, again likely due to kernel polling being disabled. That’s my best guess anywho.

The data I used to create the graphs and etc is available here.

Let me know if you are interested in me retesting anything, I may try to enable kernel polling and try again if I get a chance.

Note that these are *my* experiences with each webserver, your testing and experiences may be different. As with most things there are pro’s, con’s, trade offs and pitfalls. The only way to find out what will work best for your environment is to test, test and test.

Update:

I performed the upper half of the tests again to see if there were any changes to sporadic jumps in the graphs http performance graphs. My initial test using the old laptop I saw the same results. I then ran the tests from a VM (running Ubuntu 8.10 in a KVM VM) on my dual core machine and found that the results were much more even. Unfortunately it’s the same machine that the webservers are running on but the results look much better. The first set is using the same setup as before but just adjusted to have the top half test. The second is the same test but with kernel polling turned on in Erlang.

All of them are very even and close, no real winners here.

Looks like Nginx is the clear winner with Yaws next, followed by MochiWeb.

Pretty much the same as last time (likely a little higher across the board due to running the tests in a VM on the same machine). Note that Nginx is a system process, so for Yaws and MochiWeb follow the blue line and Nginx follow the green.

About the same as before, other than being higher due to running a VM.

Pretty much the same as before again, Nginx seems the lowest.

Now for the tests with kernel polling enabled in Erlang (erl +K true).

With kernel polling on it looks like Yaws actually performs better in the reply rate test with MochiWeb performing worse and Nginx in the middle

In the response time test a huge change is noted, MochiWeb goes from roughly a ~14 ms response time at 2000 requests to ~65 ms. Also noted Yaws performs much better matching or beating Nginx.

With kernel polling in the Erlang webservers Nginx still seems to come out on top for CPU usage.

Following the performance trend we saw above Yaws sees a drop in context switches and MochiWeb increases.

Load-wise things stay roughly the same with Nginx being the lowest.

While it certainly seems that my old laptop that I did the original tests on is too slow or has a network issue, hopefully with these new tests we have some more clarity. It seems that Yaws improves with kernel polling enabled and competes well with Nginx. MochiWeb on the other hand apparently has issues with kernel polling and actually degrades performance. If anyone has more info on the internals of MochiWeb and possible causes I would be certainly interested.

If anyone would like the data from the second round of tests it is available here.

Update 2:

I did some more testing to see what the issue might be with MochiWeb, response times and kernel polling. I did a few tests with different versions of Erlang, with and without kernel polling and testing from within and outside a KVM VM. From what I can tell the issue seems to be isolated to testing from within a VM with MochiWeb and kernel polling. Seems to be sorta strange but all my testing and retesting shows the same issue. Just to be clear on my setup, I am running httperf from with in a VM to MochiWeb running outside the VM. Here is the latest round of testing to show this point.

Even though the numbers are higher from within the VM without kernel polling, it certainly seems to be an issue with the combination of MochiWeb, KVM and kernel polling. Since I did not see the same spike from within a VM in the earlier tests with Yaws and kernel polling I assume it is not an issue with Erlang or it’s kernel polling mechanism conflicting with KVM. I am not entirely sure what to make of this other than MochiWeb, kernel polling and KVM don’t play well together and that kernel polling actually helps MochiWeb significantly when KVM is not involved. If anyone has any ideas on why that may be I am all ears.