Deathmatch may be an overstatement but here are the results from some performance benchmarking.
The Setup:
Server:
- CENTOS 5.1
- Dual 2.4GHz Xeon CPUs
- 4GB RAM
- RAID5 (4 x 15k disks)
- Server and test client were connected via a consumer grade 10/100 switch
Configurations:
- Basic static vhost
- Keepalive turned on and with timeout of 15 seconds
- GZIP turned on
I used autobench to perform the tests. Basically this is a perl script that sits on top of httperf and will run multiple tests in succession outputing the results to CSV. Awfully convenient.
All the tests were run against the same robots.txt file. Autobench ran the following command 20 times incrementing the request rate by 10 each time. I started at 10 requests per second and went up to 200.
httperf –timeout=5 –client=0/1 –server=HOST –port=80 –uri=/robots.txt –rate=X –send-buffer=4096 –recv-buffer=16384 –num-conns=5000 –num-calls=10
I performed two samples and arbitrarily used the second as the results shown here. At the bottom of this post I will have spreadsheet containing the data from these tests so you can check out all the results.
The Results:
Both web servers performed well in all the tests and had no issues completing the requests. So I will not mention the metrics that they finished very closely on, only the ones that they did not have similar results.
There were three httperf related tests that Nginx and Apache differed on more than small amount, reply rate, network I/O and response time.
This one really piqued my interest. It seems strange to me why we would see such a result from Apache. In both tests there was a big difference at the 700 request mark. Statistically the difference was only on the max reply rate. The average and minimum are within a few tenths of a percent consistent through the tests. The max for Apache in the first test was 734.7 and in the second 758.7, standard deviations of 13.9 and 22.9 respectively. I suppose the real question here is whether or not this is my test or how Apache acts. If it is the later it seems strange that dealing with 700 requests would be any different than dealing with 800. From 800 requests to 2000, the larger differences in these results seems more realistic, controlled and gradual.
The network I/O graph I find interesting mostly because I don’t know how to take it. On one hand it seems Apache is simply using more bandwidth to do the same number of requests as Nginx. Which would seem bad. On the other it could just mean that Apache does a better job of consuming and using the available pipe. Which would seem good. My hunch is that it is the former.
The response times are also interesting since Nginx responds consistently at 0.4 ms. I am not sure why this is since I don’t know the internals to Nginx but I imagine that it is something that is built into how it deals with requests.
While the httperf tests were running I collected sar data from that time. The results show that Nginx uses a good amount less CPU and produces equally less load.
Apache:
CPU:
Load:
Nginx:
CPU:
Load:
Thats all I got, pretty cool. Nginx seems to compete pretty well with Apache and there doesn’t seem like there is a good reason not to use it especially in CPU usage constrained situations (ie huge traffic, slow machines and etc).
Here’s my results spreadsheet for the detailed results of each httperf metric.
Hey Joe,
Did your test system have dual core, I hear Nginx will see both cores, I have been playing around with Apache and can only get it to see one.
Hi Uncle Ed,
It was a dual CPU (single core) system.
I think nginx you can setup multiple workers to deal with requests if you have more than one core or cpu.
Check out (search for “worker_proceses”):
http://wiki.codemongers.com/NginxMainAndEventModules
As far as Apache goes it shouldn’t be worried about the type of CPU. It will just spin off child processes on whatever core or CPU is available at that time. So you can have numerous Apache processes running and have them on multiple cores/cpus.
For more info check out: http://httpd.apache.org/docs/2.0/mod/prefork.html#how-it-works
Hope this helps.
On the net_io graph, I’m wondering if Apache is sending more headers in the response than Nginx. That might account for the extra (and increasing from more requests) bandwidth usage.
You used the Prefork MPM? Why not Worker or Event?
Where are the config files anyway? What version of Nginx and Apache? Are these packages or builds, and if builds what options were used?
Dan P : Thats a good point, it is something I was wondering myself. Since I used a small file to test with the headers would constitute a large percent of the traffic which may be the cause of the difference.
Anynymous: The default redhat/centos configuration packages and versions were used for the tests. The only change was to turn on gzip in both cases and turn on keepalives in Apache. Nginx has these on by default. Hope this helps.
joe: thanks for the reply.
It’d be appreciated if you’d post the configs in the future. A lot of us don’t have Redhat boxes laying around. Even if we did, it’s a lot easier to read exactly what YOU had if you post it.
And of course this blog post will probably last longer than the current version of a distro. When someone visits in one or two years time, will the stock versions and configs be the same?
Hey Joe,
I appreciate your work but you are comparing a Suzuki Swift with a Lexus LS and that’s simply because Apache is dynamic in any respect and that’s what lighttpd and nginx are missing though it makes sense they exist. In fact you can even trim Apache down to the state of lighttpd and nginx but then again you would loose a lot of functionality which is mandatory in most setups.
If you read the wiki or irc channel of lighttpd e.g. you might notice that there are situations where lighttpd and probably nginx are slower because of their nature. Apache in contrast tries to serve the needs of anyone in the default configuration and this of course can’t be the fastest one though you _can_ configure apache for your needs and that’s where apache’s flexibility comes in.
My experiences with nginx and lighttpd are very positive because on slow machines I really could save the cpu’s life 2 years ago but today I changed my setup in the way that I use Apache for dynamic content (for webapps) and lighttpd and nginx for proxying and static content. There is nothing easier to proxy with lighttpd and nginx to an apache instance that serves dynamic content and especially for slow clients it would save your ass and even primitive loadbalancing is possible in minutes without anyone taking your whole money
.
But whenever you plan to use lighttpd and nginx standalone keep in mind that configuring lighttpd and nginx in the first run appears very easy … just because it _is_ easy but whenever you need something special you need to get in touch with the wiki or the developer team to ask _if_ it’s possible and if it works.
Another possible problem with nginx especially is the fact that the development is done by only one person so even if companies are using nginx to proxy to their apache or iis farm (:P) I recommend lighttpd as its community is much more involved and reports bugs very fast.
All these things you have to take into account and my advice is to use the best of both nginx and apache or lighttpd and apache.
But if we are honest anything I said to be an advantage of lighttpd and nginx can also be done with apache itself. It’s just a matter of configuring and compiling your apache for your special needs. The default configuration most linux distributions ship with is for the broad mass and not for the one being in the need of maintaining a high-load webserver.
Still I prefer lighttpd and nginx for proxying simply because I can’t sleep well with two different apache instances fiddling around
…
Andreas: Thanks for your comment. I couldn’t agree more. This is why I attempted to shy away from making a suggestion or conclusion on which is better. Each situation, work load and environment may have a use for any of these three web servers. Testing is the only true way to find out which will work best in a specific case.
Joe, your test result simply helped me a lot and saved a lot of time for me not to build up another test lab to compare nginx and apache. I just want to know more about comparison between nginx and lighttpd but someone had done this before, but the report is a little bit old (at http://blog.kovyrin.net/2006/08/22/high-performance-rails-nginx-lighttpd-mongrel/), thanks all of you.
Thanks, I am glad it was helpful. I haven’t had a chance to do an in depth comparison between lighttpd and nginx like this one. I will say that I have liked working with nginx better than lighttpd (manageability, config files, config options and etc).
Hey
I try to understand the different between –rate and –num_call. When you use
“httperf –num-conn 10 –num-call 1 ” in apache’s log you can see 10 connections, when
“httperf –num-conn 10 –num-call 2 ” – 20 connection’s
but when you increase the “–rate”, number of connection (in access_log) don’t change.
So how to simulate the “users”? Maybe autobench should be run in script with different –num-cal?
thanks.
There are a number of ways you can modify the work load to simulate users and take in account the type of traffic you are dealing with. As you noted the rate and num-calls variables can do this but also modify wsess and also burst-length and/or think-timeout.
To your question, rate “specifies the fixed rate at which connections or sessions are created.” and num-calls “specifies the total number of calls to issue on each connection before closing it.”
Check http://www.hpl.hp.com/research/linux/httperf/httperf-man-0.9.txt for more information. It has concise descriptions of each option and is really helpful.
Thank you very much for this great benchmark. I think nginx become more and more important in future. It’s really fast and very reliable…
Thank you for the detailed report and for the follow up comments. This has greatly helped me understand a few things I was unsure about. Currently using Apache and was considering lighttpd or nginx. Thanks!
Marco: if you want a more recent comparison between nginx and lighttpd, check out http://www.wikivs.com/wiki/Lighttpd_vs_nginx and http://blog.grik.net/2007/08/nginx-vs-lighttpd-web-server-of-choice.html.
Great post, very interesting. I´ve heard about nginx in the past, but I didn´t use it. After this post – I think it´s time to do some testing. In my opinion, apache is a good product but additional competition isn´t that bad…
Great!! and thanks….
Is nginx can be manage by webmin on Mandriva 2009 system.?
Cause i’m a newbie in Linux & mostly i use webmin to configure my server
@ovan, I’m not all that familiar with the capabilities of webmin. I would check their docs.
Hey! Wake up, all these benchmarks are just benchmarks .. have you seen a hosting company use nginx web servers ? that would be a “no” i guess ! prolly in the future we will have the pleasure to have a choice, but now-a-days the webserver is only one and its name is Apache. Cheers!
By using the default RedHat configuration
of Apache, these tests become worthless for looking
at a high performance web server.
First, the default configuration is loaded with
perhaps a hundred modules which you won’t use.
RedHat simple loads just about every module
ever made just in case someone wants to use it.
Commenting out even the obvious ones significantly
improves performance.
Other basic settings like the maximum
number of simultaneous requests are also not
set for performance by default.
I useful comparison would use a properly configured
Apache set up for performance.
I wonder how to get the best of both worlds since Apache has by far more features than Nginx or lighttpd.
I’ve read about reverse proxy setup perhaps this would be a good solution…
Andreas Schipplock said:
“I appreciate your work but you are comparing a Suzuki Swift with a Lexus LS and that’s simply because Apache is dynamic in any respect and that’s what lighttpd and nginx are missing though it makes sense they exist. In fact you can even trim Apache down to the state of lighttpd and nginx but then again you would loose a lot of functionality which is mandatory in most setups.”
I’d say he’s comparing a Lexus LS and a Ferrari (if we must compare to cars). The Lexus does have a lot of niceties that the Ferrari doesn’t, but it’s also not tunable to get to 190MPH either. It isn’t just a simple case of removing the seat heaters to “trim it down”. There are deep design decisions that affect the ability of the Lexus to reach those speeds.
Similarly, Apache is process-based and Nginx (and Lighttpd) are asynchronous. This means that under load, Apache is guaranteed to do one of two things (most likely both): use tons of RAM or drop requests, because of the inherent overhead in using threads (specifically stack allocation and context-switching). Nginx and Lighttpd were written specifically to address the problems facing threaded servers under heavy load. Apache’s problems under heavy load aren’t the result of featuritis, but rather a deep architectural problem that can’t be easily addressed. On the other hand, both Nginx and Lighttpd can (and certainly will) eventually adopt all the features of Apache without heavy penalty.
In any case, there is definitely a place for Apache if you need particular modules (mod_svn, mod_jakarta, etc), but I think that this is more of a niche and that Nginx can serve the needs of the majority of websites if people are willing to trade a small learning curve in exchange for massively reduced resource utilization.
There is a flaw in your test.
The pipe used is simply not big enough to stress anything. If you don’t have a gig switch, try using gig nics with a crossover cable.
Apache wins
That’s really nice article. Thanks for your great work!
well i am still learning about nginx, i am thinking of putting it up as proxy in front of apache to serve static files
Ty
TrustLeap G-WAN is up to 20 times faster than both.
May be worth giving it a try:
http://trustleap.com/
nginx does scream, especially with fastcgi. Seems to use less resources too.