Nginx VS Nginx + Varnish

Server NGINX
1

Hey,

Do you think it’s a good idea to add Varnish to Nginx to serve static
files?

I’m running a dedicated server with 2GB RAM and a pentium dual core
E2180 for static files.

I’m only using Nginx 0.6.32 on Centos 5/linux 2.6.24: no database, no
php, no gzip mod…

Nginx sends 60 images/seconde (max 100 images/sec) among 3.200.000 jpeg
images (one image = 5-10kB).

Thanks in advance!

My Nginx file :

user nginx nginx;
worker_processes 4;

events
{
worker_connections 1024;
}

http
{
include mime.types;
default_type application/octet-stream;

access_log off;

sendfile on;
tcp_nopush on;
tcp_nodelay on;

client_body_timeout 5;
client_header_timeout 5;
keepalive_timeout 5 5;
send_timeout 5;

gzip off;

server
{
listen 80;

     root         /home/www;

     expires      90d;

     # CSS & Javascript files configuration
     location ~* ^.+.(css|js)$
        {
           gzip_static        on;
           gzip_http_version  1.1;
           gzip_proxied       off;
           gzip_min_length    0;
           gzip_disable       "MSIE [1-6]\.";
           gzip_vary          off;
        }
  }

}

2

From my experience, 60 small static files / second used 2% cpu on a
OS X Xeon Server. Are you suspecting that your server is being limited
in some way ?

Cheers

Dave

3

I think you should increase this, as you have more CPU and also you are
using 4 workers. Try to set this at lets say “worker_connections
2048;”. It might work.

4

Compression, gzip. You should try it, maybe for jpeg is not actually
worth it… Hmm, try what can i say.

5

Dave C. wrote:

From my experience, 60 small static files / second used 2% cpu on a
OS X Xeon Server. Are you suspecting that your server is being limited
in some way ?

Right now, I don’t have problems.

I just want to optimize my web server for the best result (it takes
400-500ms to serve a 5kB image).

http://img29.picoodle.com/img/img29/3/10/10/f_monitoring0m_f2441cf.png

http://img33.picoodle.com/img/img33/3/10/10/f_monitoring0m_bb25457.png

Those images show my web site performance (from pingdom.com and
site-perf.com).

I’m wondering if there is something to reduce those timings…

Thanks.

6

Have you actually tried putting varnish/ncache in place and benchmarking
the before/after results? As I’ve found with all server set-ups: YMMV.

Gen G. wrote:

400-500ms to serve a 5kB image).

Thanks.

Attachments:
http://www.ruby-forum.com/attachment/2797/monitoring00.png

–

Phillip B Oldham
The Activity People
[email protected] mailto:[email protected]

Policies

This e-mail and its attachments are intended for the above named
recipient(s) only and may be confidential. If they have come to you in
error, please reply to this e-mail and highlight the error. No action
should be taken regarding content, nor must you copy or show them to
anyone.

This e-mail has been created in the knowledge that Internet e-mail is
not a 100% secure communications medium, and we have taken steps to
ensure that this e-mail and attachments are free from any virus. We must
advise that in keeping with good computing practice the recipient should
ensure they are completely virus free, and that you understand and
observe the lack of security when e-mailing us.

7

Compression, gzip. You should try it, maybe for jpeg is not actually
worth it… Hmm, try what can i say.

I already use static gzip on css and javascript file.
And for jpeg, it is not worth it, indeed.

Moreover, if you look at the monitoring image (in attachment), the loss
of time is while sending request and receiving http-headers from server
(25% of the total time).

I was wondering if we can reduce it?

PS: watch the last line of the image in attachment: it’s the javascript
from Google Analytics, and the timing is really, really short!

8

Have you actually tried putting varnish/ncache in place and benchmarking
the before/after results? As I’ve found with all server set-ups: YMMV.

I’m trying to get tips and advices from experienced users like you
before diving in the benchmarking work.

And don’t worry, I will share my varnish+nginx benchmarks and
experiences at the end… ^_-

9

On Fri, Oct 10, 2008 at 02:03:50PM +0200, Gen G. wrote:

http://img29.picoodle.com/img/img29/3/10/10/f_monitoring0m_f2441cf.png

http://img33.picoodle.com/img/img33/3/10/10/f_monitoring0m_bb25457.png

Those images show my web site performance (from pingdom.com and
site-perf.com).

I’m wondering if there is something to reduce those timings…

What I see on these images is that TCP connection times are about
100ms-500ms. These values depend on client speed connection and kernel
internel processing only. User-level applications can not affect on it.
Thus, if you have 100-500ms TCP round trip, then small data chunks
will be processed in the same time, even if nginx will not block on disk
reads.

As to disk blocking, you have 30G images and 2G memory. nginx is
probably
blocking on disk - see “wa” percents in top. You may try to increase
worker_processes to 10-20.

As to using varnish+nginx, I think varnish will not help in this case.

10

oh yes I forgot, I have confronted one of the developers on #varnish
with my results. The said developer stated that nginx currently does
have the edge performance wise, even though they have the “superior”
architecture.

-Victor

11

I have tried both approaches. nginx on ZFS (arc caching) is superior in
performance to varnish in front of nginx. I can easily choke varnish
with many requests.

http://letsgetdugg.com/view/Nginx_hidden_performance_option

works very well for us in our production environment.

-Victor

12

On Sat, Oct 11, 2008 at 7:26 AM, Victor I. [email protected]
wrote:

–
Posted via http://www.ruby-forum.com/.

On Sat, Oct 11, 2008 at 7:30 AM, Victor I. [email protected]
wrote:

oh yes I forgot, I have confronted one of the developers on #varnish
with my results. The said developer stated that nginx currently does
have the edge performance wise, even though they have the “superior”
architecture.

and would this be on solaris only? or does it apply to other platforms
as well?

-jf

–
In the meantime, here is your PSA:
“It’s so hard to write a graphics driver that open-sourcing it would not
help.”
– Andrew Fear, Software Product Manager, NVIDIA Corporation

13

don’t know, benchmark and find out. I have seen nginx edge out varnish
in performance on solaris and freebsd.

-Victor

14

Serving thumbnails efficiently is not easy. See
YouTube Architecture - High Scalability - (scroll down to
‘Serving thumbnails’).

You’re spending a good chunk of time doing disk seeks. Even if read
time is 0, average seek time is about 10ms. So you’re looking at a
limit of 100 images/sec. This is the ideal case. Add read time + TCP
delays + other delays, you’re looking 50-60 images/sec.

Serve off memory, try using memcache.

-praveen

On Fri, Oct 10, 2008 at 5:03 AM, Gen G. [email protected]
wrote:

Attachments:
http://www.ruby-forum.com/attachment/2797/monitoring00.png

–
Posted via http://www.ruby-forum.com/.

–
Praveen

Can you yell “Movie!” in a crowded firestation?

15

Yes. There is caching at the OS level too, which needs to be tuned
(like cache sizes, inode vs content etc). I did not look at your
benchmarks, but was commenting more on Gen’s post. In the graphs he’s
given, the latency between connect and sending first byte (the green
part) is high, which might explain disk seek/read.

-p

On Tue, Oct 14, 2008 at 11:36 AM, Victor I. [email protected]
wrote:

umm, linux file cache? solaris ZFS ARC cache? who says you will be doing
disk seeks for frequently used files?

seems like varnish is clashing with what is already in place.

-Victor

Posted via http://www.ruby-forum.com/.

–
Praveen

Can you yell “Movie!” in a crowded firestation?

16

umm, linux file cache? solaris ZFS ARC cache? who says you will be doing
disk seeks for frequently used files?

seems like varnish is clashing with what is already in place.

-Victor