Ruby Forum NGINX > Weird timeouts, not sure if I've set the right threshholds

It is proxying (basically just load balancing) to 3 upstream nginx
webservers which do FastCGI/PHP/normal static files.

I get a lot of "upstream timed out" errors and I can't determine why
exactly. Perhaps my buffers are too high, too low, timeouts are too
high, too low? I bumped up the timeouts on my proxy machine higher
than I normally would set so it wouldn't timeout as often but it still
does. All the machines are quad-core xeons with 4GB RAM, SATA2 disks,
dedicated to web/fastcgi/php, all connected via a private gigabit
VLAN... the files are hosted on NFS, but I'm not seeing any errors
related to that in logs either..


nginx PROXY:

user www-data www-data;
worker_processes 4;
worker_cpu_affinity 0001 0010 0100 1000;
working_directory /var/run;
error_log /var/log/nginx/error.log error;
pid /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    upstream webservers {
        server web01:80;
        server web02:80;
        server web03:80;
    }
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    client_max_body_size 100m;
    client_header_buffer_size 8k;
    large_client_header_buffers 12 6k;
    keepalive_timeout 5;
    gzip on;
    gzip_static on;
    gzip_proxied any;
    gzip_min_length 1100;
    #gzip_http_version 1.0;
    gzip_comp_level 2;
    gzip_types text/plain text/html text/css application/x-javascript
text/xml application/xml application/xml+rss
    gzip_disable "MSIE [1-6]\.";
    gzip_vary on;
    server_names_hash_max_size 4096;
    server_names_hash_bucket_size 128;
    server {
        listen 80;
        access_log off;
        location / {
            proxy_pass http://mikehost;
            proxy_next_upstream error timeout http_500 http_503 
invalid_header;
            proxy_max_temp_file_size 0;
            proxy_read_timeout 50;
            proxy_connect_timeout 30;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_ignore_client_abort on;
        }
    }
}


nginx WEBSERVERS:

user www-data www-data;
worker_processes 4;
worker_cpu_affinity 0001 0010 0100 1000;
working_directory /var/run;
error_log  /var/log/nginx/error.log debug;
pid /var/run/nginx.pid;



events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    set_real_ip_from 10.13.5.16;
    access_log off;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    client_max_body_size 100m;
    client_header_buffer_size 8k;
    large_client_header_buffers 12 6k;
    keepalive_timeout  5;
    server_tokens off;
    gzip off;
    gzip_static off;
    server_names_hash_max_size 4096;
    server_names_hash_bucket_size 128;

# then an example vhost block
 server {
    listen 80;
    server_name michaelshadle.com www.michaelshadle.com;
    index index.php;
    root /home/mike/web/michaelshadle.com/;
    location ~ .php {
        include /etc/nginx/fastcgi.conf;
        fastcgi_pass 127.0.0.1:11000;
        fastcgi_index index.php;
    }
    if (!-e $request_filename) {
        rewrite ^(.+)$ /wordpress/index.php?q=$1 last;
    }
}


Any thoughts? Would turning off some buffers on the proxy make it
better, or would turning off buffering on the webservers make it
better? Not quite sure here where the best place would be to change
(if anywhere...)

Thanks..

On Wed, Apr 30, 2008 at 09:37:09PM -0700, mike wrote:

> related to that in logs either..
The timeout errors have a "while ..." string that describes conditions,
when the error has happened.

>     worker_connections  1024;
>     sendfile on;
>     #gzip_http_version 1.0;
>         location / {
> }
> 
>     set_real_ip_from 10.13.5.16;
>     gzip_static off;
>         include /etc/nginx/fastcgi.conf;
>         fastcgi_pass 127.0.0.1:11000;
>         fastcgi_index index.php;
>     }
>     if (!-e $request_filename) {
>         rewrite ^(.+)$ /wordpress/index.php?q=$1 last;
>     }

Instead of this "if" it's better to use:

      location / {
          error_page  404  = //wordpress/index.php?q=$uri;
      }

> }
> 
> 
> Any thoughts? Would turning off some buffers on the proxy make it
> better, or would turning off buffering on the webservers make it
> better? Not quite sure here where the best place would be to change
> (if anywhere...)

The timeout errors has no relation to buffers. These errors usually
means that backends are too slow.

On 4/30/08, Igor Sysoev <is@rambler-co.ru> wrote:

> The timeout errors has no relation to buffers. These errors usually
> means that backends are too slow.

What should I look at then to speed them up?

The backends are Linux, NFS server is FBSD7, proxy server is Linux.
Any pointers are appreciated.

The load on the machines seems to stay quite low...

 22:07:38 up  7:12,  2 users,  load average: 0.08, 0.08, 0.08

On Wed, Apr 30, 2008 at 10:07:59PM -0700, mike wrote:

> On 4/30/08, Igor Sysoev <is@rambler-co.ru> wrote:
> 
> > The timeout errors has no relation to buffers. These errors usually
> > means that backends are too slow.
> 
> What should I look at then to speed them up?
> 
> The backends are Linux, NFS server is FBSD7, proxy server is Linux.
> Any pointers are appreciated.

Could you show some timeout messages ?

On the proxy. I've changed the IPs and hostnames to protect the
innocent (except for the upstream IPs - which are the 10.13.5.x ones)

2008/04/30 22:02:44 [error] 24799#0: *77643721 upstream timed out
(110: Connection timed out) while reading upstream, client:
21.5.4.247, server: lvs01.myhosting.net, request: "GET
/gallery/videos/FLV_LG/ics2.flv HTTP/1.1", upstream:
"http://10.13.5.14:80/gallery/videos/FLV_LG/ics2.flv", host:
"media.clientdomain.net", referrer:
"http://clientdomain.net/images/tempVid_lg.swf"

2008/04/30 22:04:45 [error] 24801#0: *77661724 upstream timed out
(110: Connection timed out) while reading response header from
upstream, client: 12.107.23.215, server: lvs01.myhosting.net, request:
"GET /forum/aus-classifieds/171768-e46-sub-box-2-x-r-f-p2-subs.html
HTTP/1.1", upstream:
"http://10.13.5.14:80/forum/aus-classifieds/171768-e46-sub-box-2-x-r-f-p2-subs.html",
host: "www.clientdomain.net", referrer:
"http://images.google.com/imgres?imgurl=http://memimage.cardomain.net/member_images/12/web/2124000-2124999/2124693_10_full.jpg&imgrefurl=http://www.clientdomain.net/forum/aus-classifieds/171768-e46-sub-box-2-x-r-f-p2-subs.html&start=100&h=431&w=575&sz=170&tbnid=vF8z4lznXJyOxM:&tbnh=100&tbnw=134&hl=en&prev=/images%3Fq%3De46%26start%3D80%26imgsz%3Dlarge%257Cxlarge%257Cxxlarge%257Chuge%26gbv%3D1%26hl%3Den%26sa%3DN"

2008/04/30 22:08:18 [error] 24798#0: *77681622 upstream timed out
(110: Connection timed out) while reading response header from
upstream, client: 66.29.2.76, server: lvs01.myhosting.net, request:
"GET /forum/general-discussion/117997-how-hard-repaint-car.html
HTTP/1.1", upstream:
"http://10.13.5.10:80/forum/general-discussion/117997-how-hard-repaint-car.html",
host: "www.clientdomain.net"

i'm not actually seeing anything for that request in the backend error
log. sometimes i see "client aborted connection" etc. but i enabled
the "fastcgi_ignore_client_abort on;" on the backend, and
"proxy_ignore_client_abort on;" on the proxy...

On Wed, Apr 30, 2008 at 09:37:09PM -0700, mike wrote:

>     location ~ .php {

BTW, this regex should be:

      location ~ \.php$ {

On 4/30/08, Igor Sysoev <is@rambler-co.ru> wrote:
> On Wed, Apr 30, 2008 at 09:37:09PM -0700, mike wrote:
>
> >     location ~ .php {
>
> BTW, this regex should be:
>
>      location ~ \.php$ {

I originally had that. The problem is some of my apps do a rewrite from

foo.com/bar/baz -> foo.com/index.php/bar/baz

In which case, FastCGI won't be executed because php$ is not matched.
I had to remove the restriction on it ending with PHP for that. (Yes,
nowadays I know better, but I can't change those apps right now)

I could at least add in that \ before the period though, I suppose.

On Wed, Apr 30, 2008 at 10:26:16PM -0700, mike wrote:

> 
> (110: Connection timed out) while reading response header from
> upstream, client: 66.29.2.76, server: lvs01.myhosting.net, request:
> "GET /forum/general-discussion/117997-how-hard-repaint-car.html
> HTTP/1.1", upstream:
> "http://10.13.5.10:80/forum/general-discussion/117997-how-hard-repaint-car.html",
> host: "www.clientdomain.net"

The first line is error in the middle of ics2.flv transfer.
The last two are did not get responses at all: "while reading response 
header".
I think these two are handled by PHP and it may be slow.
But the first timeout of static FLV is strange. Are you show that it 
handled
by nginx but not PHP ?

yeah, i issued the request manually, there is no PHP there.

that could have been a client timeout or something for all i know. i
am sure with a few million requests per day there will be a handful of
crappy connections, proxies, tor exit nodes changing, etc...

On Wed, Apr 30, 2008 at 10:53:59PM -0700, mike wrote:

> yeah, i issued the request manually, there is no PHP there.

What do you mean by manually ?

> that could have been a client timeout or something for all i know. i
> am sure with a few million requests per day there will be a handful of
> crappy connections, proxies, tor exit nodes changing, etc...

No, this are timeout errors between nginx and its backend.

On 4/30/08, Igor Sysoev <is@rambler-co.ru> wrote:

> What do you mean by manually ?

I mean I loaded it manually in my browser to double check it loads,
and there is no handler on that vhost that does fastcgi no matter what
either.

> No, this are timeout errors between nginx and its backend.

Oh, hmm...

On Wed, Apr 30, 2008 at 11:27:21PM -0700, mike wrote:

> On 4/30/08, Igor Sysoev <is@rambler-co.ru> wrote:
> 
> > What do you mean by manually ?
> 
> I mean I loaded it manually in my browser to double check it loads,
> and there is no handler on that vhost that does fastcgi no matter what
> either.

Are the corresponding messages on other nginx ?

Not that time, I think ever since I did the ignore client aborts/etc I
don't get the corresponding errors.

On Wed, Apr 30, 2008 at 11:55:32PM -0700, mike wrote:

> Not that time, I think ever since I did the ignore client aborts/etc I
> don't get the corresponding errors.

You should look upstream errors on proxy nginx side and client errors
on other nginx side: the proxy nginx is client for nginxes of second 
level.

Just got a ton of these on the proxy:

2008/05/02 00:39:18 [error] 30191#0: *88449678 upstream timed out
(110: Connection timed out) while reading response header from
upstream

Not anything in the upstream error log - but it looks like NFS might
have frozen.

Shouldn't the upstream's error log report something about failure to
read a file or something?

Here's some weird ones:

2008/05/02 01:24:37 [info] 6656#0: *562998 client timed out (110:
Connection timed out) while reading client request line, client:
10.13.5.16, server: 0.0.0.0:80
2008/05/02 01:24:38 [info] 6658#0: *563002 client timed out (110:
Connection timed out) while reading client request line, client:
10.13.5.16, server: 0.0.0.0:80

10.13.5.16 is my proxy server (nginx)

and lots of these, in groups:

2008/05/02 01:26:08 [error] 30190#0: *88628174 recv() failed (104:
Connection reset by peer) while reading response header from upstream,
client: 52.94.24.43, server: lvs01.myhost.net, request: "GET
/i/rate/100/294/100294392_48e1c_h.jpg HTTP/1.1", upstream:
"http://10.13.5.10:80/i/rate/100/294/100294392_48e1c_h.jpg", host:
"media.client.com", referrer: "http://www.client.com/photos.php"

this is from the proxy server error log

below were from the upstream error log

You have a setup ?

client -> nginx1 -> nginx2

so nginx2 is waiting for nginx1 to send it something, but nginx1 is
expecting nginx2 to sending it something, nginx2 gets tired of waiting
and closes the connection.

What are your proxy settings? grep proxy nginx.conf

Cheers

Dave

Yes that's the setup.

I get timeouts from client <-> nginx1 (which is probably normal)
and nginx1 <-> nginx2
and nginx2 <-> nfs (which I am trying to address, I know some of the
timeouts are related but I dont believe all of them are.)

So I don't repost it again here are the configs:
http://article.gmane.org/gmane.comp.web.nginx.english/4738

On Fri, May 02, 2008 at 10:46:44PM +1000, Dave Cheney wrote:

> You have a setup ?
> 
> client -> nginx1 -> nginx2
> 
> so nginx2 is waiting for nginx1 to send it something, but nginx1 is  
> expecting nginx2 to sending it something, nginx2 gets tired of waiting  
> and closes the connection.

No. nginx1 gets all client request, then it connects to nginx2 and
passes full request to it. Then nginx1 wait nginx2 response.

On Fri, May 02, 2008 at 10:22:13AM -0700, mike wrote:

> Yes that's the setup.
> 
> I get timeouts from client <-> nginx1 (which is probably normal)
> and nginx1 <-> nginx2
> and nginx2 <-> nfs (which I am trying to address, I know some of the
> timeouts are related but I dont believe all of them are.)

nginx2 goes to NFS ? Why ? It shoul get files from local filesystem.
This scheme: client > nginx1 > nginx2 is replacement of

             client > nginx1 > NFS

nginx1 is going to be swapped out with LVS soon

so it will be client -> LVS -> nginx <-> NFS

all the data and files are on the shared NFS volume. i can't put
things on local filesystem. it's too much data. i am trying to get
scripts/templates to local filesystem and use something like mogilefs
for data but that's still far off.

On Fri, 2008-05-02 at 11:23 -0700, mike wrote:
> nginx1 is going to be swapped out with LVS soon
> 
> so it will be client -> LVS -> nginx <-> NFS
> 
> all the data and files are on the shared NFS volume. i can't put
> things on local filesystem. it's too much data. i am trying to get
> scripts/templates to local filesystem and use something like mogilefs
> for data but that's still far off.

You are misunderstanding what Igor has suggested.  Run Nginx *on the NFS
server* and use it to serve the static images *in place of NFS*.  Then
the files will be local.  Don't use NFS at all.

Regards,
Cliff

ohh.

well, i am not sure the NFS server is up to that much load... i've
never really looked at doing something like that.

besides, i still need NFS there for all writes and normal filesystem
access. it's not just an HTTP GET environment..

On Fri, 2008-05-02 at 12:46 -0700, mike wrote:
> ohh.
> 
> well, i am not sure the NFS server is up to that much load... i've
> never really looked at doing something like that.

Using Nginx will probably reduce the load.  I haven't used NFS in a long
while but I don't recall it being all that lightweight.

I suspect that if your NFS server seems heavily loaded it's exactly
because you are using NFS to serve public files.

> besides, i still need NFS there for all writes and normal filesystem
> access. it's not just an HTTP GET environment..

You can run both. The main point is to only serve files to the public
using Nginx and reduce NFS access to only internal use.

Regards,
Cliff

yeah i get it.

it is an interesting idea to think about.

Hi guys,

Can anyone explain the prejudice against NFS?

Specifically, why would additional proxy hop be faster than serving 
files from
NFS?
I can see two points in favor of NFS:
- NFS client caches files while Nginx doesn't (yet)
- Nginx doesn't support keepalive connections to upstream, hence 
additional
latencies and traffic for TCP handshake/finalization. NFS doesn't have 
this
issue since it typically works over UDP.

I do have a couple boxes serving a lot of traffic (mostly PHP) from NFS. 
It
works just fine, though it did take some NFS tuning.

On Friday 02 May 2008 16:05:21 Cliff Wells wrote:
> because you are using NFS to serve public files.
>
> > besides, i still need NFS there for all writes and normal filesystem
> > access. it's not just an HTTP GET environment..
>
> You can run both. The main point is to only serve files to the public
> using Nginx and reduce NFS access to only internal use.
>
> Regards,
> Cliff

--
Denis.

Would you mind sharing your tuning?

and what OS/specs the clients and server have?

you can reply off list if you want.

Clients and the server are RHEL4.
The mount options are as follows:
nolock,rsize=8192,wsize=8192,nocto,nfsvers=2

On Fri, 2008-05-02 at 16:44 -0400, Denis S. Filimonov wrote:
> Hi guys,
> 
> Can anyone explain the prejudice against NFS?

I hadn't noticed any.  It's more a question of using the wrong tool for
this job.  NFS is great if you need normal filesystem access for a
workgroup.  It's the wrong tool for serving web content.

> Specifically, why would additional proxy hop be faster than serving files from 
> NFS? 

Because I'm betting Nginx is faster than NFS.

> I can see two points in favor of NFS:
> - NFS client caches files while Nginx doesn't (yet)

But you *must* use Nginx if you want your files available on the web (or
more exactly, some HTTP server).  Caching is almost wholly irrelevant if
you go through a layer that doesn't cache.  The only overhead being
eliminated by NFS caching is that of NFS itself.

> - Nginx doesn't support keepalive connections to upstream, hence additional 
> latencies and traffic for TCP handshake/finalization. NFS doesn't have this 
> issue since it typically works over UDP.

Except that you ignore all the other overhead NFS imposes at a different
layer.  NFS may use a fairly efficient network protocol, however it's
also a fairly complete virtual filesystem, and this emulation imposes
significant overhead that a simple HTTP server doesn't have.

> I do have a couple boxes serving a lot of traffic (mostly PHP) from NFS. It 
> works just fine, though it did take some NFS tuning.

I'm sure apps would run just fine off NFS, since they are typically
read-once, run-many.  Serving a few static files from NFS would probably
be okay too, because the caching would help.  Serving lots of static
files over NFS is probably going to suck badly unless your NFS server
has tons of RAM for cache and even then I doubt it will perform very
well.

At the end of the day, I suggest the OP *try* both methods and let the
tests decide.  In the time it took to write this reply the OP could have
tested both cases and this discussion would have been moot.

Regards,
Cliff

> Can anyone explain the prejudice against NFS?

NFS *can* cause blocking problems.

I haven't done a detailed analysis of how nginx is serving files, but 
I've
seen NFS flakiness cause massive problems because lots of processes 
trying
to access the NFS share end up "locking up" waiting on the NFS server to
respond. Because the lock occurs inside the kernel (eg when the process 
is
doing a read() call, or accessing an mmaped() region of a file), the
processes are almost completely uninterruptable.

With nginx, this would be even worse. nginx uses a small process count +
non-blocking event loop model for serving files. If something is causes 
that
loop to "block" (eg. waiting on the NFS server), nginx will basically 
freeze
up and stop serving files completely. In theory, using epoll() and
sendfile() should push that blocking down into the kernel which 
shouldn't
affect nginx, but as I said, I haven't done a detailed analysis. Even 
doing
things like stat() which can't be made non-blocking on an NFS mounted 
file
can block badly causing an entire nginx process to freeze.

Some things then to check.

How does nginx handle file IO?

If you're not using sendfile(), does nginx use read() with O_NONBLOCK? 
Does
the linux kernel block a read() call on an NFS file if the NFS server is
having problems even if you're using O_NONBLOCK?

If you're using sendfile() on an NFS file, does the linux kernel block 
the
call if there's a problem accessing the NFS server?

Does nginx use stat() calls to verify a file/path exists? Does the linux
kernel block a stat() call on an NFS file if the NFS server is having
problems?

mike: Can we get an strace (run something like "strace -T -p nginxpid -o
/tmp/st.out") of one of your nginx processes for about 30 seconds (NOT 
the
parent process, I want to see what one of the child serving processes is
doing) while you're having problems. That should show if any system 
calls
are taking a long time and causing problems.

Rob

On Friday 02 May 2008 19:56:45 Cliff Wells wrote:
> On Fri, 2008-05-02 at 16:44 -0400, Denis S. Filimonov wrote:
> > Hi guys,
> >
> > Can anyone explain the prejudice against NFS?
>
> I hadn't noticed any.  It's more a question of using the wrong tool for
> this job.  NFS is great if you need normal filesystem access for a
> workgroup.  It's the wrong tool for serving web content.
>
That's exactly the prejudice I was talking about :-)
I've yet to see any proof of this statement while my experience suggests 
that
NFS suits serving web content very well (at least in some cases).

> you go through a layer that doesn't cache.  The only overhead being
> eliminated by NFS caching is that of NFS itself.
>
That's not true.
If I understand the proposed schemes correctly, they are
proxy <-(1)-> nginx/fcgi <-(2)-> NFS server
vs.
proxy <-(1)-> nginx <-(2)-> nginx/fcgi (over local fs)

Even though all traffic goes through link (1) anyway, we still want to 
reduce
the latency in the link (2). And I maintain that the scheme with NFS can 
do
better, partly because of caching.

> > - Nginx doesn't support keepalive connections to upstream, hence
> > additional latencies and traffic for TCP handshake/finalization. NFS
> > doesn't have this issue since it typically works over UDP.
>
> Except that you ignore all the other overhead NFS imposes at a different
> layer.  NFS may use a fairly efficient network protocol, however it's
> also a fairly complete virtual filesystem, and this emulation imposes
> significant overhead that a simple HTTP server doesn't have.
>
The only significant overhead NFS imposes is due to default consistency
requirements which are excessive for most web applications. Relaxing 
those
requirements goes a long way in improving NFS performance but for some 
reason
it's usually neglected. Seriously, having disabled close-to-open 
consistency
(totally irrelevant to a typical web application) I have observed 
*multifold*
reduction of NFS packets and huge drop in latency.

> > I do have a couple boxes serving a lot of traffic (mostly PHP) from NFS.
> > It works just fine, though it did take some NFS tuning.
>
> I'm sure apps would run just fine off NFS, since they are typically
> read-once, run-many.  Serving a few static files from NFS would probably
> be okay too, because the caching would help.  
I agree.

> Serving lots of static 
> files over NFS is probably going to suck badly unless your NFS server
> has tons of RAM for cache and even then I doubt it will perform very
> well.
True, it can perform poorly in extreme circumstances but I'm sure a web 
server
would do even worse under the same conditions.

> At the end of the day, I suggest the OP *try* both methods and let the
> tests decide.  In the time it took to write this reply the OP could have
> tested both cases and this discussion would have been moot.
>
Agreed. Hope he posts the results, I did my research on this subject 
years
ago. Much has changed, it'd be nice to have an update.

On 5/2/08, Rob Mueller <robm@fastmail.fm> wrote:

> mike: Can we get an strace (run something like "strace -T -p nginxpid -o
> /tmp/st.out") of one of your nginx processes for about 30 seconds (NOT the
> parent process, I want to see what one of the child serving processes is
> doing) while you're having problems. That should show if any system calls
> are taking a long time and causing problems.

It's going to be difficult to try to get this stuff going on my
production cluster. My largest client is already cutting my profit
margins down to a loss due to downtime related to problems, much less
testing.

I am -not- happy about NFS but at the moment I don't see any options
for removing it without changing all my clients to use some other
application-aware solution like MogileFS, etc. I've looked at AFS/Coda
and some other things. NFS seems to be the most widespread and widely
used solution...

I've tried ISCSI+OCFS2. I've tried NFS v4, v3, tcp, udp, jumbo frames,
32k rsize/wsize, 8k rsize/wsize, all that stuff. I already have plans
to at least get the largest client (using the largest % of resources
by far) to move to a MogileFS solution (for data) and just have local
webserver copies of scripts/templates (and use rsync + a "build
script")

I was going to at least try FreeBSD for the clients as well. Then at
least for now, I'd hopefully have a stable-r environment. I mean, I'm
not pushing that much data. People run NFS with thousands of clients
pushing lots of traffic. I think I'm pushing maybe 3MB/sec total
during peak time over 3 heavy and 2 light clients.

Would there be a way to setup nginx to dump some extended debugging
during issue time (and not normally or I'd have logfiles too large to
look through)

I'd like to add that 'having problems' doesn't just mean crashing. It
can be as benign as stalling for 1/4 of a second because of a high IO
backlog that needs to flushed to disk or a poorly configured NIC.
During that 1/4 of a second (an eon on modern cpus) nothing will
happen as the nginx worker process is blocked in kernel.

The suggestion from the wiki is to increase the number of workers if
nginx is heavily IO bound. The traditional pattern is 2 x spindles or
CPU's, whicheven is the larger. You setup is rather unconventional so
you'll have to experiment.

Cheers

Dave

On Sat, May 03, 2008 at 11:14:14AM +1000, Rob Mueller wrote:

> processes are almost completely uninterruptable.
> 
> With nginx, this would be even worse. nginx uses a small process count + 
> non-blocking event loop model for serving files. If something is causes 
> that loop to "block" (eg. waiting on the NFS server), nginx will basically 
> freeze up and stop serving files completely. In theory, using epoll() and 
> sendfile() should push that blocking down into the kernel which shouldn't 
> affect nginx, but as I said, I haven't done a detailed analysis. Even doing 
> things like stat() which can't be made non-blocking on an NFS mounted file 
> can block badly causing an entire nginx process to freeze.

You are right completely except epoll()/sendfile().

> Some things then to check.
> 
> How does nginx handle file IO?
> 
> If you're not using sendfile(), does nginx use read() with O_NONBLOCK? Does 
> the linux kernel block a read() call on an NFS file if the NFS server is 
> having problems even if you're using O_NONBLOCK?

Yes, kernel blocks.

> If you're using sendfile() on an NFS file, does the linux kernel block the 
> call if there's a problem accessing the NFS server?

Yes, kernel blocks.

> Does nginx use stat() calls to verify a file/path exists? Does the linux 
> kernel block a stat() call on an NFS file if the NFS server is having 
> problems?

Yes, kernel blocks.

On Fri, May 02, 2008 at 04:44:21PM -0400, Denis S. Filimonov wrote:

> I do have a couple boxes serving a lot of traffic (mostly PHP) from NFS. It 
> works just fine, though it did take some NFS tuning.

All filesystems read operations are blocking operations, i.e. if file 
page
is not in VM cache, a process must wait for it. The only exception are
aio_read(), but it has its own drawbacks. Local filesystem with 
non-faulty
disks has constant blocking time: about 10-20ms, seek time. NFS may 
block
longer.

And blocked nginx worker can not handle other its connections, those
can be handled fast from VM cache/etc. You do not see it in PHP case,
because each PHP process handles the single connection at the same time.

On Fri, May 02, 2008 at 12:46:13PM -0700, mike wrote:

> ohh.
> 
> well, i am not sure the NFS server is up to that much load... i've
> never really looked at doing something like that.
> 
> besides, i still need NFS there for all writes and normal filesystem
> access. it's not just an HTTP GET environment..

You may use NFS for anything. The only exception is nginx that should
run on NFS server and serves files from local filesystem.

Your suggestion is

nginx on NFS server for anything I can do (basically reads)

NFS for things I can't do (obviously without having to do DAV stuff
for writes, deletes, etc, etc)

Right?

On 5/2/08, Igor Sysoev <is@rambler-co.ru> wrote:

On Fri, May 02, 2008 at 11:35:44PM -0700, mike wrote:

> Your suggestion is
> 
> nginx on NFS server for anything I can do (basically reads)
> 
> NFS for things I can't do (obviously without having to do DAV stuff
> for writes, deletes, etc, etc)
> 
> Right?

Yes. You should use nginx on NFS server to handle HTTP GETs, and NFS for
all other things.

Almost forgot: the server has "async" option in exports.

On Saturday 03 May 2008 02:04:37 Igor Sysoev wrote:
> >
> > I do have a couple boxes serving a lot of traffic (mostly PHP) from NFS.
> > It works just fine, though it did take some NFS tuning.
>
> All filesystems read operations are blocking operations, i.e. if file page
> is not in VM cache, a process must wait for it. The only exception are
> aio_read(), but it has its own drawbacks. Local filesystem with non-faulty
> disks has constant blocking time: about 10-20ms, seek time. NFS may block
> longer.
>
That's only true under the assumption of empty IO queue.

At any rate, assuming the NFS server has the same disk latency, it adds 
the
network latency on top of that (CPU time is negligible compared to disk
seek). Roundtrip on a lightly loaded properly configured network takes 
under
1ms, i.e. an order of magnitude lower than a disk seek.

> And blocked nginx worker can not handle other its connections, those
> can be handled fast from VM cache/etc. You do not see it in PHP case,
> because each PHP process handles the single connection at the same time.

That's true, however one can increase the number of workers by the 
amount of
latency increase to archive the same level of concurrency, in this case 
it's
only 10%. That's really not a problem.

The problem with NFS happens when all necessary data blocks are cached: 
a
local FS would just happily return the cached data without accessing the 
disk
while NFS client still issues a request to see if the file has changed. 
Thus,
NFS tends to flood network with tiny requests and that's the cause of 
its
slowness. My point is that in most cases it can be easily prevented by
relaxing cache coherency protocol without sacrifying safety.

On Sat, May 03, 2008 at 03:14:05AM -0400, Denis S. Filimonov wrote:

> > > doesn't have this issue since it typically works over UDP.
> That's only true under the assumption of empty IO queue.
> 
> At any rate, assuming the NFS server has the same disk latency, it adds the 
> network latency on top of that (CPU time is negligible compared to disk 
> seek). Roundtrip on a lightly loaded properly configured network takes under 
> 1ms, i.e. an order of magnitude lower than a disk seek.

I agree, but any packet loss, retransmission, etc. will affect whole 
worker.

Also, as I understand, in modern Linux it's not easy to find the cause
of stall: in FreeBSD in top/ps you will see that a process waits on
"nfsrcv" WCHAN or so. Probably, modern NFS in FreeBSD became much better
(I saw many Yahoo!, Isilon, etc. developers NFS commits), but in one old
setup (FreeBSD 4 as client and FreeBSD 3 as server), I saw many Apache
stalls after some NFS things go wrong.

On Fri, 2 May 2008 20:17:20 -0700
mike <mike503@gmail.com> wrote:

> I am -not- happy about NFS but at the moment I don't see any options
> for removing it without changing all my clients to use some other
> application-aware solution like MogileFS, etc. I've looked at AFS/Coda
> and some other things.

Any reason you don't want to use AFS? The read caching should be much
more aggressive than NFS, and you have the option of a memory-only
cache.

I tried to set it up once and I got confused, and lazy. Basically, it
sucks to do any switches, because depending on the type of switch,
it's downtime for my clients. They're footing the bill for this setup
too...

As I've said, I'm going to try FreeBSD, it seems to have a better
memory/IO/VM subsystem, stronger NFS too. (I say better due to
people's complaints on other mailing lists with the Linux scheduler
right now and other things, how it swaps and allocates memory, etc)

Ideally the applications I host will be smart enough to use something
more distributed with less overhead (or at least capable of spreading
the overhead among a lot of nodes) - pNFS might be cool, some day...