nginx1 is going to be swapped out with LVS soon
so it will be client -> LVS -> nginx <-> NFS
all the data and files are on the shared NFS volume. i can’t put
things on local filesystem. it’s too much data. i am trying to get
scripts/templates to local filesystem and use something like mogilefs
for data but that’s still far off.
On Fri, May 02, 2008 at 10:22:13AM -0700, mike wrote:
Yes that’s the setup.
I get timeouts from client <-> nginx1 (which is probably normal)
and nginx1 <-> nginx2
and nginx2 <-> nfs (which I am trying to address, I know some of the
timeouts are related but I dont believe all of them are.)
nginx2 goes to NFS ? Why ? It shoul get files from local filesystem.
This scheme: client > nginx1 > nginx2 is replacement of
client > nginx1 > NFSohh.
well, i am not sure the NFS server is up to that much load… i’ve
never really looked at doing something like that.
besides, i still need NFS there for all writes and normal filesystem
access. it’s not just an HTTP GET environment…
On Fri, 2008-05-02 at 11:23 -0700, mike wrote:
nginx1 is going to be swapped out with LVS soon
so it will be client -> LVS -> nginx <-> NFS
all the data and files are on the shared NFS volume. i can’t put
things on local filesystem. it’s too much data. i am trying to get
scripts/templates to local filesystem and use something like mogilefs
for data but that’s still far off.
You are misunderstanding what Igor has suggested. Run Nginx on the NFS
server and use it to serve the static images in place of NFS. Then
the files will be local. Don’t use NFS at all.
Regards,
Cliff
yeah i get it.
it is an interesting idea to think about.
On Fri, 2008-05-02 at 12:46 -0700, mike wrote:
ohh.
well, i am not sure the NFS server is up to that much load… i’ve
never really looked at doing something like that.
Using Nginx will probably reduce the load. I haven’t used NFS in a long
while but I don’t recall it being all that lightweight.
I suspect that if your NFS server seems heavily loaded it’s exactly
because you are using NFS to serve public files.
besides, i still need NFS there for all writes and normal filesystem
access. it’s not just an HTTP GET environment…
You can run both. The main point is to only serve files to the public
using Nginx and reduce NFS access to only internal use.
Regards,
Cliff
Hi guys,
Can anyone explain the prejudice against NFS?
Specifically, why would additional proxy hop be faster than serving
files from
NFS?
I can see two points in favor of NFS:
I do have a couple boxes serving a lot of traffic (mostly PHP) from NFS.
It
works just fine, though it did take some NFS tuning.
On Friday 02 May 2008 16:05:21 Cliff W. wrote:
because you are using NFS to serve public files.
besides, i still need NFS there for all writes and normal filesystem
access. it’s not just an HTTP GET environment…You can run both. The main point is to only serve files to the public
using Nginx and reduce NFS access to only internal use.Regards,
Cliff
–
Denis.
Clients and the server are RHEL4.
The mount options are as follows:
nolock,rsize=8192,wsize=8192,nocto,nfsvers=2
Would you mind sharing your tuning?
and what OS/specs the clients and server have?
you can reply off list if you want.
On Fri, 2008-05-02 at 16:44 -0400, Denis S. Filimonov wrote:
Hi guys,
Can anyone explain the prejudice against NFS?
I hadn’t noticed any. It’s more a question of using the wrong tool for
this job. NFS is great if you need normal filesystem access for a
workgroup. It’s the wrong tool for serving web content.
Specifically, why would additional proxy hop be faster than serving files from
NFS?
Because I’m betting Nginx is faster than NFS.
I can see two points in favor of NFS:
- NFS client caches files while Nginx doesn’t (yet)
But you must use Nginx if you want your files available on the web (or
more exactly, some HTTP server). Caching is almost wholly irrelevant if
you go through a layer that doesn’t cache. The only overhead being
eliminated by NFS caching is that of NFS itself.
- Nginx doesn’t support keepalive connections to upstream, hence additional
latencies and traffic for TCP handshake/finalization. NFS doesn’t have this
issue since it typically works over UDP.
Except that you ignore all the other overhead NFS imposes at a different
layer. NFS may use a fairly efficient network protocol, however it’s
also a fairly complete virtual filesystem, and this emulation imposes
significant overhead that a simple HTTP server doesn’t have.
I do have a couple boxes serving a lot of traffic (mostly PHP) from NFS. It
works just fine, though it did take some NFS tuning.
I’m sure apps would run just fine off NFS, since they are typically
read-once, run-many. Serving a few static files from NFS would probably
be okay too, because the caching would help. Serving lots of static
files over NFS is probably going to suck badly unless your NFS server
has tons of RAM for cache and even then I doubt it will perform very
well.
At the end of the day, I suggest the OP try both methods and let the
tests decide. In the time it took to write this reply the OP could have
tested both cases and this discussion would have been moot.
Regards,
Cliff
Can anyone explain the prejudice against NFS?
NFS can cause blocking problems.
I haven’t done a detailed analysis of how nginx is serving files, but
I’ve
seen NFS flakiness cause massive problems because lots of processes
trying
to access the NFS share end up “locking up” waiting on the NFS server to
respond. Because the lock occurs inside the kernel (eg when the process
is
doing a read() call, or accessing an mmaped() region of a file), the
processes are almost completely uninterruptable.
With nginx, this would be even worse. nginx uses a small process count +
non-blocking event loop model for serving files. If something is causes
that
loop to “block” (eg. waiting on the NFS server), nginx will basically
freeze
up and stop serving files completely. In theory, using epoll() and
sendfile() should push that blocking down into the kernel which
shouldn’t
affect nginx, but as I said, I haven’t done a detailed analysis. Even
doing
things like stat() which can’t be made non-blocking on an NFS mounted
file
can block badly causing an entire nginx process to freeze.
Some things then to check.
How does nginx handle file IO?
If you’re not using sendfile(), does nginx use read() with O_NONBLOCK?
Does
the linux kernel block a read() call on an NFS file if the NFS server is
having problems even if you’re using O_NONBLOCK?
If you’re using sendfile() on an NFS file, does the linux kernel block
the
call if there’s a problem accessing the NFS server?
Does nginx use stat() calls to verify a file/path exists? Does the linux
kernel block a stat() call on an NFS file if the NFS server is having
problems?
mike: Can we get an strace (run something like “strace -T -p nginxpid -o
/tmp/st.out”) of one of your nginx processes for about 30 seconds (NOT
the
parent process, I want to see what one of the child serving processes is
doing) while you’re having problems. That should show if any system
calls
are taking a long time and causing problems.
Rob
On 5/2/08, Rob M. [email protected] wrote:
mike: Can we get an strace (run something like “strace -T -p nginxpid -o
/tmp/st.out”) of one of your nginx processes for about 30 seconds (NOT the
parent process, I want to see what one of the child serving processes is
doing) while you’re having problems. That should show if any system calls
are taking a long time and causing problems.
It’s going to be difficult to try to get this stuff going on my
production cluster. My largest client is already cutting my profit
margins down to a loss due to downtime related to problems, much less
testing.
I am -not- happy about NFS but at the moment I don’t see any options
for removing it without changing all my clients to use some other
application-aware solution like MogileFS, etc. I’ve looked at AFS/Coda
and some other things. NFS seems to be the most widespread and widely
used solution…
I’ve tried ISCSI+OCFS2. I’ve tried NFS v4, v3, tcp, udp, jumbo frames,
32k rsize/wsize, 8k rsize/wsize, all that stuff. I already have plans
to at least get the largest client (using the largest % of resources
by far) to move to a MogileFS solution (for data) and just have local
webserver copies of scripts/templates (and use rsync + a “build
script”)
I was going to at least try FreeBSD for the clients as well. Then at
least for now, I’d hopefully have a stable-r environment. I mean, I’m
not pushing that much data. People run NFS with thousands of clients
pushing lots of traffic. I think I’m pushing maybe 3MB/sec total
during peak time over 3 heavy and 2 light clients.
Would there be a way to setup nginx to dump some extended debugging
during issue time (and not normally or I’d have logfiles too large to
look through)
On Friday 02 May 2008 19:56:45 Cliff W. wrote:
On Fri, 2008-05-02 at 16:44 -0400, Denis S. Filimonov wrote:
Hi guys,
Can anyone explain the prejudice against NFS?
I hadn’t noticed any. It’s more a question of using the wrong tool for
this job. NFS is great if you need normal filesystem access for a
workgroup. It’s the wrong tool for serving web content.
That’s exactly the prejudice I was talking about 
I’ve yet to see any proof of this statement while my experience suggests
that
NFS suits serving web content very well (at least in some cases).
you go through a layer that doesn’t cache. The only overhead being
eliminated by NFS caching is that of NFS itself.
That’s not true.
If I understand the proposed schemes correctly, they are
proxy <-(1)-> nginx/fcgi <-(2)-> NFS server
vs.
proxy <-(1)-> nginx <-(2)-> nginx/fcgi (over local fs)
Even though all traffic goes through link (1) anyway, we still want to
reduce
the latency in the link (2). And I maintain that the scheme with NFS can
do
better, partly because of caching.
- Nginx doesn’t support keepalive connections to upstream, hence
additional latencies and traffic for TCP handshake/finalization. NFS
doesn’t have this issue since it typically works over UDP.Except that you ignore all the other overhead NFS imposes at a different
layer. NFS may use a fairly efficient network protocol, however it’s
also a fairly complete virtual filesystem, and this emulation imposes
significant overhead that a simple HTTP server doesn’t have.
The only significant overhead NFS imposes is due to default consistency
requirements which are excessive for most web applications. Relaxing
those
requirements goes a long way in improving NFS performance but for some
reason
it’s usually neglected. Seriously, having disabled close-to-open
consistency
(totally irrelevant to a typical web application) I have observed
multifold
reduction of NFS packets and huge drop in latency.
I do have a couple boxes serving a lot of traffic (mostly PHP) from NFS.
It works just fine, though it did take some NFS tuning.I’m sure apps would run just fine off NFS, since they are typically
read-once, run-many. Serving a few static files from NFS would probably
be okay too, because the caching would help.
I agree.
Serving lots of static
files over NFS is probably going to suck badly unless your NFS server
has tons of RAM for cache and even then I doubt it will perform very
well.
True, it can perform poorly in extreme circumstances but I’m sure a web
server
would do even worse under the same conditions.
At the end of the day, I suggest the OP try both methods and let the
tests decide. In the time it took to write this reply the OP could have
tested both cases and this discussion would have been moot.
Agreed. Hope he posts the results, I did my research on this subject
years
ago. Much has changed, it’d be nice to have an update.
I’d like to add that ‘having problems’ doesn’t just mean crashing. It
can be as benign as stalling for 1/4 of a second because of a high IO
backlog that needs to flushed to disk or a poorly configured NIC.
During that 1/4 of a second (an eon on modern cpus) nothing will
happen as the nginx worker process is blocked in kernel.
The suggestion from the wiki is to increase the number of workers if
nginx is heavily IO bound. The traditional pattern is 2 x spindles or
CPU’s, whicheven is the larger. You setup is rather unconventional so
you’ll have to experiment.
Cheers
Dave
On Sat, May 03, 2008 at 11:14:14AM +1000, Rob M. wrote:
processes are almost completely uninterruptable.
With nginx, this would be even worse. nginx uses a small process count +
non-blocking event loop model for serving files. If something is causes
that loop to “block” (eg. waiting on the NFS server), nginx will basically
freeze up and stop serving files completely. In theory, using epoll() and
sendfile() should push that blocking down into the kernel which shouldn’t
affect nginx, but as I said, I haven’t done a detailed analysis. Even doing
things like stat() which can’t be made non-blocking on an NFS mounted file
can block badly causing an entire nginx process to freeze.
You are right completely except epoll()/sendfile().
Some things then to check.
How does nginx handle file IO?
If you’re not using sendfile(), does nginx use read() with O_NONBLOCK? Does
the linux kernel block a read() call on an NFS file if the NFS server is
having problems even if you’re using O_NONBLOCK?
Yes, kernel blocks.
If you’re using sendfile() on an NFS file, does the linux kernel block the
call if there’s a problem accessing the NFS server?
Yes, kernel blocks.
Does nginx use stat() calls to verify a file/path exists? Does the linux
kernel block a stat() call on an NFS file if the NFS server is having
problems?
Yes, kernel blocks.
On Fri, May 02, 2008 at 12:46:13PM -0700, mike wrote:
ohh.
well, i am not sure the NFS server is up to that much load… i’ve
never really looked at doing something like that.besides, i still need NFS there for all writes and normal filesystem
access. it’s not just an HTTP GET environment…
You may use NFS for anything. The only exception is nginx that should
run on NFS server and serves files from local filesystem.
Your suggestion is
nginx on NFS server for anything I can do (basically reads)
NFS for things I can’t do (obviously without having to do DAV stuff
for writes, deletes, etc, etc)
Right?
On 5/2/08, Igor S. [email protected] wrote:
On Fri, May 02, 2008 at 11:35:44PM -0700, mike wrote:
Your suggestion is
nginx on NFS server for anything I can do (basically reads)
NFS for things I can’t do (obviously without having to do DAV stuff
for writes, deletes, etc, etc)Right?
Yes. You should use nginx on NFS server to handle HTTP GETs, and NFS for
all other things.
On Fri, May 02, 2008 at 04:44:21PM -0400, Denis S. Filimonov wrote:
I do have a couple boxes serving a lot of traffic (mostly PHP) from NFS. It
works just fine, though it did take some NFS tuning.
All filesystems read operations are blocking operations, i.e. if file
page
is not in VM cache, a process must wait for it. The only exception are
aio_read(), but it has its own drawbacks. Local filesystem with
non-faulty
disks has constant blocking time: about 10-20ms, seek time. NFS may
block
longer.
And blocked nginx worker can not handle other its connections, those
can be handled fast from VM cache/etc. You do not see it in PHP case,
because each PHP process handles the single connection at the same time.
Almost forgot: the server has “async” option in exports.
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
Sponsor our Newsletter | Privacy Policy | Terms of Service | Remote Ruby Jobs