# NFS 0-byte problem

**URL:** <https://www.ruby-forum.com/t/nfs-0-byte-problem/213290>\
**Category:** NGINX\
**Created:** [October 29, 2011, 12:43am UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290 "2011-10-29T00:43:04Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [October 29, 2011, 12:43am UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/1 "2011-10-29T00:43:04Z")

</div>

Hello all ,

We’re using nginx 1.1.6 ver. 5 server connected NetApp storage via NFS .  
(/var/www )

Sometimes 1 or 2 server return cache size 0-byte and don’t answer http  
request. Other 3 server working well .  
If i restarted nginx or delete cache file , 0-byte problem resolve.

Example service conf :

upstream [upstream.test.com](http://upstream.test.com) {  
server 1.1.1.1:80;  
server 2.2.2.2:80 backup;  
server 3.3.3.3:80 backup;  
}

server {  
listen 80;  
server\_name [video.test.com](http://video.test.com);  
access\_log off;

```
    location / {
            proxy_pass http://upstream.test.com;
            proxy_ignore_headers Expires Cache-Control Set-Cookie;
            proxy_cache_valid 200 30d;
            proxy_cache $host;
            proxy_connect_timeout 5;
            proxy_read_timeout 10;
            proxy_next_upstream error timeout http_500 http_502

```

http\_503 http\_504 http\_404;  
set $maincache “main-a:”;  
proxy\_cache\_key $maincache$host$uri;  
proxy\_redirect off;  
proxy\_set\_header Host $proxy\_host;  
proxy\_max\_temp\_file\_size 0;  
proxy\_hide\_header Cache-Control;  
proxy\_hide\_header X-Powered-By;  
proxy\_hide\_header X-AspNet-Version;  
expires 30d;

```
    }

    location ~ /delete(/.*) {
            allow 5.5.5.5;
            deny all;
            set $deluri $1;
            set $maincache "main-a:";
            proxy_cache_purge $host $maincache$host$deluri;
    }

```

}

Where i’m mising ?

Thanks

Posted at Nginx Forum:

> **[NFS 0-byte problem](https://forum.nginx.org/read.php?2,217496,217496#msg-217496)**
>
> Hello all , We're using nginx 1.1.6 ver. 5 server connected NetApp storage via NFS . (/var/www ) Sometimes 1 or 2 server return cache size 0-byte and don't answer http request. Other 3 server working well . If i restarted nginx or delete cache file ,...

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 29, 2011, 1:00am UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/2 "2011-10-29T01:00:57Z")

</div>

Is nginx accessing the NFS or just the upstream servers?

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 29, 2011, 1:07am UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/3 "2011-10-29T01:07:07Z")

</div>

nginx caching on NFS if MISS via upstream.

Posted at Nginx Forum:

> **[Re: NFS 0-byte problem](https://forum.nginx.org/read.php?2,217496,217501#msg-217501)**
>
> Hello all , We're using nginx 1.1.6 ver. 5 server connected NetApp storage via NFS . (/var/www ) Sometimes 1 or 2 server return cache size 0-byte and don't answer http request. Other 3 server working well . If i restarted nginx or delete cache file ,...

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 29, 2011, 2:28am UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/4 "2011-10-29T02:28:04Z")

</div>

On Oct 28, 2011, at 7:06 PM, kernelheaders wrote:

> nginx caching on NFS if MISS via upstream.

I wouldn’t recommend doing that. Cache should be on local “fast” disk.  
You need atomic renames, etc, to work reliably.

–Brian

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 29, 2011, 3:19am UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/5 "2011-10-29T03:19:13Z")

</div>

## Brian A. Wrote:

> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)

We’re using 12 x 600 GB SAS 15K Hard drives on NetApp storage, so  
storage I/O operations very fast. But my problem is not speed .

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,217496,217508#msg-217508](http://forum.nginx.org/read.php?2,217496,217508#msg-217508)

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 29, 2011, 10:25am UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/6 "2011-10-29T10:25:03Z")

</div>

Are they all using the same cache directoy/NFS mount point? Sharing a  
cache directory seems not to be supported.

Posted at Nginx Forum:

> **[Re: NFS 0-byte problem](https://forum.nginx.org/read.php?2,217496,217511#msg-217511)**
>
> Hello all , We're using nginx 1.1.6 ver. 5 server connected NetApp storage via NFS . (/var/www ) Sometimes 1 or 2 server return cache size 0-byte and don't answer http request. Other 3 server working well . If i restarted nginx or delete cache file ,...

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 29, 2011, 10:28am UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/7 "2011-10-29T10:28:45Z")

</div>

2011/10/29 kernelheaders [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)

> > 
> 
> storage I/O operations very fast. But my problem is not speed .

- what is your netapp appliance ? (just curious)

- what are your mount options ?

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 29, 2011, 2:26pm UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/8 "2011-10-29T14:26:29Z")

</div>

On Oct 28, 2011, at 9:18 PM, kernelheaders wrote:

> We’re using 12 x 600 GB SAS 15K Hard drives on NetApp storage, so  
> storage I/O operations very fast. But my problem is not speed .

The cache is expected to be used only by a single nginx instance.  
Sharing it will not work. The cache is made up of the cache directory,  
temp files, and shared memory.

If you really need shared cache as well, then look into using memc with  
srcache modules.

–Brian

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 29, 2011, 7:06pm UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/9 "2011-10-29T19:06:36Z")

</div>

## JackB Wrote:

> Are they all using the same cache directoy/NFS  
> mount point? Sharing a cache directory seems not  
> to be supported.

Yeap , same cache and temp directory .

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,217496,217515#msg-217515](http://forum.nginx.org/read.php?2,217496,217515#msg-217515)

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 29, 2011, 7:07pm UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/10 "2011-10-29T19:07:14Z")

</div>

## Jérôme Loyet Wrote:

> > >
> 
> * * *
> 
> - what is your netapp appliance ? (just curious)
> 
> - what are your mount options ?
> 
> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)

- Storage FA2020
- I try UDP , TCP a lot of version . But problem is not about mount.

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,217496,217516#msg-217516](http://forum.nginx.org/read.php?2,217496,217516#msg-217516)

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 30, 2011, 3:00am UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/11 "2011-10-30T03:00:27Z")

</div>

## Brian A. Wrote:

> The cache is expected to be used only by a single  
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)

Can you please explain why only one distance ? If you have same cachekey  
and structure , cache work all intance even remote locations .  
And system works well , sometimes and some servers occurs problem .

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,217496,217533#msg-217533](http://forum.nginx.org/read.php?2,217496,217533#msg-217533)

---

<div class="post-metadata">

**Author:** ![kernelheaders](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/df788c/32.png) [@kernelheaders](https://www.ruby-forum.com/u/kernelheaders)\
**Post date:** [October 31, 2011, 1:48am UTC](https://www.ruby-forum.com/t/nfs-0-byte-problem/213290/12 "2011-10-31T01:48:32Z")

</div>

Hello!

On Sat, Oct 29, 2011 at 09:59:43PM -0400, kernelheaders wrote:

> > 
> 
> and structure , cache work all intance even remote locations .  
> And system works well , sometimes and some servers occurs problem .

As Brian already wrote, nginx currently doesn’t allow sharing of  
the same cache directory between several nginx instances. It  
doesn’t expect cache files to appear or be changed by another  
nginx instance. If you share cache directory between multiple  
nginx instances - weird things will happen, and problem you see is  
one of them.

Maxim D.