# Nginx NAS serving large files **URL:** \ **Category:** NGINX\ **Created:** [October 22, 2011, 8:29pm UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073 "2011-10-22T20:29:05Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\ **Post date:** [October 22, 2011, 8:29pm UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073/1 "2011-10-22T20:29:05Z")
Hello Ladies and Gentlemen, We have a very frustrating problem with a NAS configuration that we recently set-up. After running a filesharing website on 8 servers for 1+ years we decided to switch to a NAS configuration. We set-up our NAS (24x1TB Raid 5) to serve files to 9 front end servers over a private switch - doing 9 servers on 1 disk(logical)… not sure if nginx can handle this. Since we’ve moved to the NAS BW usage has dropped significantly even though our traffic has stayed the same. Download speeds are pretty bad aswell but this is due to IO issues that are being created by NGINX. so here’s the problem we are facing… In theory nginx is creating an io bottleneck because it’s opening to many times without closing. We’re using nginx on our front end servers to grab files off our nas - each file will stay alive though for 45+ minutes. Instead of opening it for 10 seconds, reading into memory, and closing - it’s opening it for 10 seconds, reading into memory, streaming it for 45 minutes, THEN closing. We ran a test with nginx and we were only able to download at 30kbps on port 182, how-ever over apache we were able to reach speeds of 20mb/s. We’ve re-compiled nignx and installed the most recent stable version but that didn’t fix anything. Can someone please suggest a fix here? I’ve ran out of ideas… Thank you! ---
**Author:** ![3quid](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/3/f08c70/32.png) [@3quid](https://www.ruby-forum.com/u/3quid)\ **Post date:** [October 23, 2011, 2:59am UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073/2 "2011-10-23T02:59:48Z")
Do you have sendfile on or off? We generally run it off for NAS. Also, are you using the open file cache? ---
**Author:** ![3quid](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/3/f08c70/32.png) [@3quid](https://www.ruby-forum.com/u/3quid)\ **Post date:** [October 24, 2011, 6:43pm UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073/3 "2011-10-24T18:43:42Z")
Brian A. wrote in post #1027955: > Do you have sendfile on or off? We generally run it off for NAS. Also, > are > you using the open file cache? Hello, yes at the moment we have sendfile set to off. And yes we are using open file cache. And our readahead is 2mb incase you’re wondering. Any other ideas? ---
**Author:** ![3quid](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/3/f08c70/32.png) [@3quid](https://www.ruby-forum.com/u/3quid)\ **Post date:** [October 25, 2011, 10:24am UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073/4 "2011-10-25T10:24:34Z")
Graham, Forgive my ignorance, what’s “port 182” in your original email? In your tests did you use a single nginx instance and a single Apache against that 1 logical disk on Netapp? Can you show sections on nginx configuration relevant to disk operations (also, how many workers per nginx server)? Do you use any specific tunables to adjust the network settings for Netapp? How many files on Netapp partition? What’s your OS? ---
**Author:** ![3quid](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/3/f08c70/32.png) [@3quid](https://www.ruby-forum.com/u/3quid)\ **Post date:** [October 25, 2011, 6:19am UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073/5 "2011-10-25T06:19:16Z")
Any ideas guys - I need to get this resolved asap. Otherwise i need to figure another solution for our NSF NAS Nginx configuration… ---
**Author:** ![3quid](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/3/f08c70/32.png) [@3quid](https://www.ruby-forum.com/u/3quid)\ **Post date:** [October 25, 2011, 11:26am UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073/6 "2011-10-25T11:26:22Z")
On Sat, Oct 22, 2011 at 08:29:05PM +0200, Graham D. wrote: > created by NGINX. so here’s the problem we are facing… > We’ve re-compiled nignx and installed the most recent stable version but > that didn’t fix anything. > > Can someone please suggest a fix here? I’ve ran out of ideas… What OS do you use ? Try the following: worker\_processes 32; http { sendfile off; output\_buffers 1 1m; … } – Igor S. ---
**Author:** ![3quid](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/3/f08c70/32.png) [@3quid](https://www.ruby-forum.com/u/3quid)\ **Post date:** [October 25, 2011, 11:04am UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073/7 "2011-10-25T11:04:44Z")
Hi, On 25.10.2011 06:19, Graham D. wrote: > Any ideas guys - I need to get this resolved asap. > > Otherwise i need to figure another solution for our NSF NAS Nginx > configuration… Which HW do you have (Networkcard, …)? Do you have anything in the messages/syslog file about some limits? Do you use connection track in the iptables? Cheers Aleks ---
**Author:** ![3quid](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/3/f08c70/32.png) [@3quid](https://www.ruby-forum.com/u/3quid)\ **Post date:** [October 25, 2011, 6:06pm UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073/8 "2011-10-25T18:06:22Z")
Andrew A. wrote in post #1028314: > Graham, > > Forgive my ignorance, what’s “port 182” in your original email? > > In your tests did you use a single nginx instance and a single Apache > against that 1 logical disk on Netapp? > > Can you show sections on nginx configuration relevant to disk operations > (also, how many workers per nginx server)? > > Do you use any specific tunables to adjust the network settings for > Netapp? > > How many files on Netapp partition? > > What’s your OS? Hello Andrew, where not using a high end NAS and our OS is Centos 64 Bit Aleksandar L. wrote in post #1028318: > Which HW do you have (Networkcard, …)? > Do you have anything in the messages/syslog file about some limits? > Do you use connection track in the iptables? > > Cheers > Aleks Hello Aleksandar, Our NAS is working fine and there are no errors in the logs. The problem is with nginx it self. > What OS do you use ? > > Try the following: > > worker\_processes 32; > > http { > sendfile off; > output\_buffers 1 1m; > … > } > > – > Igor S. Hello Igor, we are using Centos 64 Bit. We haven’t tried using output\_buffers yet so we will try that shortly… Right now nginx is keeping the filehandle open after the file is cached in ram. So we end up with 3,000 filehandles open for the duration of a 45 minute download. This is a very frustrating problem. ---
**Author:** ![3quid](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/3/f08c70/32.png) [@3quid](https://www.ruby-forum.com/u/3quid)\ **Post date:** [October 26, 2011, 12:43am UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073/9 "2011-10-26T00:43:25Z")
On 25.10.2011 18:06, Graham D. wrote: [snipp] > the > logs. The problem is with nginx it self. I mean on your nginx server. ---
**Author:** ![3quid](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/3/f08c70/32.png) [@3quid](https://www.ruby-forum.com/u/3quid)\ **Post date:** [October 25, 2011, 9:10pm UTC](https://www.ruby-forum.com/t/nginx-nas-serving-large-files/213073/10 "2011-10-25T21:10:05Z")
We are using a NFS appliance with a very simple nginx config and having no issues. we have this set: sendfile off; tcp\_nopush off And that’s all we really did as far as tuning is concerned.