# Optimize nginx for uploading/download huge files **URL:** \ **Category:** NGINX\ **Created:** [January 3, 2012, 11:36pm UTC](https://www.ruby-forum.com/t/optimize-nginx-for-uploading-download-huge-files/215420 "2012-01-03T23:36:19Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\ **Post date:** [January 3, 2012, 11:36pm UTC](https://www.ruby-forum.com/t/optimize-nginx-for-uploading-download-huge-files/215420/1 "2012-01-03T23:36:19Z")
Hello everyone, Firstly, I’m sorry for posting the same question here and there (How to forum). It is urgent issue to me. Could you give me a tip for which directives and values I have to set for the optimization? I installed nginx on virtual machine as reverse proxy server. It checks the authentication and bypasses the client request to the backend servers. The most transactions are to upload/download files, from 1M to 1G… The CPU usage hits 100% when the 1M file download request reaches around 25 TPS… around 20 TPS for upload request… I set client\_body\_buffer\_size, proxy\_buffer\_size, proxy\_buffers and so on… But not so good yet… ☹ client\_body\_buffer\_size is set to 2M and it helps to increase TPS for uploading, but not enough. proxy\_buffer\_size = 1M and proxy\_buffers = 4 1M help to increase TPS for downloading little bit, but not so much… Other directives are set by its default. The below is the result of profiling with google\_proftools\_module. As you can see, the most time is spent on readv() and writev(). Please give me any advice or your experience. Thank you for your time. Total: 22244 samples 11928 53.6% 53.6% 11930 53.6% readv 8212 36.9% 90.5% 8212 36.9% writev 907 4.1% 94.6% 907 4.1% recv 359 1.6% 96.2% 359 1.6% \_\_connect\_nocancel 248 1.1% 97.3% 248 1.1% \_\_close\_nocancel 35 0.2% 97.5% 46 0.2% \_IO\_str\_pbackfail 34 0.2% 97.7% 34 0.2% \_\_write\_nocancel 31 0.1% 97.8% 31 0.1% socket 28 0.1% 97.9% 28 0.1% memset and Total: 64475 samples 34117 52.9% 52.9% 34117 52.9% writev 14173 22.0% 74.9% 14181 22.0% readv 11981 18.6% 93.5% 11981 18.6% recv 1046 1.6% 95.1% 1049 1.6% \_\_connect\_nocancel 878 1.4% 96.5% 878 1.4% \_\_close\_nocancel 343 0.5% 97.0% 343 0.5% epoll\_wait 160 0.2% 97.2% 160 0.2% \_\_write\_nocancel 89 0.1% 97.4% 89 0.1% \_\_xstat64 81 0.1% 97.5% 115 0.2% \_IO\_str\_pbackfail Here is the information of my VM: ronjpark@PSEProxy:~/release/REL.0.13$ cat /proc/cpuinfo processor : 0 vendor\_id : GenuineIntel cpu family : 6 model : 2 model name : QEMU Virtual CPU version 0.14.1 stepping : 3 cpu MHz : 2393.998 cache size : 4096 KB fpu : yes fpu\_exception : yes cpuid level : 4 wp : yes flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm up rep\_good nopl pni cx16 popcnt hypervisor lahf\_lm bogomips : 4787.99 clflush size : 64 cache\_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ronjpark@PSEProxy:~/release/REL.0.13$ cat /proc/meminfo MemTotal: 4057052 kB MemFree: 2651832 kB Buffers: 80984 kB Cached: 1100692 kB SwapCached: 3856 kB Active: 325744 kB Inactive: 910176 kB Active(anon): 48884 kB Inactive(anon): 5684 kB Active(file): 276860 kB Inactive(file): 904492 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 2095100 kB SwapFree: 2082688 kB Dirty: 4 kB Writeback: 0 kB AnonPages: 51356 kB Mapped: 15740 kB Shmem: 324 kB Slab: 66252 kB SReclaimable: 56796 kB SUnreclaim: 9456 kB KernelStack: 1160 kB PageTables: 4124 kB NFS\_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 4123624 kB Committed\_AS: 176112 kB VmallocTotal: 34359738367 kB VmallocUsed: 28816 kB VmallocChunk: 34359707236 kB HardwareCorrupted: 0 kB HugePages\_Total: 0 HugePages\_Free: 0 HugePages\_Rsvd: 0 HugePages\_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 42996 kB DirectMap2M: 4151296 kB ronjpark@PSEProxy:~/release/REL.0.13$ uname -a Linux PSEProxy 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC 2011 x86\_64 x86\_64 x86\_64 GNU/Linux Posted at Nginx Forum: > **[Optimize nginx for uploading/download huge files..](https://forum.nginx.org/read.php?2,220778,220778#msg-220778)** > > Hello everyone, Firstly, I'm sorry for posting the same question here and there (How to forum). It is urgent issue to me. Could you give me a tip for which directives and values I have to set for the optimization? I installed nginx on virtual machi ---
**Author:** ![ronjpark](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/9de0a6/32.png) [@ronjpark](https://www.ruby-forum.com/u/ronjpark)\ **Post date:** [January 5, 2012, 12:32am UTC](https://www.ruby-forum.com/t/optimize-nginx-for-uploading-download-huge-files/215420/2 "2012-01-05T00:32:34Z")
On Tue, Jan 3, 2012 at 5:35 PM, ronjpark [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > It checks the authentication and bypasses the client request to the > > Thank you for your time. > 28 0.1% 97.9% 28 0.1% memset > 81 0.1% 97.5% 115 0.2% \_IO\_str\_pbackfail > cpu MHz : 2393.998 > cache\_alignment : 64 > Inactive: 910176 kB > AnonPages: 51356 kB > CommitLimit: 4123624 kB > DirectMap4k: 42996 kB > DirectMap2M: 4151296 kB > ronjpark@PSEProxy:~/release/REL.0.13$ uname -a > Linux PSEProxy 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC > 2011 x86\_64 x86\_64 x86\_64 GNU/Linux The bottleneck is neither CPU nor memory. VM disk performance will be bad. Any throughput you get on the virtual disk will degrade as each concurrent upload client triggers reads and writes. This is one case where you need a SAN, ZFS filesystem, or an SSD drive; configure with large block size if files are \>1GB. Even server class hard drives will thrash and slow down everything with enough clients contending for the disk. Remember, in relative time, if a cpu operation takes a second, an operation on the disk takes one month, under ideal conditions, and you are giving nginx less than ideal conditions. Stefan C. > **[ScaleEngine is the Technical Video Streaming CDN Delivering Perfect Streams...](https://www.scaleengine.com/contact)** > > Video CDN, streaming live and on demand securely and reliably. Perfect streams to the world with our developer API, exact usage billing, and direct support ---
**Author:** ![ronjpark](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/9de0a6/32.png) [@ronjpark](https://www.ruby-forum.com/u/ronjpark)\ **Post date:** [January 5, 2012, 1:38am UTC](https://www.ruby-forum.com/t/optimize-nginx-for-uploading-download-huge-files/215420/3 "2012-01-05T01:38:25Z")
## stefancaunter Wrote: > drive; configure with > Stefan C. > [Contact ScaleEngine - ScaleEngine](http://scaleengine.com/contact) Thank you Stefan. I also doubted what you said and tested with 1MB file. client\_body\_buffer\_size is set to 2M and proxy\_buffer\_size = 2M also. But it doesn’t work. As my understainding, nginx doesn’t make disk I/O for this file transaction because of enough big buffer size. Right? Please fix me if I’m wrong. Posted at Nginx Forum: [http://forum.nginx.org/read.php?2,220778,220818#msg-220818](http://forum.nginx.org/read.php?2,220778,220818#msg-220818) ---
**Author:** ![ronjpark](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/9de0a6/32.png) [@ronjpark](https://www.ruby-forum.com/u/ronjpark)\ **Post date:** [January 5, 2012, 2:05pm UTC](https://www.ruby-forum.com/t/optimize-nginx-for-uploading-download-huge-files/215420/4 "2012-01-05T14:05:25Z")
Hello! On Wed, Jan 04, 2012 at 07:38:08PM -0500, ronjpark wrote: > > writes. This is one case > > you are giving nginx less than ideal conditions. > > As my understainding, nginx doesn’t make disk I/O for this file > transaction because of enough big buffer size. Right? > Please fix me if I’m wrong. Yes. To limit disk I/O for big requests/responses there are following options available: 1. Using larger buffers, notably client\_body\_buffer\_size, proxy\_buffer\_size, proxy\_buffers. This is basically what you’ve already done. 2. For responses you may also completely disable disk buffering using “proxy\_max\_temp\_file\_size 0”. This implies that nginx won’t be able to read full response from a backend though, and connection to a backend will be busy for a time needed to client to download a response. Directive description may be found here: > **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)** > > ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com Maxim D.