# Memory Management ( \> 25GB memory usage)

**URL:** <https://www.ruby-forum.com/t/memory-management-25gb-memory-usage/235157>\
**Category:** NGINX\
**Created:** [June 3, 2013, 2:57pm UTC](https://www.ruby-forum.com/t/memory-management-25gb-memory-usage/235157 "2013-06-03T14:57:47Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![alcina](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/a587f6/32.png) [@alcina](https://www.ruby-forum.com/u/alcina)\
**Post date:** [June 3, 2013, 2:57pm UTC](https://www.ruby-forum.com/t/memory-management-25gb-memory-usage/235157/1 "2013-06-03T14:57:47Z")

</div>

Hello nginx!

I have one worker-process, which uses over 25GB memory (and doesn’t stop  
to  
do that).  
My configuration is… let’s say special:

So there is nginx, which proxies all requests to the PHP backend and the  
PHP  
backend sends a large request back to nginx. I set the fastcgi\_buffers  
very  
enormous huge to avoid nginx creating temporary files on my disk - which  
would result in high CPU load.

Here is my configuration: (reduced to the problem)

worker\_processes 1;  
worker\_rlimit\_nofile 80000;  
worker\_priority -20;

events {  
worker\_connections 10240;  
multi\_accept on;  
}

# …

```
    # fastcgi settings
    fastcgi_buffers 20480 1k;
    fastcgi_connect_timeout 30;
    fastcgi_read_timeout 30;
    fastcgi_send_timeout 30;
    fastcgi_keep_conn on;
    upstream php-backend {
            server 127.0.0.1:9000;
            keepalive 10000;
    }

```

As you can see the buffers are extreme large, to avoid disk buffering.  
The  
problem is that nginx doesn’t free the buffers. It just eats and eats. I  
know it’s my fault and not nginx’ fault. What am I doing wrong?

The response of my php backend could be from 1k to 300mb.

What is the best setting for my situation?

Thanks

Posted at Nginx Forum:

> **[Memory Management ( \> 25GB memory usage)](https://forum.nginx.org/read.php?2,239795,239795#msg-239795)**
>
> Hello nginx! I have one worker-process, which uses over 25GB memory (and doesn't stop to do that). My configuration is... let's say special: So there is nginx, which proxies all requests to the PHP backend and the PHP backend sends a large request...

---

<div class="post-metadata">

**Author:** ![Belly](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/f4b2a3/32.png) [@Belly](https://www.ruby-forum.com/u/Belly)\
**Post date:** [June 3, 2013, 3:05pm UTC](https://www.ruby-forum.com/t/memory-management-25gb-memory-usage/235157/2 "2013-06-03T15:05:58Z")

</div>

Hello,

```
   Please assign worker-processors according to the number of cpus 

```

your  
server have and reduce or comment the values regarding fastcgi buffers  
and  
examine the changes after doing it.

---

<div class="post-metadata">

**Author:** ![Belly](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/f4b2a3/32.png) [@Belly](https://www.ruby-forum.com/u/Belly)\
**Post date:** [June 3, 2013, 3:14pm UTC](https://www.ruby-forum.com/t/memory-management-25gb-memory-usage/235157/3 "2013-06-03T15:14:09Z")

</div>

Thanks for your reply!

I don’t have performance issues. It’s just the memory usage.  
I don’t know how nginx handles its memory, but if I increase the number  
of  
worker processes wouldn’t this lead into much higher memory usage  
(worker\_process\*25GB) ?

The one worker is able to handle all requests, I don’t see the point of  
adding more workers.  
I want to know why nginx does not free the buffers after a request is  
finished.

Posted at Nginx Forum:

> **[Re: Memory Management ( \> 25GB memory usage)](https://forum.nginx.org/read.php?2,239795,239797#msg-239797)**
>
> Hello nginx! I have one worker-process, which uses over 25GB memory (and doesn't stop to do that). My configuration is... let's say special: So there is nginx, which proxies all requests to the PHP backend and the PHP backend sends a large request...

---

<div class="post-metadata">

**Author:** ![Belly](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/f4b2a3/32.png) [@Belly](https://www.ruby-forum.com/u/Belly)\
**Post date:** [June 3, 2013, 3:51pm UTC](https://www.ruby-forum.com/t/memory-management-25gb-memory-usage/235157/4 "2013-06-03T15:51:33Z")

</div>

Hello!

On Mon, Jun 03, 2013 at 08:57:21AM -0400, Belly wrote:

> # …
> 
> ```
> # fastcgi settings
> fastcgi_buffers 20480 1k;
> 
> ```

Just a side note: each buffer structure takes about 100 bytes of  
memory on 64-bit platforms, and using 1k buffers results in about  
10% overhead just because of this.

> As you can see the buffers are extreme large, to avoid disk buffering. The  
> problem is that nginx doesn’t free the buffers. It just eats and eats. I  
> know it’s my fault and not nginx’ fault. What am I doing wrong?
> 
> The response of my php backend could be from 1k to 300mb.

With your settings each connection can allocate up to 20M of  
buffers. That is, 1500k connections are enough to allocate 25G of  
memory. So the basic question is - how many connections are open?

With pessimistic assumption of 10k connections as per  
worker\_connections, you configuration will result in more than  
200G memory used.

> What is the best setting for my situation?

I would recommend using “fastcgi\_max\_temp\_file\_size 0;” if you  
want to disable disk buffering (see [1]), and configuring some  
reasonable number of reasonably sized fastcgi\_buffers. I would  
recommend starting tuning with something like 32 x 64k buffers.

[1] [Module ngx\_http\_fastcgi\_module](http://nginx.org/r/fastcgi_max_temp_file_size)

–  
Maxim D.

> **[nginx: donation](http://nginx.org/en/donation.html)**

---

<div class="post-metadata">

**Author:** ![Belly](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/f4b2a3/32.png) [@Belly](https://www.ruby-forum.com/u/Belly)\
**Post date:** [June 3, 2013, 5:13pm UTC](https://www.ruby-forum.com/t/memory-management-25gb-memory-usage/235157/5 "2013-06-03T17:13:58Z")

</div>

Hello!

On Mon, Jun 03, 2013 at 10:13:03AM -0400, Belly wrote:

> > > I have one worker-process, which uses over 25GB memory (and doesn’t  
> > > would result in high CPU load.  
> > > }
> 
> > > As you can see the buffers are extreme large, to avoid disk
> > 
> > want to disable disk buffering (see [1]), and configuring some  
> > will do a synchronous job inside an asynchronous one? Will it block the  
> > event loop?

Docs ask linked document the effect as follows:

: Value of zero disables buffering of responses to temporary files.

This is what it actually does - it stops nginx from using disk  
buffering. Instead, if fastcgi\_buffers configured isn’t enough,  
nginx will wait for some buffers to be sent to a client before  
reading more data from a backend. Note this means the backend  
will be busy sending the response for a longer time.

–  
Maxim D.

> **[nginx: donation](http://nginx.org/en/donation.html)**

---

<div class="post-metadata">

**Author:** ![Belly](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/f4b2a3/32.png) [@Belly](https://www.ruby-forum.com/u/Belly)\
**Post date:** [June 3, 2013, 4:13pm UTC](https://www.ruby-forum.com/t/memory-management-25gb-memory-usage/235157/6 "2013-06-03T16:13:46Z")

</div>

Thanks Maxim for you answer!

## Maxim D. Wrote:

> > worker\_processes 1;
> 
> Just a side note: each buffer structure takes about 100 bytes of  
> memory on 64-bit platforms, and using 1k buffers results in about  
> 10% overhead just because of this.

Very interesting! - Didn’t know that… Thanks!

> > As you can see the buffers are extreme large, to avoid disk

1000 - 2000… got your point.

> [1] [Module ngx\_http\_fastcgi\_module](http://nginx.org/r/fastcgi_max_temp_file_size)

I read about fastcgi\_max\_temp\_file\_size, but I’m a bit afraid of.  
fastcgi\_max\_temp\_file\_size 0; states that data will be transfered  
synchronously. What does it mean exactly? Is it faster/better than disk  
buffering? Nginx is built in an asynchronous way. What happens if a  
worker  
will do a synchronous job inside an asynchronous one? Will it block the  
event loop?

> –  
> Maxim D.  
> [nginx: donation](http://nginx.org/en/donation.html)
> 
> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,239795,239801#msg-239801](http://forum.nginx.org/read.php?2,239795,239801#msg-239801)

---

<div class="post-metadata">

**Author:** ![Belly](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/f4b2a3/32.png) [@Belly](https://www.ruby-forum.com/u/Belly)\
**Post date:** [June 4, 2013, 9:48am UTC](https://www.ruby-forum.com/t/memory-management-25gb-memory-usage/235157/7 "2013-06-04T09:48:56Z")

</div>

Thanks Maxim and Jonathan for making these things clear to me!  
Disabling disk buffering by using fastcgi\_max\_temp\_file\_size 0; and  
reducing  
the number of buffers solved the problem and made my service more  
efficient.

Best Regards,  
Belly Dancer

Posted at Nginx Forum:

> **[Re: Memory Management ( \> 25GB memory usage)](https://forum.nginx.org/read.php?2,239795,239822#msg-239822)**
>
> Hello nginx! I have one worker-process, which uses over 25GB memory (and doesn't stop to do that). My configuration is... let's say special: So there is nginx, which proxies all requests to the PHP backend and the PHP backend sends a large request...

---

<div class="post-metadata">

**Author:** ![Belly](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/f4b2a3/32.png) [@Belly](https://www.ruby-forum.com/u/Belly)\
**Post date:** [June 3, 2013, 5:18pm UTC](https://www.ruby-forum.com/t/memory-management-25gb-memory-usage/235157/8 "2013-06-03T17:18:39Z")

</div>

On Jun 3, 2013, at 10:13 AM, Belly wrote:

> I read about fastcgi\_max\_temp\_file\_size, but I’m a bit afraid of.  
> fastcgi\_max\_temp\_file\_size 0; states that data will be transfered  
> synchronously. What does it mean exactly? Is it faster/better than disk  
> buffering? Nginx is built in an asynchronous way. What happens if a worker  
> will do a synchronous job inside an asynchronous one? Will it block the  
> event loop?

It’s always been my understanding that in this context, “synchronously”  
means that nginx is proxying the data from php/fcgi to the client in  
real time.

This sounds like a typical problem of application load balancing.

The disk buffering / temp files allows for nginx to immediately “slurp”  
the entire response from the backend process, and then serves the files  
to the downstream client. This has the advantage of allowing you to  
immediately re-use the fcgi process for dynamic content slow or hangup  
connections downstream won’t tie up your pool of fcgi/apache processes.

restated with blocking - the temp files allow for blocking within nginx  
instead of php ( nginx can handle 10k connections, php is limited to the  
number of processes ). by removing the tempfiles, blocking will happen  
within php instead.

my advice would be to use URL partitioning to segment this type of  
behavior. I would only allow specific URLs to have no tmp files , and I  
would proxy them back to a different pool of fcgi (or apache) servers  
running with a tweaked config. this would allow the blocking activity  
from the routes serving large files to not affect the “global” pool of  
php processes.

i would also look into periodic reloads of nginx, to see if that frees  
things up. if so, that might be a simpler/more elegant solution.

I encountered problems like this about 10years ago with mod\_perl under  
apache. The aggressive code optimizations and memory/process management  
were tailored to making the application work very well but did not play  
nice with the rest of the box. The fix was to keep a low number of  
max\_requests , and move to a “vanilla + mod\_perl apache” system. Years  
later, nginx became the vanilla apache.

similar issues like this happen to people in the python and ruby  
communities as well more expensive or intensive routes are often  
sectioned off and dispatched to a different pool of servers , so their  
workload doesn’t affect the rest of requests.