High memory consumption when proxying to a Comet server

Server NGINX
1

Folks,

When proxying to a Comet server, I was able to notice that Nginx is
eating and not releasing a massive amount of memory.

As we all know, Comet servers can have this “streaming” mode, where we
receive a chunked HTTP request that never ends.

Is there some problem that avoids Nginx to release memory when in
proxy mode, before the connection ends at all?

I am using Nginx to balance between some instances of the Comet server:
upstream comet {
server comet:8081;
server comet:8082;
…
}

Memory consumption for Nginx procs for each minute during the tests:
271.3 MiB + 368.0 KiB = 271.6 MiB nginx (9)
273.3 MiB + 368.0 KiB = 273.6 MiB nginx (9)
274.8 MiB + 368.0 KiB = 275.2 MiB nginx (9)
276.6 MiB + 368.0 KiB = 277.0 MiB nginx (9)
278.8 MiB + 368.0 KiB = 279.2 MiB nginx (9)
281.0 MiB + 368.0 KiB = 281.4 MiB nginx (9)
283.2 MiB + 368.0 KiB = 283.6 MiB nginx (9)
285.4 MiB + 368.0 KiB = 285.8 MiB nginx (9)
287.2 MiB + 368.0 KiB = 287.6 MiB nginx (9)
288.9 MiB + 368.0 KiB = 289.3 MiB nginx (9)

I could also notice that the bigger is the message I send through the
Comet server (thus, through Nginx too) the bigger is the memory
allocation in the Nginx procs.
Also, if this helps, Nginx only writes down the access_log when the
client closes the connection.

What could I do to avoid this “leak” behaviour for long lasting proxy
connections in Nginx?

Regards,

Rogério Schneider

http://stockrt.github.com

2

Hello!

On Sun, Apr 11, 2010 at 09:16:33PM -0300, Rogério Schneider wrote:

When proxying to a Comet server, I was able to notice that Nginx is
eating and not releasing a massive amount of memory.

As we all know, Comet servers can have this “streaming” mode, where we
receive a chunked HTTP request that never ends.

Is there some problem that avoids Nginx to release memory when in
proxy mode, before the connection ends at all?

[…]

nginx does memory allocations on per-request basis, and frees
memory when request ends. Though after sending response headers
possible allocations are limited by various *_buffers settings.

For proxied requests you may want to make sure your proxy_buffers
aren’t set too high, see here:

http://wiki.nginx.org/NginxHttpProxyModule#proxy_buffers

Maxim D.

3

Maxim, thanks.

I have tried to reduce memory usage by reducing the default values for:
proxy_buffers
from “8 4k” to “2 4k”.

With this one would expect the memory usage to be 4 times lower, but
that was not what I saw.
I saw the memory usage stay in the same growth step.

May I mention that for the fact I am using Comet, I turned proxy
buffering off?
proxy_buffering off;
I think this is important to note and it can explain why tuning the
proxy buffers down did not lower the memory consumption.

Could I debug Nginx procs to see where the memory is being allocated
and for what purpose?
How could I debug it? Do you have some guideline?

Many thanks.

Regards,
Rogério Schneider

On Sun, Apr 11, 2010 at 9:49 PM, Maxim D. [email protected]
wrote:

Is there some problem that avoids Nginx to release memory when in

Module ngx_http_proxy_module

Maxim D.

nginx mailing list
[email protected]
nginx Info Page

–
Rogério Schneider

+55 (21) 9856 7847

MSN: [email protected]
GTalk: [email protected]
Skype: stockrt
http://stockrt.github.com

4

No, changing 8 buffers to 2 buffers doesn’t mean memory usage will
be 4 times less. 1) this isn’t the only allocations made, and 2)
not all requests allocate all proxy buffers, only ones with big
responses and slow clients. But see below.

Great, I didn’t know that.

Next candidates is gzip_buffers (only matters if you are using
gzip, up to 32 * 4k by default; and keep in mind that about 200k
or so will be allocated per request anyway with gzip enabled).

I am not using gzip. Nor the request come with Accept-Encoding,
neither I am turning “gzip on;”, so I think this is not the case too.

Another thing to notice is about the “per request” usage. I have
only one request per client, but this request lasts for many minutes,
still, when monitoring Nginx memory usage, it increases every minute,
even with no new requests made from the client side. In this period
of monitoring only the Comet server was writing data, up to the
Nginx proxy.

Remaining ones are output_buffers and fastcgi_buffers, these
shouldn’t matter in your case anyway.

I could not find where are the docs about output_buffers. I am
interested on reading about it.

Also, I am using this version of Nginx:

nginx version: nginx/0.7.65
built by gcc 4.3.0 20080428 (Red Hat 4.3.0-8) (GCC)
configure arguments: --prefix=/opt/tr/nginx --without-select_module
–without-poll_module --without-http_charset_module
–without-http_ssi_module --without-http_auth_basic_module
–without-http_autoindex_module --without-http_geo_module
–without-http_map_module --without-http_referer_module
–without-http_rewrite_module --without-http_fastcgi_module
–without-http_memcached_module --without-http_limit_zone_module
–without-http_limit_req_module --without-http_empty_gif_module
–without-http_browser_module --without-http_upstream_ip_hash_module
–without-mail_pop3_module --without-mail_imap_module
–without-mail_smtp_module --with-http_stub_status_module --with-debug

You may switch on debug log, it will be possible to trace big
allocations there. See here:

A debugging log

I posted this log of some tests, and if you can please take a
look at it, I would appreciate.

It was separated in sections, the start, the small msg, the big
msg, the close.
Then another run, the small msg after many big msgs, and the
close (with lots of “strange” free()s):

Regards,

Rogério Schneider
http://stockrt.github.com

5

Hello!

On Mon, Apr 12, 2010 at 12:43:23AM -0300, Rogério Schneider wrote:

[…]

Then another run, the small msg after many big msgs, and the
close (with lots of “strange” free()s):

nginx and comet (memory consumption behaviour) - Pastebin.com

Yep, I see what’s going on. It’s chunked filter which eats memory

  • it allocates small buffers for chunk-size and crlf markers.
    They aren’t big, but they are allocated for each buffer sent,
    and they aren’t reused.

Here is snipped from log where it causes another 4k allocation
from system (note “malloc” line):

2010/04/12 00:06:04 [debug] 32748#0: *55 recv: fd:16 1024 of 1024
2010/04/12 00:06:04 [debug] 32748#0: *55 http output filter
“/push/user1/xhrinteractive/canal2.b1?”
2010/04/12 00:06:04 [debug] 32748#0: *55 copy filter:
“/push/user1/xhrinteractive/canal2.b1?”
2010/04/12 00:06:04 [debug] 32748#0: *55 http chunk: 1024
2010/04/12 00:06:04 [debug] 32748#0: *55 malloc: 096E9878:4096
2010/04/12 00:06:04 [debug] 32748#0: *55 write new buf t:1 f:0 00000000,
pos 096E985C, size: 5 file: 0, size: 0
2010/04/12 00:06:04 [debug] 32748#0: *55 write new buf t:0 f:0 00000000,
pos 096E8140, size: 1024 file: 0, size: 0
2010/04/12 00:06:04 [debug] 32748#0: *55 write new buf t:0 f:0 00000000,
pos 080A6EE5, size: 2 file: 0, size: 0
2010/04/12 00:06:04 [debug] 32748#0: *55 http write filter: l:0 f:1
s:1031
2010/04/12 00:06:04 [debug] 32748#0: *55 http write filter limit 0
2010/04/12 00:06:04 [debug] 32748#0: *55 writev: 1031

As a workaround you may want to increase proxy_buffer_size to
reduce number of such allocation (and/or just drop connections
periodically). Correct fix would be to make these buffers
reusable once sent to client.

Maxim D.

6

As a workaround you may want to increase proxy_buffer_size to
reduce number of such allocation (and/or just drop connections
periodically). Correct fix would be to make these buffers
reusable once sent to client.

Maxim, thanks for the review of the logs.

Increasing proxy_buffer_size wouldn’t end up on malloc()ing more
memory at the end of a period?

I think on sending small chunks (say, 500 bytes) each minute,
during hours, to the clients.

For each small chunk, I think, we will be malloc()ing one buffer, or
am I wrong?

Regards,

Rogério Schneider
http://stockrt.github.com

7

Hello!

On Sun, Apr 11, 2010 at 10:08:26PM -0300, Rogério Schneider wrote:

Maxim, thanks.

I have tried to reduce memory usage by reducing the default values for:
proxy_buffers
from “8 4k” to “2 4k”.

With this one would expect the memory usage to be 4 times lower, but
that was not what I saw.
I saw the memory usage stay in the same growth step.

No, changing 8 buffers to 2 buffers doesn’t mean memory usage will
be 4 times less. 1) this isn’t the only allocations made, and 2)
not all requests allocate all proxy buffers, only ones with big
responses and slow clients. But see below.

May I mention that for the fact I am using Comet, I turned proxy buffering off?
proxy_buffering off;
I think this is important to note and it can explain why tuning the
proxy buffers down did not lower the memory consumption.

With proxy_buffering off; setting proxy_buffers indeed doesn’t
matter at all.

Next candidates is gzip_buffers (only matters if you are using
gzip, up to 32 * 4k by default; and keep in mind that about 200k
or so will be allocated per request anyway with gzip enabled).

Remaining ones are output_buffers and fastcgi_buffers, these
shouldn’t matter in your case anyway.

Could I debug Nginx procs to see where the memory is being allocated
and for what purpose?
How could I debug it? Do you have some guideline?

You may switch on debug log, it will be possible to trace big
allocations there. See here:

http://nginx.org/en/docs/debugging_log.html

Maxim D.

p.s. Please do not top-post. Thank you.

8

Hello!

On Mon, Apr 12, 2010 at 05:17:10PM -0300, Rogério Schneider wrote:

As a workaround you may want to increase proxy_buffer_size to
reduce number of such allocation (and/or just drop connections
periodically). Â Correct fix would be to make these buffers
reusable once sent to client.

Maxim, thanks for the review of the logs.

Increasing proxy_buffer_size wouldn’t end up on malloc()ing more
memory at the end of a period?

It’s size of single proxy buffer used for passing data with
proxy_buffering off. It will be allocated once during request
start. And by allocating bigger buffer you reduce number of
chunked encoding markers needed to pass big responses…

I think on sending small chunks (say, 500 bytes) each minute,
during hours, to the clients.

… but once your chunks are small enough to fit into your
proxy_buffer_size - there is no difference.

For each small chunk, I think, we will be malloc()ing one buffer, or
am I wrong?

Not exactly. It allocates 2 buffer structures (52 bytes each on
i386) and 18 bytes to hold chunk-size and initial CRLF (final CRLF
uses static string). This gives something about 122 bytes per
chunk, i.e. about 30 chunks per 4k malloc’ed from system (±
alignment and allocation overheads).

And here is the numbers in your logs:

$ egrep ‘malloc:|recv:’ zz2 | uniq -c
…
1 2010/04/12 00:06:04 [debug] 32748#0: *55 malloc: 096E9878:4096
31 2010/04/12 00:06:04 [debug] 32748#0: *55 recv: fd:16 1024 of 1024
1 2010/04/12 00:06:04 [debug] 32748#0: *55 malloc: 096F5160:4096
30 2010/04/12 00:06:04 [debug] 32748#0: *55 recv: fd:16 1024 of 1024
1 2010/04/12 00:06:04 [debug] 32748#0: *55 malloc: 096F6168:4096
…

Numbers vary slightly due to process not being stable enough, but
it doesn’t really matter.

Maxim D.

9

I have the very same problem trying to proxying a MJPEG stream. I really
would appreciate a solution not involving periodically resetting the
connection.

Many thanks,
Arrigo Zanette

Posted at Nginx Forum:

10

It’s size of single proxy buffer used for passing data with
proxy_buffering off. It will be allocated once during request
start. And by allocating bigger buffer you reduce number of
chunked encoding markers needed to pass big responses…
…
Not exactly. It allocates 2 buffer structures (52 bytes each on
i386) and 18 bytes to hold chunk-size and initial CRLF (final CRLF
uses static string). This gives something about 122 bytes per
chunk, i.e. about 30 chunks per 4k malloc’ed from system (±
alignment and allocation overheads).
…
…
1 2010/04/12 00:06:04 [debug] 32748#0: *55 malloc: 096E9878:4096
31 2010/04/12 00:06:04 [debug] 32748#0: *55 recv: fd:16 1024 of 1024
1 2010/04/12 00:06:04 [debug] 32748#0: *55 malloc: 096F5160:4096
30 2010/04/12 00:06:04 [debug] 32748#0: *55 recv: fd:16 1024 of 1024
1 2010/04/12 00:06:04 [debug] 32748#0: *55 malloc: 096F6168:4096
…

Is there some chance in free()ing this allocations after the chunk is
sent, but before the connection is finished? Since it is a long
lasting connection, I need to free some space at regular interval, so
I do not run out of memory.

Many thanks for all the explanations.

Sincerely,

Rogério Schneider
http://stockrt.github.com

11

On Fri, May 28, 2010 at 10:54 AM, zanettea [email protected] wrote:

I have the very same problem trying to proxying a MJPEG stream. I really would appreciate a solution not involving periodically resetting the connection.

I have seen your post referring mine, here
Re: memory consumption proxying a comet server and also have seen this
another post related to the matter here
FastCGI vs Proxy requests but no one
seems to bother in giving a direct response if that is or isn’t
possible to free memory for chunked responses BEFORE the connection
ends.

Regards,

Rogério Schneider
http://stockrt.github.com

12

The patch I attached actually frees memory before the connection ends
but do a lot of assumptions that might not work. The cleanest solution
is probably just to disable chunked transfer encoding with the option:

chunked_transfer_encoding off;

which removes the problem at the root (if it is ok for you to disable
chunked transfer encoding, of if you already implement it in the proxied
server).

AFAIK nginx does not support memory freeing/reuse before connection ends
and, with current architecture, is not a trivial job to do (requires to
handle reuse of non-contiguous allocations).

Regards,
Arrigo Zanette

Rogério Schneider Wrote:

FastCGI vs Proxy requests
http://stockrt.github.com

nginx mailing list
[email protected]
nginx Info Page

Posted at Nginx Forum:

13

No, small allocations can’t be freed in nginx. I outlined correct
fix as soon as the problem was identified, here:

High memory consumption when proxying to a Comet server

One should sit and code buffers reuse in chunked module. I have
this in my TODO list, but didn’t done this yet due to ENOTIME.

Ok Maxim, now I got it correctly.

Thanks for the feedback, again.

Regards,

Rogério Schneider
http://stockrt.github.com

14

Hello!

On Mon, Jun 07, 2010 at 11:15:20PM -0300, Rogério Schneider wrote:

On Fri, May 28, 2010 at 10:54 AM, zanettea [email protected] wrote:

I have the very same problem trying to proxying a MJPEG stream. I really would appreciate a solution not involving periodically resetting the connection.

I have seen your post referring mine, here
Re: memory consumption proxying a comet server and also have seen this
another post related to the matter here
FastCGI vs Proxy requests but no one
seems to bother in giving a direct response if that is or isn’t
possible to free memory for chunked responses BEFORE the connection
ends.

No, small allocations can’t be freed in nginx. I outlined correct
fix as soon as the problem was identified, here:

http://nginx.org/pipermail/nginx/2010-April/019845.html

One should sit and code buffers reuse in chunked module. I have
this in my TODO list, but didn’t done this yet due to ENOTIME.

Maxim D.

15

On Tue, Jun 08, 2010 at 11:55:01AM -0300, Rogério Schneider wrote:

The patch I attached actually frees memory before the connection ends but do a lot of assumptions that might not work. The cleanest solution is probably just to disable chunked transfer encoding with the option:

chunked_transfer_encoding off;

I am using 0.7.65 and it does not support this option.

Try 0.7.66.

–
Igor S.
http://sysoev.ru/en/

16

The patch I attached actually frees memory before the connection ends but do a lot of assumptions that might not work. The cleanest solution is probably just to disable chunked transfer encoding with the option:

chunked_transfer_encoding off;

I am using 0.7.65 and it does not support this option.

which removes the problem at the root (if it is ok for you to disable chunked transfer encoding, of if you already implement it in the proxied server).

AFAIK nginx does not support memory freeing/reuse before connection ends and, with current architecture, is not a trivial job to do (requires to handle reuse of non-contiguous allocations).

Thanks for these pointings, Arrigo.

Regards,

Rogério Schneider
http://stockrt.github.com

17

I am using 0.7.65 and it does not support this option.

Try 0.7.66.

I tried it and the behavior is the same.

Perhaps 0.8 series solved this already?

Regards,

Rogério Schneider
http://stockrt.github.com