Question: http video streaming and nginx ram usage

[luislavena]

Hi everyone,

I’m using Nginx to forward incoming http requests (from the Internet) to
an Aviosys 9100A+ (low-cost) IP video server.

My Nginx 0.7.64 runs on a small embedded linux device under OpenWRT -
the device has only 13 MB of
free ram when nginx is loaded. The user’s http requests get forwarded to
nginx on the OpenWRT box, and nginx
reverse proxies the requests on to the video server over LAN.

The job of nginx is to prevent people on the Internet from changing the
settings of the video server via the http interface - only viewing of
the streams should be allowed! Therefore, nginx acts as a reverse-proxy
and forwards only the allowed URL on to the server. So the viewers of
the stream cannot change the video server’s settings via the web
interface, because my nginx config does not forward the “banned” URLs.

It is OK for me for now that e.g. 3 simultaneous viewers consume 3 times
the bandwidth.

[ Just if you are wondering : It is not possible to set a password on
the video server, since I can only disable authentication altogether. It
is a low-cost device and configuration options are very limited. ]

Now here is where I get stuck:

This video server transfers the images to the viewing person’s web
browser in one long http response in real-time.

As I wrote, In my setup, I reverse-proxy only the allowed URLs (i.e. the
ones to “get the video”) on to the video server with nginx. This works
fine, but nginx seems to buffer the http data that it forwards locally.
So while viewing the video stream, nginx consumes more and more RAM
until the little linux device crashes.

However, if I stop viewing the stream after 1 or 2 minutes, before RAM
runs out, the memory gets freed again.
I believe it frees the ram because the request is finished.

On a Linux PC server, it all works perfectly. But in order to save
electricity, I would like to use the small linux device.

Has anyone got a hint for me what I could do to stop nginx from keeping
the long request in RAM (this is what I think what happens, at least)?

Any help would be greatly appreciated!

Here is my nginx.conf. 192.168.3.46 Is the IP of the video server.

user root;
worker_processes 1;

#access_log off;
error_log off;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;

events {
worker_connections 10;
}

http {
include mime.types;
default_type application/octet-stream;

#log_format  main  '$remote_addr - $remote_user [$time_local]

“$request” ’
# '$status $body_bytes_sent “$http_referer” ’
# ‘“$http_user_agent” “$http_x_forwarded_for”’;

#access_log  logs/access.log  main;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;

proxy_buffering off;
proxy_buffer_size 4k;
proxy_buffers 10 4k;
proxy_max_temp_file_size 0;

client_body_buffer_size 4k;
client_header_buffer_size 1k;
client_max_body_size 4k;
large_client_header_buffers 1 1k;



#gzip  on;

server {
    listen       80;
    server_name  localhost;

    #charset koi8-r;

    #access_log  logs/host.access.log  main;

location / {

root html;

index index.html index.htm;

}

location /GetStatus.cgi?JsVar=sStatus {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/GetStatus.cgi?JsVar=sStatus;
    }
location /Simple/home.htm?IMG {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/Simple/home.htm?IMG;
    }
location /Simple/home.htm?ATV {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/Simple/home.htm?IMG;
    }


location /Simple/core.jar {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46:80;
    }


location /Simple/home.htm {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/Simple/home.htm;
    }


location /GetData.cgi {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/GetData.cgi;
    }

location /GetStatus.cgi {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/GetStatus.cgi;
    }



location /Jpeg/ {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/Jpeg/;
    }

location /Simple/WinWebLite.cab {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/Simple/WinWebLite.cab;
    }
location /Simple/bg1.jpg {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/Simple/bg1.jpg;
    }


location /html/favicon.ico {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/html/favicon.ico;
    }



location /simple_load.swf {
        proxy_buffering           off;
        proxy_ignore_client_abort off;
        proxy_intercept_errors    on;
        proxy_next_upstream       error timeout invalid_header;
        proxy_redirect            off;
        proxy_set_header          X-Host $http_host;

proxy_set_header X-Forwarded-For $remote_addr;

        proxy_connect_timeout     60;
        proxy_send_timeout        60;
        proxy_read_timeout        60;
        proxy_pass http://192.168.3.46/simple_load.swf;
    }



    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }


}

}

Posted at Nginx Forum:

Question: http video streaming and nginx ram usage

[birdy]

On Tue, Mar 08, 2011 at 12:15:23AM -0500, birdy wrote:

Hi there,

The short answer is: I don’t know the answer to your question.

But read on anyway…

the stream cannot change the video server’s settings via the web
interface, because my nginx config does not forward the “banned” URLs.

Note that location{}s do not match ?query_strings. So some of your
location blocks are not used at all. If you enable the debug log, you’ll
see exactly which location is used for each request.

You may be allowing through more URLs than you
think. /Simple/home.htm?DO_EVIL, for example, will be proxied.

This video server transfers the images to the viewing person’s web
browser in one long http response in real-time.

As I wrote, In my setup, I reverse-proxy only the allowed URLs (i.e. the
ones to “get the video”) on to the video server with nginx. This works
fine, but nginx seems to buffer the http data that it forwards locally.

According to the notes at
Module ngx_http_proxy_module that shouldn’t
happen. So if it does, it’s a bug. (Or a documentation/version error.)

Can you get logs, from nginx or the system, showing the memory use? That
might allow you…

So while viewing the video stream, nginx consumes more and more RAM
until the little linux device crashes.

…to kill and restart nginx before the system falls over. (Or, if it
shows something other than nginx eating the ram, then you’ll know what
else to fix.)

However, if I stop viewing the stream after 1 or 2 minutes, before RAM
runs out, the memory gets freed again.
I believe it frees the ram because the request is finished.

Has anyone got a hint for me what I could do to stop nginx from keeping
the long request in RAM (this is what I think what happens, at least)?

Any help would be greatly appreciated!

As you already have,

proxy_buffering off;

should be enough. So there’s something else going on.

I’d suggest to try simplifying the nginx config: get rid of

location /GetStatus.cgi?JsVar=sStatus
location /Simple/home.htm?IMG
location /Simple/home.htm?ATV

since they are (almost certainly) never matched (unless you actually
make a request like “/Simple/home.htm%3fATV”).

Then take all of the repeated identical proxy_* directives and put them
in the server block, outside all location{}s. And then remove them from
the individual location{} blocks, leaving just proxy_pass there.

That should shrink the file size quite a bit, which might make it easier
for someone else to analyse the problem.

Good luck with it,

f

Francis D. [email protected]

[birdy]

Hello!

On Tue, Mar 08, 2011 at 12:15:23AM -0500, birdy wrote:

I’m using Nginx to forward incoming http requests (from the Internet) to
an Aviosys 9100A+ (low-cost) IP video server.

[…]

As I wrote, In my setup, I reverse-proxy only the allowed URLs (i.e. the
ones to “get the video”) on to the video server with nginx. This works
fine, but nginx seems to buffer the http data that it forwards locally.
So while viewing the video stream, nginx consumes more and more RAM
until the little linux device crashes.

However, if I stop viewing the stream after 1 or 2 minutes, before RAM
runs out, the memory gets freed again.
I believe it frees the ram because the request is finished.

nginx frees (most of) memory allocated for request only when
request is finished, this is how it’s expected to work.

Known issue with long-running requests is that chunked filter will
allocate some small amount of memory for each chunk send, and this
may grow big enough for long-running requests with “proxy_buffering”
set to “off”.

Workaround is

chunked_transfer_encoding off;

as available in nginx 0.8.35+.

Maxim D.

p.s. As already pointed out, your config isn’t correct. Location
directive can’t be used to test query string arguments. This is
irrelevant though.

[birdy]

Hi Francis, Hi Maxim,

thank you very much for your advice!

I will clean up my config…

The real work to do for me is to adapt and compile Nginx 0.8.35 or newer
for the OpenWRT distro I’m using.

This is because the latest nginx version available pre-built (as a
package) for the “atheros” platform of my router is 0.7.64, and I could
not find a distro that has a newer version.

Philipp

Posted at Nginx Forum:

Re: Question: http video streaming and nginx ram usage