Forum: NGINX Timeout serving large requests

I'm struggling to find the source of a timeout. I am running Nginx 
(1.2.6),
Passenger 3.0.18, and Rails 3.2.11. On serving one long running request, 
I
find the results (CSV file) truncated after being returned from the 
client.
The Rails application however continues to serve up the data until 
complete,
at which time it reports:

    Couldn't forward the HTTP response back to the HTTP client: It seems 
the
user clicked on the 'Stop' button in his browser.

Nginx access.log however reports a 200 for the request.

Some things I have tried:
1) I modified the Nginx read/send timeouts in the Passenger gem
(ext/nginx/Configuration.c) and recompiled, with no impact.
2) I ran a test of bypassing Nginx/Passenger by going direct to my app
running in Unicorn, and it serves up the result fine. I then took 
Passenger
out of the equation by configuring Nginx to pass the requests to 
Unicorn,
and the truncated result is back again.
3) I've played with the Nginx keepalive_timeout, proxy_read_timeout,
proxy_send_timeout and send_timeout - all with no impact.

Any thoughts/advice would be much appreciated.

Joe

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,236201,236201#msg-236201

>
> Any thoughts/advice would be much appreciated.
>
> Joe
>
> Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,236201,236201#msg-236201
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

It happened for us many times, there is only one option - upstream.
It can be Unicorn/PassengerStandalone/whatever.

We spent a lot of time debugging Nginx, so there is no real solution
for passenger built-in module.


Anatoly

That was a bug in rails not the web server.

>> Nginx access.log however reports a 200 for the request.
>>
>
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

You can find a lot of the same issues on Passenger bugtracker, the bug 
is years old.
Passenger did the great job for us and it's good to continue using it as 
standalone way.
But what if you want to have true Zero-downtime deployment?
Passenger provides it only with commercial paid version.

Anatoly

There is no bug in Rails neither in Webserver!
It's a bug on passenger module implementation.

Anatoly

The Unicorn test I ran disabled the passenger module and passed on 
requests
via upstream in Nginx - but still returned a partial request. The only 
test
that returns a full request is to bypass Nginx entirely and go direct to
Unicorn. So based on this, it seems specific to my Nginx configuration.

I find it hard to believe this is a generic defect in Nginx. It must be
configuration related, right?

Joe

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,236201,236206#msg-236206

Anatoly,

See my other reply. I can take Passenger entirely out of the mix and 
still
reproduce (using Nginx, Unicorn, configured via upstream). So this is 
not
Passenger in this case (although usually I suspect it for most other 
issues
;) ).

Joe

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,236201,236207#msg-236207

Joe,

ok, Can I have the full configuration please?

Anatoly

Here is nginx version (note: although it is compiled with passenger, I 
have
not turned on the passenger directive in nginx.conf):

nginx version: nginx/1.2.6
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --with-http_ssl_module
--with-http_gzip_static_module --with-cc-opt=-Wno-error
--add-module=/usr/local/rvm/gems/ruby-1.9.3-p286/gems/passenger-3.0.18/ext/nginx

Here is nginx.conf:

user nobody;
worker_processes  1;
pid /etc/nginx/nginx.pid;

events {
  worker_connections  1024;
  accept_mutex off;
}

include conf.d/*.conf;

http {
  upstream app_server {
    server 127.0.0.1:8080 fail_timeout=0;
  }
  server {
    listen 80 default;
    server_name myapp.com;
    root /var/www/myapp/public;

    try_files $uri/index.html $uri.html $uri @app;

    location @app {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      proxy_set_header Host $http_host;

      proxy_redirect off;

      proxy_pass http://app_server;
    }

  }

  # Try extreme timeouts to see if issue will reproduce

  client_header_timeout 600s;
  client_body_timeout 600s;
  keepalive_timeout 600s;
  proxy_read_timeout 600s;
  proxy_send_timeout 600s;
  lingering_timeout 600s;
  lingering_time 600s;
  send_timeout 600s;

  error_log  /var/log/nginx/error.log;
  access_log /var/log/nginx/access.log;

  include mime.types;
  default_type application/octet-stream;

  sendfile on;
  tcp_nopush off;
  gzip on;
  gzip_http_version 1.0;
  gzip_proxied any;
  gzip_min_length 500;
  gzip_disable "MSIE [1-6]\.";
  gzip_types text/plain text/html text/xml text/css
             text/comma-separated-values
             text/javascript application/x-javascript
             application/atom+xml;

  include sites.d/*.conf;

  include blockips.conf;
}

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,236201,236210#msg-236210

Try to build it clean without passenger module, then update config with:

http {
    client_max_body_size      25m;
    client_body_buffer_size   128k;
    client_body_temp_path     /tmp/client_body_temp;

}

Anatoly

I recompiled without passenger (--with-http_ssl_module
--with-http_gzip_static_module --with-cc-opt=-Wno-error) and to my 
surprise,
the interrupted request issue has disappeared. This means that having 
the
passenger module compiled into an application changes some aspect of
timeouts for all requests, independent of whether you have turned the
passenger directive on or not.

Wow... Now I understand why Phusion sells commercial versions of 
passenger
that add support for timeout configuration parameters. Time to abandon
Passenger.

I sort of suspected I wasn't having an nginx issue. ;)

Joe

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,236201,236212#msg-236212

Actually you can still use passenger standalone to have it independent
from the nginx binary (no need to compile it inside). In that case
passenger should not affect on request interruptions.

The reason to use something else can be zero-downtime practice,
if you would go with Unicorn, try to check the ready-to-use config:
https://gist.github.com/mikhailov/3052776

You know, it's the same approach as for any Unix
signals-based binary (master/workers) USR2+QUIT roll out.

Anatoly