Timeout serving large requests

BrindleFly · February 13, 2013, 5:20pm

I’m struggling to find the source of a timeout. I am running Nginx
(1.2.6),
Passenger 3.0.18, and Rails 3.2.11. On serving one long running request,
I
find the results (CSV file) truncated after being returned from the
client.
The Rails application however continues to serve up the data until
complete,
at which time it reports:

Couldn't forward the HTTP response back to the HTTP client: It seems

the
user clicked on the ‘Stop’ button in his browser.

Nginx access.log however reports a 200 for the request.

Some things I have tried:

I modified the Nginx read/send timeouts in the Passenger gem
(ext/nginx/Configuration.c) and recompiled, with no impact.
I ran a test of bypassing Nginx/Passenger by going direct to my app
running in Unicorn, and it serves up the result fine. I then took
Passenger
out of the equation by configuring Nginx to pass the requests to
Unicorn,
and the truncated result is back again.
I’ve played with the Nginx keepalive_timeout, proxy_read_timeout,
proxy_send_timeout and send_timeout - all with no impact.

Any thoughts/advice would be much appreciated.

Joe

Posted at Nginx Forum:

BrindleFly · February 13, 2013, 5:31pm

Any thoughts/advice would be much appreciated.

Joe

Posted at Nginx Forum:
Timeout serving large requests

nginx mailing list
[email protected]
nginx Info Page

It happened for us many times, there is only one option - upstream.
It can be Unicorn/PassengerStandalone/whatever.

We spent a lot of time debugging Nginx, so there is no real solution
for passenger built-in module.

Anatoly

BrindleFly · February 13, 2013, 5:38pm

That was a bug in rails not the web server.

BrindleFly · February 13, 2013, 5:42pm

There is no bug in Rails neither in Webserver!
It’s a bug on passenger module implementation.

Anatoly

BrindleFly · February 13, 2013, 5:49pm

The Unicorn test I ran disabled the passenger module and passed on
requests
via upstream in Nginx - but still returned a partial request. The only
test
that returns a full request is to bypass Nginx entirely and go direct to
Unicorn. So based on this, it seems specific to my Nginx configuration.

I find it hard to believe this is a generic defect in Nginx. It must be
configuration related, right?

Joe

Posted at Nginx Forum:

BrindleFly · February 13, 2013, 5:40pm

Nginx access.log however reports a 200 for the request.

nginx mailing list
[email protected]
nginx Info Page

You can find a lot of the same issues on Passenger bugtracker, the bug
is years old.
Passenger did the great job for us and it’s good to continue using it as
standalone way.
But what if you want to have true Zero-downtime deployment?
Passenger provides it only with commercial paid version.

Anatoly

BrindleFly · February 13, 2013, 5:59pm

Joe,

ok, Can I have the full configuration please?

Anatoly

BrindleFly · February 13, 2013, 5:51pm

Anatoly,

See my other reply. I can take Passenger entirely out of the mix and
still
reproduce (using Nginx, Unicorn, configured via upstream). So this is
not
Passenger in this case (although usually I suspect it for most other
issues
).

Joe

Posted at Nginx Forum:

BrindleFly · February 13, 2013, 6:12pm

Here is nginx version (note: although it is compiled with passenger, I
have
not turned on the passenger directive in nginx.conf):

nginx version: nginx/1.2.6
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --with-http_ssl_module
–with-http_gzip_static_module --with-cc-opt=-Wno-error
–add-module=/usr/local/rvm/gems/ruby-1.9.3-p286/gems/passenger-3.0.18/ext/nginx

Here is nginx.conf:

user nobody;
worker_processes 1;
pid /etc/nginx/nginx.pid;

events {
worker_connections 1024;
accept_mutex off;
}

include conf.d/*.conf;

http {
upstream app_server {
server 127.0.0.1:8080 fail_timeout=0;
}
server {
listen 80 default;
server_name myapp.com;
root /var/www/myapp/public;

try_files $uri/index.html $uri.html $uri @app;

location @app {
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  proxy_set_header Host $http_host;

  proxy_redirect off;

  proxy_pass http://app_server;
}

}

Try extreme timeouts to see if issue will reproduce

client_header_timeout 600s;
client_body_timeout 600s;
keepalive_timeout 600s;
proxy_read_timeout 600s;
proxy_send_timeout 600s;
lingering_timeout 600s;
lingering_time 600s;
send_timeout 600s;

error_log /var/log/nginx/error.log;
access_log /var/log/nginx/access.log;

include mime.types;
default_type application/octet-stream;

sendfile on;
tcp_nopush off;
gzip on;
gzip_http_version 1.0;
gzip_proxied any;
gzip_min_length 500;
gzip_disable “MSIE [1-6].”;
gzip_types text/plain text/html text/xml text/css
text/comma-separated-values
text/javascript application/x-javascript
application/atom+xml;

include sites.d/*.conf;

include blockips.conf;
}

Posted at Nginx Forum:

BrindleFly · February 13, 2013, 6:17pm

Try to build it clean without passenger module, then update config with:

http {
client_max_body_size 25m;
client_body_buffer_size 128k;
client_body_temp_path /tmp/client_body_temp;

}

Anatoly

BrindleFly · February 13, 2013, 6:41pm

I recompiled without passenger (–with-http_ssl_module
–with-http_gzip_static_module --with-cc-opt=-Wno-error) and to my
surprise,
the interrupted request issue has disappeared. This means that having
the
passenger module compiled into an application changes some aspect of
timeouts for all requests, independent of whether you have turned the
passenger directive on or not.

Wow… Now I understand why Phusion sells commercial versions of
passenger
that add support for timeout configuration parameters. Time to abandon
Passenger.

I sort of suspected I wasn’t having an nginx issue.

Joe

Posted at Nginx Forum:

BrindleFly · February 13, 2013, 8:48pm

Actually you can still use passenger standalone to have it independent
from the nginx binary (no need to compile it inside). In that case
passenger should not affect on request interruptions.

The reason to use something else can be zero-downtime practice,
if you would go with Unicorn, try to check the ready-to-use config:

gist.github.com

https://gist.github.com/mikhailov/3052776

0. nginx_setup.sh

# Nginx+Unicorn best-practices congifuration guide. Heartbleed fixed.
# We use latest stable nginx with fresh **openssl**, **zlib** and **pcre** dependencies.
# Some extra handy modules to use: --with-http_stub_status_module --with-http_gzip_static_module
#
# Deployment structure
#
# SERVER:
#  /etc/init.d/nginx                             (1. nginx)
#  /home/app/public_html/app_production/current  (Capistrano directory)
#

This file has been truncated. show original

1. nginx

#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon

This file has been truncated. show original

2. nginx.conf

# Nginx main block configuration file.
# The most important directives here are ssl_protocols and ssl_ciphers
# Keep nginx configuration in repo, then just include it in /opt/nginx/conf/nginx.conf
# And DDoS prevent attack with directive limit_req_zone (limit 10 request/sec from 1 IP address)
# then it enables in the server block by "limit_req zone=one".
#
# config/server/production/nginx.conf


user                          app;

This file has been truncated. show original

There are more than three files. show original

You know, it’s the same approach as for any Unix
signals-based binary (master/workers) USR2+QUIT roll out.

Anatoly

BrindleFly · September 19, 2013, 11:59am

Hi Anatoly.

In case you haven’t seen my other reply: my name is Hongli L., and I am
the
CTO of Phusion and main developer behind Phusion Passenger. We noticed
this
topic moments ago. As I’ve mentioned in my other reply, we take pride in
writing stable, robust, well-performing software, whether we charge for
it
or not. The open source version of Phusion Passenger is a
fully-featured,
mature product that stands on its own, even without Enterprise
enhancements.
Deliberately making software fail with obscure errors, such as what
appears
to be the case for BrindleFly, goes against every part of our philosophy
and
morals. I can assure you that the problems described in this topic have
got
absolutely nothing to do with us selling a commercial version.

You mentioned “a lot of the same issues on Passenger bugtracker”, and
“the
bug is years old”. Can you point out which ones exactly? The described
issue
bears similarity to some old issues in Phusion Passenger 3 (and indeed,
BrindleFly is using version 3). Those issues have been fixed in Phusion
Passenger 4, released in May 2013. Even in the open source version.
Version
4 is a major improvement and closed many issues on our bug tracker. In
the
past half year alone we’ve managed to reduce the number of outstanding
issues in our issue tracker by 50%.

You see, in Phusion Passenger 3, the core I/O engine was multithreaded
and
there were certain I/O patterns (which involve file uploads) that it
didn’t
handle correctly. In version 4, the core I/O engine has been entirely
rewritten in an evented manner, not unlike how Nginx itself works. All
I/O
patterns are now handled correctly.

I would like to invite you, or anybody who experiences similar problems,
to
contact us through the Phusion Passenger community discussion forum:
Redirecting to Google Groups and to provide
more information. If there is a bug, we’ll do our best to fix it,
Enterprise
user or not.

With kind regards,
Hongli L.

Posted at Nginx Forum:

BrindleFly · September 19, 2013, 11:54am

Hi BrindleFly.

My name is Hongli L., and I am the CTO of Phusion and main developer
behind
Phusion Passenger. We noticed this topic moments ago.

I apologize for the inconvenience that this problem has caused you, but
I’d
like to point out that your problem has got absolutely nothing to do
with us
selling a commercial version. For one, we take pride in writing stable,
robust, well-performing software, whether we charge for it or not. The
open
source version of Phusion Passenger is a fully-featured, mature product
that
stands on its own, even without Enterprise enhancements. Deliberately
making
software fail with obscure errors, such as what appears to be the case
for
you, goes against every part of our philosophy and morals.

Second, the only technical changes in the Enterprise version are as
advertised on Passenger - Enterprise grade web app server for Ruby, Node.js, Python and as
documented
in the manual. There are absolutely no technical changes between the
open
source and the Enterprise version which would cause a difference in
behavior
like this. We only sell on features, not on core stability.

In other words: the problem you’re experiencing is most likely just a
bug.
If that’s the case then the bug would exist in both the open source as
well
as the Enterprise version. Your problem description is most peculiar:
you
mentioned that compiling out the Phusion Passenger module in Nginx makes
things work. However the Phusion Passenger module doesn’t do anything
unless you explicitly set passenger_enabled on for that context, and
in no
way changes Nginx’s core behavior. Having said that, I do not exclude
the
possibility that there are subtle interactions which could still
introduce
incompatibilities.

I would like to invite you, or anybody who experiences similar problems,
to
contact us through the Phusion Passenger community discussion forum:
Redirecting to Google Groups and to provide
more information.
If this really is a bug, we’ll do our best to fix it, Enterprise user or
not.

With kind regards,
Hongli L.

Posted at Nginx Forum: