Forum: NGINX Timeout serving large requests

2974d09ac2541e892966b762aad84943?d=identicon&s=25 BrindleFly (Guest)
on 2013-02-13 17:20
(Received via mailing list)
I'm struggling to find the source of a timeout. I am running Nginx
(1.2.6),
Passenger 3.0.18, and Rails 3.2.11. On serving one long running request,
I
find the results (CSV file) truncated after being returned from the
client.
The Rails application however continues to serve up the data until
complete,
at which time it reports:

    Couldn't forward the HTTP response back to the HTTP client: It seems
the
user clicked on the 'Stop' button in his browser.

Nginx access.log however reports a 200 for the request.

Some things I have tried:
1) I modified the Nginx read/send timeouts in the Passenger gem
(ext/nginx/Configuration.c) and recompiled, with no impact.
2) I ran a test of bypassing Nginx/Passenger by going direct to my app
running in Unicorn, and it serves up the result fine. I then took
Passenger
out of the equation by configuring Nginx to pass the requests to
Unicorn,
and the truncated result is back again.
3) I've played with the Nginx keepalive_timeout, proxy_read_timeout,
proxy_send_timeout and send_timeout - all with no impact.

Any thoughts/advice would be much appreciated.

Joe

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,236201,236201#msg-236201
Ca3fb71285bbde1570881acafa6b56a5?d=identicon&s=25 Anatoly Mikhailov (Guest)
on 2013-02-13 17:31
(Received via mailing list)
>
> Any thoughts/advice would be much appreciated.
>
> Joe
>
> Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,236201,236201#msg-236201
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

It happened for us many times, there is only one option - upstream.
It can be Unicorn/PassengerStandalone/whatever.

We spent a lot of time debugging Nginx, so there is no real solution
for passenger built-in module.


Anatoly
Ad8d77560977cd32ed5a78d1c5ff2f5c?d=identicon&s=25 Harold Sinclair (Guest)
on 2013-02-13 17:38
(Received via mailing list)
That was a bug in rails not the web server.
Ca3fb71285bbde1570881acafa6b56a5?d=identicon&s=25 Anatoly Mikhailov (Guest)
on 2013-02-13 17:40
(Received via mailing list)
>> Nginx access.log however reports a 200 for the request.
>>
>
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

You can find a lot of the same issues on Passenger bugtracker, the bug
is years old.
Passenger did the great job for us and it's good to continue using it as
standalone way.
But what if you want to have true Zero-downtime deployment?
Passenger provides it only with commercial paid version.

Anatoly
Ca3fb71285bbde1570881acafa6b56a5?d=identicon&s=25 Anatoly Mikhailov (Guest)
on 2013-02-13 17:42
(Received via mailing list)
There is no bug in Rails neither in Webserver!
It's a bug on passenger module implementation.

Anatoly
2974d09ac2541e892966b762aad84943?d=identicon&s=25 BrindleFly (Guest)
on 2013-02-13 17:49
(Received via mailing list)
The Unicorn test I ran disabled the passenger module and passed on
requests
via upstream in Nginx - but still returned a partial request. The only
test
that returns a full request is to bypass Nginx entirely and go direct to
Unicorn. So based on this, it seems specific to my Nginx configuration.

I find it hard to believe this is a generic defect in Nginx. It must be
configuration related, right?

Joe

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,236201,236206#msg-236206
2974d09ac2541e892966b762aad84943?d=identicon&s=25 BrindleFly (Guest)
on 2013-02-13 17:51
(Received via mailing list)
Anatoly,

See my other reply. I can take Passenger entirely out of the mix and
still
reproduce (using Nginx, Unicorn, configured via upstream). So this is
not
Passenger in this case (although usually I suspect it for most other
issues
;) ).

Joe

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,236201,236207#msg-236207
Ca3fb71285bbde1570881acafa6b56a5?d=identicon&s=25 Anatoly Mikhailov (Guest)
on 2013-02-13 17:59
(Received via mailing list)
Joe,

ok, Can I have the full configuration please?

Anatoly
2974d09ac2541e892966b762aad84943?d=identicon&s=25 BrindleFly (Guest)
on 2013-02-13 18:12
(Received via mailing list)
Here is nginx version (note: although it is compiled with passenger, I
have
not turned on the passenger directive in nginx.conf):

nginx version: nginx/1.2.6
built by gcc 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --with-http_ssl_module
--with-http_gzip_static_module --with-cc-opt=-Wno-error
--add-module=/usr/local/rvm/gems/ruby-1.9.3-p286/gems/passenger-3.0.18/ext/nginx

Here is nginx.conf:

user nobody;
worker_processes  1;
pid /etc/nginx/nginx.pid;

events {
  worker_connections  1024;
  accept_mutex off;
}

include conf.d/*.conf;

http {
  upstream app_server {
    server 127.0.0.1:8080 fail_timeout=0;
  }
  server {
    listen 80 default;
    server_name myapp.com;
    root /var/www/myapp/public;

    try_files $uri/index.html $uri.html $uri @app;

    location @app {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

      proxy_set_header Host $http_host;

      proxy_redirect off;

      proxy_pass http://app_server;
    }

  }

  # Try extreme timeouts to see if issue will reproduce

  client_header_timeout 600s;
  client_body_timeout 600s;
  keepalive_timeout 600s;
  proxy_read_timeout 600s;
  proxy_send_timeout 600s;
  lingering_timeout 600s;
  lingering_time 600s;
  send_timeout 600s;

  error_log  /var/log/nginx/error.log;
  access_log /var/log/nginx/access.log;

  include mime.types;
  default_type application/octet-stream;

  sendfile on;
  tcp_nopush off;
  gzip on;
  gzip_http_version 1.0;
  gzip_proxied any;
  gzip_min_length 500;
  gzip_disable "MSIE [1-6]\.";
  gzip_types text/plain text/html text/xml text/css
             text/comma-separated-values
             text/javascript application/x-javascript
             application/atom+xml;

  include sites.d/*.conf;

  include blockips.conf;
}

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,236201,236210#msg-236210
Ca3fb71285bbde1570881acafa6b56a5?d=identicon&s=25 Anatoly Mikhailov (Guest)
on 2013-02-13 18:17
(Received via mailing list)
Try to build it clean without passenger module, then update config with:

http {
    client_max_body_size      25m;
    client_body_buffer_size   128k;
    client_body_temp_path     /tmp/client_body_temp;

}

Anatoly
2974d09ac2541e892966b762aad84943?d=identicon&s=25 BrindleFly (Guest)
on 2013-02-13 18:41
(Received via mailing list)
I recompiled without passenger (--with-http_ssl_module
--with-http_gzip_static_module --with-cc-opt=-Wno-error) and to my
surprise,
the interrupted request issue has disappeared. This means that having
the
passenger module compiled into an application changes some aspect of
timeouts for all requests, independent of whether you have turned the
passenger directive on or not.

Wow... Now I understand why Phusion sells commercial versions of
passenger
that add support for timeout configuration parameters. Time to abandon
Passenger.

I sort of suspected I wasn't having an nginx issue. ;)

Joe

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,236201,236212#msg-236212
Ca3fb71285bbde1570881acafa6b56a5?d=identicon&s=25 Anatoly Mikhailov (Guest)
on 2013-02-13 20:48
(Received via mailing list)
Actually you can still use passenger standalone to have it independent
from the nginx binary (no need to compile it inside). In that case
passenger should not affect on request interruptions.

The reason to use something else can be zero-downtime practice,
if you would go with Unicorn, try to check the ready-to-use config:
https://gist.github.com/mikhailov/3052776

You know, it's the same approach as for any Unix
signals-based binary (master/workers) USR2+QUIT roll out.

Anatoly
2974d09ac2541e892966b762aad84943?d=identicon&s=25 FooBarWidget (Guest)
on 2013-09-19 11:54
(Received via mailing list)
Hi BrindleFly.

My name is Hongli Lai, and I am the CTO of Phusion and main developer
behind
Phusion Passenger. We noticed this topic moments ago.

I apologize for the inconvenience that this problem has caused you, but
I'd
like to point out that your problem has got absolutely nothing to do
with us
selling a commercial version. For one, we take pride in writing stable,
robust, well-performing software, whether we charge for it or not. The
open
source version of Phusion Passenger is a fully-featured, mature product
that
stands on its own, even without Enterprise enhancements. Deliberately
making
software fail with obscure errors, such as what *appears* to be the case
for
you, goes against every part of our philosophy and morals.

Second, the *only* technical changes in the Enterprise version are as
advertised on https://www.phusionpassenger.com/enterprise and as
documented
in the manual. There are absolutely no technical changes between the
open
source and the Enterprise version which would cause a difference in
behavior
like this. We only sell on features, not on core stability.

In other words: the problem you're experiencing is most likely just a
bug.
If that's the case then the bug would exist in both the open source as
well
as the Enterprise version. Your problem description is most peculiar:
you
mentioned that compiling out the Phusion Passenger module in Nginx makes
things work. However the Phusion Passenger module doesn't do *anything*
unless you explicitly set `passenger_enabled` on for that context, and
in no
way changes Nginx's core behavior. Having said that, I do not exclude
the
possibility that there are subtle interactions which could still
introduce
incompatibilities.

I would like to invite you, or anybody who experiences similar problems,
to
contact us through the Phusion Passenger community discussion forum:
https://groups.google.com/forum/#!forum/phusion-passenger and to provide
more information.
If this really is a bug, we'll do our best to fix it, Enterprise user or
not.

With kind regards,
Hongli Lai

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,236201,242983#msg-242983
2974d09ac2541e892966b762aad84943?d=identicon&s=25 FooBarWidget (Guest)
on 2013-09-19 11:59
(Received via mailing list)
Hi Anatoly.

In case you haven't seen my other reply: my name is Hongli Lai, and I am
the
CTO of Phusion and main developer behind Phusion Passenger. We noticed
this
topic moments ago. As I've mentioned in my other reply, we take pride in
writing stable, robust, well-performing software, whether we charge for
it
or not. The open source version of Phusion Passenger is a
fully-featured,
mature product that stands on its own, even without Enterprise
enhancements.
Deliberately making software fail with obscure errors, such as what
appears
to be the case for BrindleFly, goes against every part of our philosophy
and
morals. I can assure you that the problems described in this topic have
got
absolutely nothing to do with us selling a commercial version.

You mentioned "a lot of the same issues on Passenger bugtracker", and
"the
bug is years old". Can you point out which ones exactly? The described
issue
bears similarity to some old issues in Phusion Passenger 3 (and indeed,
BrindleFly is using version 3). Those issues have been fixed in Phusion
Passenger 4, released in May 2013. Even in the open source version.
Version
4 is a major improvement and closed many issues on our bug tracker. In
the
past half year alone we've managed to reduce the number of outstanding
issues in our issue tracker by 50%.

You see, in Phusion Passenger 3, the core I/O engine was multithreaded
and
there were certain I/O patterns (which involve file uploads) that it
didn't
handle correctly. In version 4, the core I/O engine has been entirely
rewritten in an evented manner, not unlike how Nginx itself works. All
I/O
patterns are now handled correctly.

I would like to invite you, or anybody who experiences similar problems,
to
contact us through the Phusion Passenger community discussion forum:
https://groups.google.com/forum/#!forum/phusion-passenger and to provide
more information. If there is a bug, we'll do our best to fix it,
Enterprise
user or not.

With kind regards,
Hongli Lai

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,236201,242984#msg-242984
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.