Forum: NGINX bug in spdy - 499 response code on long running requests

2974d09ac2541e892966b762aad84943?d=identicon&s=25 justin (Guest)
on 2013-06-23 07:32
(Received via mailing list)
We sometimes have long running HTTPS requests, over 2 minutes, and
occasionally would see 499 response codes, and duplicate (double run)
transactions. After digging around and testing, it was discovered this
only
occurs with spdy support enabled. If we turn off spdy in the server
block,
the 499 response codes, and more  importunately duplicate transactions
stopped.

Is this a known limitation of spdy, or perhaps a bug? If this is a
limitation of spdy? Is it fixed in protocol version 3, since nginx only
runs
version 2. Would love to turn spdy support back on, but I can't have
duplicate requests being fired.

Thanks.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,240278,240278#msg-240278
40b4c848b8fcd63b0cb60b9d170c3a77?d=identicon&s=25 Valentin V. Bartenev (Guest)
on 2013-06-24 16:21
(Received via mailing list)
On Sunday 23 June 2013 09:31:25 justin wrote:
> duplicate requests being fired.
>

Could you provide a debug log?
http://nginx.org/en/docs/debugging_log.html

  wbr, Valentin V. Bartenev

--
http://nginx.org/en/donation.html
2974d09ac2541e892966b762aad84943?d=identicon&s=25 justin (Guest)
on 2013-06-25 05:26
(Received via mailing list)
Valentin,

Unfortunately I did not build nginx --with-debug, additionally it would
be
somewhat tedious and hard to replicate the issue.

However, if needed, I can recompile nginx on a development server and
see if
I can replicate this error condition with spdy enabled. However the
setup
would be tedious as it requires setting up SSL, etc. Can you think of an
easier way to test this though? Maybe simply a PHP file that does
something
like:

<?php
     sleep(300);
     file_put_contents("request.log", date());
?>

Then call that from nginx with spdy enabled and see if it writes twice
to
the file request.log.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,240278,240318#msg-240318
2974d09ac2541e892966b762aad84943?d=identicon&s=25 justin (Guest)
on 2013-10-12 07:20
(Received via mailing list)
Just upgraded to nginx 1.5.6 and still seeing this behavior where long
running requests are being called twice with SPDY enabled. As soon as I
disabled SPDY it goes away.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,240278,243634#msg-243634
2974d09ac2541e892966b762aad84943?d=identicon&s=25 bahaa (Guest)
on 2013-12-22 13:12
(Received via mailing list)
I noticed the same behavior when sending long running requests (>10secs)
from chrome with nginx running as a reversy proxy with spdy enabled.
What
happens is that chrome sends a PING frame, then sends the request, then
if
it doesn't get a PING reply it will close the connection and resend the
request.
You can see the handling of PING failure here:
http://git.chromium.org/gitweb/?p=chromium/src/net...

The reason chrome isn't getting a PING reply within 10secs is because
nginx
queues the ping reply and doesn't send it until it gets a response from
upstream.
I added a call to ngx_http_spdy_send_output_queue() in
ngx_http_spdy_state_ping() to send the PING reply immediately and it
solved
the problem.

Bahaa Aidi

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,240278,245759#msg-245759
2974d09ac2541e892966b762aad84943?d=identicon&s=25 justin (Guest)
on 2013-12-22 23:47
(Received via mailing list)
Thanks for the response Bahaa. Great information. Would you say the
behavior
that nginx queues the ping reply and doesn't send it until it gets a
response from upstream is a bug? We've had to disabled SPDY until the
double
request issue gets resolved. Hoping that SPDY protocol version 3 support
in
nginx will fix it.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,240278,245775#msg-245775
2974d09ac2541e892966b762aad84943?d=identicon&s=25 bahaa (Guest)
on 2013-12-23 04:52
(Received via mailing list)
I think it's a bug. Quote from spdy2 spec:
"Receivers of a PING frame should send an identical frame to the sender
as
soon as possible (if there is other pending data waiting to be sent,
PING
should take highest priority)."

Bahaa Aidi

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,240278,245777#msg-245777
40b4c848b8fcd63b0cb60b9d170c3a77?d=identicon&s=25 Valentin V. Bartenev (Guest)
on 2013-12-23 15:11
(Received via mailing list)
On Sunday 22 December 2013 22:51:46 bahaa wrote:
> I think it's a bug. Quote from spdy2 spec:
> "Receivers of a PING frame should send an identical frame to the sender as
> soon as possible (if there is other pending data waiting to be sent, PING
> should take highest priority)."
>

The actual bug is that some control frames (not only PING one) can stay
unsent
in output queue for an undefined time after processing of read-event.

I'm going to look at it soon.

  wbr, Valentin V. Bartenev
2974d09ac2541e892966b762aad84943?d=identicon&s=25 nanne (Guest)
on 2014-04-15 18:51
(Received via mailing list)
This is somewhat of an old one, but has there been any change on account
of
this possible bug? We have currently shut down SPDY to avoid this, but
at
the cost of... well.. shutting down SPDY ;)

I have made a serverfault-topic about this by the way but currently the
only
things in there are my detailed problem/question, and my self-answer
that
refers to this thread. Nevertheless, for completeness the link:
http://serverfault.com/q/523340/77501

Nanne.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,240278,249311#msg-249311
40b4c848b8fcd63b0cb60b9d170c3a77?d=identicon&s=25 Valentin V. Bartenev (Guest)
on 2014-04-15 19:13
(Received via mailing list)
On Tuesday 15 April 2014 12:50:35 nanne wrote:
> This is somewhat of an old one, but has there been any change on account of
> this possible bug? We have currently shut down SPDY to avoid this, but at
> the cost of... well.. shutting down SPDY ;)
>
> I have made a serverfault-topic about this by the way but currently the only
> things in there are my detailed problem/question, and my self-answer that
> refers to this thread. Nevertheless, for completeness the link:
> http://serverfault.com/q/523340/77501
>

As for the bug with deferred responding to PING, that could cause such
problems, then it was fixed around 1.5.9.

But since still nobody has provided a debug log, I cannot say for
certain
that it's the same bug that all the people in this topic have
experienced.

I'd recommend to you install the latest version and check if your
problem
solved or not. In the latter case, please, provide a debug log.

  wbr, Valentin V. Bartenev
2974d09ac2541e892966b762aad84943?d=identicon&s=25 nanne (Guest)
on 2014-04-16 12:23
(Received via mailing list)
Thanks for the update on this! I'll check it out, though I believe we
will
have to keep the current 1.4.7 stable for now.

Sadly I am not able to provide a debug log on the production machine. I
can
provide one from a test-environment of course, but it seems beyond me to
recreate the issues there :(.

I'll keep trying, and if we can recreate them I'll post a debuglog, then
update to the preview branch, and try again to see what has happened.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,240278,249328#msg-249328
40b4c848b8fcd63b0cb60b9d170c3a77?d=identicon&s=25 Valentin V. Bartenev (Guest)
on 2014-04-16 13:19
(Received via mailing list)
On Wednesday 16 April 2014 06:23:13 nanne wrote:
> Thanks for the update on this! I'll check it out, though I believe we will
> have to keep the current 1.4.7 stable for now.
>
> Sadly I am not able to provide a debug log on the production machine. I can
> provide one from a test-environment of course, but it seems beyond me to
> recreate the issues there :(.
>
> I'll keep trying, and if we can recreate them I'll post a debuglog, then
> update to the preview branch, and try again to see what has happened.
>

There is a contradiction between using an innovative experimental,
actively
developed protocol and old versions of software.

I don't recommend "stable" branch with spdy since it receives only
critical
bugs (like security related and some segfaults).  A lot of bugs in the
spdy
module already have been fixed in 1.5.x.

Also note, that number of clients with spdy/2 support rapidly decreases.

  wbr, Valentin V. Bartenev
2974d09ac2541e892966b762aad84943?d=identicon&s=25 nanne (Guest)
on 2014-04-16 14:02
(Received via mailing list)
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.