Bug in spdy - 499 response code on long running requests

Server NGINX
1

We sometimes have long running HTTPS requests, over 2 minutes, and
occasionally would see 499 response codes, and duplicate (double run)
transactions. After digging around and testing, it was discovered this
only
occurs with spdy support enabled. If we turn off spdy in the server
block,
the 499 response codes, and more importunately duplicate transactions
stopped.

Is this a known limitation of spdy, or perhaps a bug? If this is a
limitation of spdy? Is it fixed in protocol version 3, since nginx only
runs
version 2. Would love to turn spdy support back on, but I can’t have
duplicate requests being fired.

Thanks.

Posted at Nginx Forum:

2

On Sunday 23 June 2013 09:31:25 justin wrote:

duplicate requests being fired.

Could you provide a debug log?
http://nginx.org/en/docs/debugging_log.html

wbr, Valentin V. Bartenev

–
http://nginx.org/en/donation.html

3

Valentin,

Unfortunately I did not build nginx --with-debug, additionally it would
be
somewhat tedious and hard to replicate the issue.

However, if needed, I can recompile nginx on a development server and
see if
I can replicate this error condition with spdy enabled. However the
setup
would be tedious as it requires setting up SSL, etc. Can you think of an
easier way to test this though? Maybe simply a PHP file that does
something
like:

<?php sleep(300); file_put_contents("request.log", date()); ?>

Then call that from nginx with spdy enabled and see if it writes twice
to
the file request.log.

Posted at Nginx Forum:

4

I noticed the same behavior when sending long running requests (>10secs)
from chrome with nginx running as a reversy proxy with spdy enabled.
What
happens is that chrome sends a PING frame, then sends the request, then
if
it doesn’t get a PING reply it will close the connection and resend the
request.
You can see the handling of PING failure here:
http://git.chromium.org/gitweb/?p=chromium/src/net.git;a=blob;f=http/http_network_transaction.cc;h=a6d50699c0f1854036772b67c2a0ea1b1e62ae63;hb=HEAD#l1409

The reason chrome isn’t getting a PING reply within 10secs is because
nginx
queues the ping reply and doesn’t send it until it gets a response from
upstream.
I added a call to ngx_http_spdy_send_output_queue() in
ngx_http_spdy_state_ping() to send the PING reply immediately and it
solved
the problem.

Bahaa Aidi

Posted at Nginx Forum:

5

Just upgraded to nginx 1.5.6 and still seeing this behavior where long
running requests are being called twice with SPDY enabled. As soon as I
disabled SPDY it goes away.

Posted at Nginx Forum:

6

I think it’s a bug. Quote from spdy2 spec:
“Receivers of a PING frame should send an identical frame to the sender
as
soon as possible (if there is other pending data waiting to be sent,
PING
should take highest priority).”

Bahaa Aidi

Posted at Nginx Forum:

7

On Sunday 22 December 2013 22:51:46 bahaa wrote:

I think it’s a bug. Quote from spdy2 spec:
“Receivers of a PING frame should send an identical frame to the sender as
soon as possible (if there is other pending data waiting to be sent, PING
should take highest priority).”

The actual bug is that some control frames (not only PING one) can stay
unsent
in output queue for an undefined time after processing of read-event.

I’m going to look at it soon.

wbr, Valentin V. Bartenev

8

Thanks for the response Bahaa. Great information. Would you say the
behavior
that nginx queues the ping reply and doesn’t send it until it gets a
response from upstream is a bug? We’ve had to disabled SPDY until the
double
request issue gets resolved. Hoping that SPDY protocol version 3 support
in
nginx will fix it.

Posted at Nginx Forum:

9

This is somewhat of an old one, but has there been any change on account
of
this possible bug? We have currently shut down SPDY to avoid this, but
at
the cost of… well… shutting down SPDY :wink:

I have made a serverfault-topic about this by the way but currently the
only
things in there are my detailed problem/question, and my self-answer
that
refers to this thread. Nevertheless, for completeness the link:

Nanne.

Posted at Nginx Forum:

10

On Tuesday 15 April 2014 12:50:35 nanne wrote:

This is somewhat of an old one, but has there been any change on account of
this possible bug? We have currently shut down SPDY to avoid this, but at
the cost of… well… shutting down SPDY :wink:

I have made a serverfault-topic about this by the way but currently the only
things in there are my detailed problem/question, and my self-answer that
refers to this thread. Nevertheless, for completeness the link:
apache 2.2 - POST request is repeated with nginx loadbalanced server (status 499) - Server Fault

As for the bug with deferred responding to PING, that could cause such
problems, then it was fixed around 1.5.9.

But since still nobody has provided a debug log, I cannot say for
certain
that it’s the same bug that all the people in this topic have
experienced.

I’d recommend to you install the latest version and check if your
problem
solved or not. In the latter case, please, provide a debug log.

wbr, Valentin V. Bartenev

11

Thanks for the update on this! I’ll check it out, though I believe we
will
have to keep the current 1.4.7 stable for now.

Sadly I am not able to provide a debug log on the production machine. I
can
provide one from a test-environment of course, but it seems beyond me to
recreate the issues there :(.

I’ll keep trying, and if we can recreate them I’ll post a debuglog, then
update to the preview branch, and try again to see what has happened.

Posted at Nginx Forum:

12

On Wednesday 16 April 2014 06:23:13 nanne wrote:

Thanks for the update on this! I’ll check it out, though I believe we will
have to keep the current 1.4.7 stable for now.

Sadly I am not able to provide a debug log on the production machine. I can
provide one from a test-environment of course, but it seems beyond me to
recreate the issues there :(.

I’ll keep trying, and if we can recreate them I’ll post a debuglog, then
update to the preview branch, and try again to see what has happened.

There is a contradiction between using an innovative experimental,
actively
developed protocol and old versions of software.

I don’t recommend “stable” branch with spdy since it receives only
critical
bugs (like security related and some segfaults). A lot of bugs in the
spdy
module already have been fixed in 1.5.x.

Also note, that number of clients with spdy/2 support rapidly decreases.

wbr, Valentin V. Bartenev

13

agree

Posted at Nginx Forum: