SPDY + proxy cache = occasional 499 errors

Hi there,

I have been piloting the SPDY patch on a separate web server pointing to
an
existing HTTP backend.

With just myself using this pilot web server, occasionally static
resources
such as css or images are not loading in Firefox (usually when pressing
F5
to reload a page)

Have done a bit of digging and have learnt this so far:

When issue occurs in Firefox, access log is showing 499 errors for the
missing static resources (when it should be 304)
Refreshing the page will eventually load the resources (and they will
show
304 in the access log)
If I turn off the proxy cache for these static resources, the issue
stops
If I turn off SPDY, the issue stops.
The issue obviously doesn’t occur on the production web server running
Nginx
1.2.x

Has this been seen before, or can you think of any reason why this might
happen?

I don’t normally use Chrome but have also witnessed this once in Chrome,
so
I don’t think it’s browser specific (however will try to get more
confidence
of this).

Thanks.

Posted at Nginx Forum:

On Sunday 02 December 2012 03:48:52 spdyg wrote:

happen?

[…]

Thank you for testing. There is a reason for this in current spdy
implementation. I’m working on it.

wbr, Valentin V. Bartenev

http://nginx.org/en/donation.html

I posted this query back in December:

And the issue was fixed shortly afterwards in the 1.3.x branch.

I’m now seeing the same issue again (static items coming from proxy
cache +
SPDY not displaying).

However this time, the log does not show 499 error anymore… it just
shows
successful 304. However, looking at firefox web console, the first few
static items come back as 304 without problems, then the next set of
static
items come back with no response at all… (no response header, no body).

The setup looks like this:

Firefox → Nginx 1.3.15 → HAproxy → Tomcat

Has this bug crept back in, or is this a new one? Also reproduced on
1.3.13.

Can’t seem to reproduce the issue if I take out HAproxy in the middle
though.

spdyg

Posted at Nginx Forum:

On Tuesday 02 April 2013 14:54:30 spdyg wrote:

successful 304. However, looking at firefox web console, the first few
Can’t seem to reproduce the issue if I take out HAproxy in the middle
though.

The behavior of “return 499” can result in weird effects if it affects
a bunch of requests at once.

wbr, Valentin V. Bartenev


http://nginx.org/en/donation.html

Hello!

On Tue, Apr 02, 2013 at 05:16:46PM +0400, Valentin V. Bartenev wrote:

However this time, the log does not show 499 error anymore… it just shows

Can’t seem to reproduce the issue if I take out HAproxy in the middle
though.

The behavior of “return 499” can result in weird effects if it affects
a bunch of requests at once.

Do you mean “return 444”?


Maxim D.
http://nginx.org/en/donation.html

Hi guys,

So perhaps I shouldn’t have reused my existing topic and confused
things,
sorry!

I only did so because the symptom is exactly the same as the issue I
originally reported (SOME static content served from proxy cache
occasionally doesn’t make it to Firefox when using SPDY). If you
disable
either SPDY or the proxy cache the problem stops happening.

The originally reported problem stopped occurring after a fix was
implemented, but now I’m seeing it again since I put haproxy as the
upstream
(to get enhanced health checks etc).

The difference this time is that I’m not seeing the 499 errors (or 444)
in
the nginx log. I’m not returning 444 (via config) in these scenarios
either.

Firefox shows this:

GET https://whatever/postrequest [HTTP/1.1 200 OK 45ms]
GET https://whatever/css1 [HTTP/1.1 304 OK 25ms]
GET https://whatever/css2 [HTTP/1.1 304 OK 260ms]
GET https://whatever/js1 [HTTP/1.1 304 OK 260ms]
GET https://whatever/js2 [HTTP/1.1 304 OK 35ms]
GET https://whatever/image1 [HTTP/1.1 304 OK 40ms]
GET https://whatever/image2 [HTTP/1.1 304 OK 40ms]
GET https://whatever/image3 [HTTP/1.1 304 OK 40ms]
GET https://whatever/image4 [HTTP/1.1 304 OK 40ms]
GET https://whatever/image5 [5ms]
GET https://google.com/analyticsjs [HTTP/1.1 304 Not Modified 50ms]
GET https://whatever/image6 [0ms]
GET https://whatever/image7 [0ms]
GET https://whatever/image8 [0ms]
GET https://whatever/image9 [0ms]
GET https://whatever/image10 [0ms]

So you can see that around the time it loads the 5th image things go
wrong
and the 5th onward images all have no content. Firefox detail shows no
response headers/content for the 5th image onward.

The Haproxy and Nginx logs show 304 successfully for all of these
requests,
so I guess the connection is dropped somewhere and Nginx isn’t picking
it up
and still thinks it worked fine.

Any suggestions on how to proceed further on this? Happy to provide
specific details/logs via email if required.

Thanks,

spdyg

Posted at Nginx Forum:

On Wednesday 03 April 2013 02:49:55 spdyg wrote:

The originally reported problem stopped occurring after a fix was
GET https://whatever/css1 [HTTP/1.1 304 OK 25ms]
GET https://whatever/image7 [0ms]
up and still thinks it worked fine.

Any suggestions on how to proceed further on this? Happy to provide
specific details/logs via email if required.

Is there anything in the error log?

wbr, Valentin V. Bartenev


http://nginx.org/en/donation.html

On Tuesday 02 April 2013 18:46:01 Maxim D. wrote:

I’m now seeing the same issue again (static items coming from proxy
Firefox → Nginx 1.3.15 → HAproxy → Tomcat
Do you mean “return 444”?
Indeed, I accidentally confused 499 with 444 and as a result I
misunderstood the
issue. Thanks for correcting me.

wbr, Valentin V. Bartenev


http://nginx.org/en/donation.html

On Thursday 04 April 2013 02:43:15 spdyg wrote:

No there’s nothing in the error log. Access log shows 304’s for all
requests that failed.

Could you provide a debug log for the issue?
Here is the guide: A debugging log

wbr, Valentin V. Bartenev


http://nginx.org/en/donation.html

Thanks Valentin. I have emailed you (off-list) some debug logs /
screenshots.

Posted at Nginx Forum:

i am also experiencing this problem running 1.4.2.

cheers,
dlg

No there’s nothing in the error log. Access log shows 304’s for all
requests that failed.

Posted at Nginx Forum:

David Gwynne Wrote:

i am also experiencing this problem running 1.4.2.

Hi David. The response I got from my Valentin off-list was that there
is a
known incompatibility with SPDY and proxy cache, due to the way proxy
cache
will kill the connection to prevent it sending the body response (but
before
it has had a chance to finish dealing with all the SPDY requests, I
think).

Only solution now is to either disable SPDY or proxy cache (I went with
SPDY, of course).

I wonder if a note in the documentation might be in order to say there
are
known issues.

There is supposed to be a rewrite coming for proxy cache that will help,
but
not sure when it is likely to appear.

Phil

Posted at Nginx Forum:

No unfortunately, but I have since filed a bug in trac:

http://trac.nginx.org/nginx/ticket/428

Posted at Nginx Forum:

It would be nice to get some more informations about the current status
and
work.
Since RHEL 6.5 / Centos 6.5 there is openssl 1.0.1e available for me and
I
would
like to activate SPDY on our systems.

Posted at Nginx Forum:

Hi,

I have the same problem (nginx 1.4.3) when spdy is enabled and proxy
cache
too (with apache2 behind).

Any news about this bug ? I would really like to be able to activate
SPDY…

Thanks

Posted at Nginx Forum: