Forum: NGINX SPDY + proxy cache = occasional 499 errors

Hi there,

I have been piloting the SPDY patch on a separate web server pointing to 
an
existing HTTP backend.

With just myself using this pilot web server, occasionally static 
resources
such as css or images are not loading in Firefox (usually when pressing 
F5
to reload a page)

Have done a bit of digging and have learnt this so far:

When issue occurs in Firefox, access log is showing 499 errors for the
missing static resources (when it should be 304)
Refreshing the page will eventually load the resources (and they will 
show
304 in the access log)
If I turn off the proxy cache for these static resources, the issue 
stops
If I turn off SPDY, the issue stops.
The issue obviously doesn't occur on the production web server running 
Nginx
1.2.x

Has this been seen before, or can you think of any reason why this might
happen?

I don't normally use Chrome but have also witnessed this once in Chrome, 
so
I don't think it's browser specific (however will try to get more 
confidence
of this).

Thanks.

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,233461,233461#msg-233461

On Sunday 02 December 2012 03:48:52 spdyg wrote:
>
> happen?
>
[...]

Thank you for testing. There is a reason for this in current spdy
implementation. I'm working on it.

 wbr, Valentin V. Bartenev

--
http://nginx.com/support.html
http://nginx.org/en/donation.html

I posted this query back in December:

http://forum.nginx.org/read.php?2,233497

And the issue was fixed shortly afterwards in the 1.3.x branch.

I'm now seeing the same issue again (static items coming from proxy 
cache +
SPDY not displaying).

However this time, the log does not show 499 error anymore.. it just 
shows
successful 304.  However, looking at firefox web console, the first few
static items come back as 304 without problems, then the next set of 
static
items come back with no response at all.. (no response header, no body).

The setup looks like this:

Firefox -> Nginx 1.3.15 -> HAproxy -> Tomcat

Has this bug crept back in, or is this a new one?  Also reproduced on
1.3.13.

Can't seem to reproduce the issue if I take out HAproxy in the middle
though.

spdyg

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,233497,238019#msg-238019

On Tuesday 02 April 2013 14:54:30 spdyg wrote:
> successful 304.  However, looking at firefox web console, the first few
> Can't seem to reproduce the issue if I take out HAproxy in the middle
> though.
>

The behavior of "return 499" can result in weird effects if it affects
a bunch of requests at once.

  wbr, Valentin V. Bartenev

--
http://nginx.org/en/donation.html

Hello!

On Tue, Apr 02, 2013 at 05:16:46PM +0400, Valentin V. Bartenev wrote:

> > However this time, the log does not show 499 error anymore.. it just shows
> >
> > Can't seem to reproduce the issue if I take out HAproxy in the middle
> > though.
> >
>
> The behavior of "return 499" can result in weird effects if it affects
> a bunch of requests at once.

Do you mean "return 444"?

--
Maxim Dounin
http://nginx.org/en/donation.html

On Tuesday 02 April 2013 18:46:01 Maxim Dounin wrote:
> > > I'm now seeing the same issue again (static items coming from proxy
> > > Firefox -> Nginx 1.3.15 -> HAproxy -> Tomcat
> Do you mean "return 444"?
Indeed, I accidentally confused 499 with 444 and as a result I 
misunderstood the
issue. Thanks for correcting me.

  wbr, Valentin V. Bartenev

--
http://nginx.org/en/donation.html

Hi guys,

So perhaps I shouldn't have reused my existing topic and confused 
things,
sorry!

I only did so because the symptom is exactly the same as the issue I
originally reported (SOME static content served from proxy cache
occasionally doesn't make it to Firefox when using SPDY).  If you 
disable
either SPDY or the proxy cache the problem stops happening.

The originally reported problem stopped occurring after a fix was
implemented, but now I'm seeing it again since I put haproxy as the 
upstream
(to get enhanced health checks etc).

The difference this time is that I'm not seeing the 499 errors (or 444) 
in
the nginx log.  I'm not returning 444 (via config) in these scenarios
either.

Firefox shows this:

GET https://whatever/postrequest [HTTP/1.1 200 OK 45ms]
GET https://whatever/css1 [HTTP/1.1 304 OK 25ms]
GET https://whatever/css2 [HTTP/1.1 304 OK 260ms]
GET https://whatever/js1 [HTTP/1.1 304 OK 260ms]
GET https://whatever/js2 [HTTP/1.1 304 OK 35ms]
GET https://whatever/image1 [HTTP/1.1 304 OK 40ms]
GET https://whatever/image2 [HTTP/1.1 304 OK 40ms]
GET https://whatever/image3 [HTTP/1.1 304 OK 40ms]
GET https://whatever/image4 [HTTP/1.1 304 OK 40ms]
GET https://whatever/image5 [5ms]
GET https://google.com/analyticsjs [HTTP/1.1 304 Not Modified 50ms]
GET https://whatever/image6 [0ms]
GET https://whatever/image7 [0ms]
GET https://whatever/image8 [0ms]
GET https://whatever/image9 [0ms]
GET https://whatever/image10 [0ms]

So you can see that around the time it loads the 5th image things go 
wrong
and the 5th onward images all have no content.  Firefox detail shows no
response headers/content for the 5th image onward.

The Haproxy and Nginx logs show 304 successfully for all of these 
requests,
so I guess the connection is dropped somewhere and Nginx isn't picking 
it up
and still thinks it worked fine.

Any suggestions on how to proceed further on this?  Happy to provide
specific details/logs via email if required.

Thanks,

spdyg

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,233497,238039#msg-238039

On Wednesday 03 April 2013 02:49:55 spdyg wrote:
> The originally reported problem stopped occurring after a fix was
> GET https://whatever/css1 [HTTP/1.1 304 OK 25ms]
> GET https://whatever/image7 [0ms]
> up and still thinks it worked fine.
>
> Any suggestions on how to proceed further on this?  Happy to provide
> specific details/logs via email if required.
>

Is there anything in the error log?

  wbr, Valentin V. Bartenev

--
http://nginx.org/en/donation.html

No there's nothing in the error log.  Access log shows 304's for all
requests that failed.

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,233497,238073#msg-238073

On Thursday 04 April 2013 02:43:15 spdyg wrote:
> No there's nothing in the error log.  Access log shows 304's for all
> requests that failed.
>

Could you provide a debug log for the issue?
Here is the guide: http://nginx.org/en/docs/debugging_log.html

  wbr, Valentin V. Bartenev

--
http://nginx.org/en/donation.html

Thanks Valentin.  I have emailed you (off-list) some debug logs /
screenshots.

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,233497,238195#msg-238195