I’m seeing some unusual behaviour in one specific client interaction
with nginx and wonder if anyone has seen anything similar or can offer
any explanation as to why this might be happening. I’m running nginx
version 0.7.65.
From a curl command line I’m issuing a POST request to nginx over https.
The POST request has a content-type of multipart/form-data and I’m using
it to send some data and upload a file of just under 1Mb.
nginx is being used as a proxy for upstream web servers built with
mochiweb. Under normal conditions, everything works fine. The file is
uploaded successfully and stored in the back-end database. During
testing I’m deliberately causing a 500 Internal Server Error to be
returned by the upstream web servers to nginx, and this is where I’m
seeing some unusual behaviour.
The 500 status code is indeed returned but there is something strange
going on with the client/nginx/upstream web server interaction. The
whole interaction takes about 10 seconds. Curl rapidly receives the
response headers with the 500 status code but then waits. It never
receives the response body sent by the client - in this case the
content-length is only 200 bytes and the client receives this length in
the Content-Length response header. Curl reports an error code 18,
meaning that it expected a response body of (in this case) 200 bytes but
received less than this (zero in fact as curl reports that the transfer
was closed with 200 bytes remaining to read).
If I reduce the size of the file uploaded by this method, the problem
disappears and curl receives the entire response body just fine. Also,
everything is just fine with other web services where I deliberately
introduce a 500 response code. It’s just for the file upload of a
significantly sized file (about 1Mb) that I see this behaviour.
I know that nginx will receive the file in its entirety and store it in
a temporary location on the server before it talks to the upstream web
server, and it clearly is talking to the upstream web server so it seems
it has received the file completely before it attempts to return the 500
code.
I’ve also bypassed nginx and let curl talk directly to the upstream web
server. In this case, the 500 response is received rapidly and in full
(body intact).
On Fri, Jun 18, 2010 at 07:04:07AM -0400, woodsmar wrote:
mochiweb. Under normal conditions, everything works fine. The file is
content-length is only 200 bytes and the client receives this length in
the Content-Length response header. Curl reports an error code 18,
meaning that it expected a response body of (in this case) 200 bytes but
received less than this (zero in fact as curl reports that the transfer
was closed with 200 bytes remaining to read).
If I reduce the size of the file uploaded by this method, the problem
disappears and curl receives the entire response body just fine. Also,
everything is just fine with other web services where I deliberately
introduce a 500 response code. It’s just for the file upload of a
significantly sized file (about 1Mb) that I see this behaviour.
Please produce debug log. See here for details:
If I’m right in my assumption that your backend returns 500 before
actually reading request body - the following patch may help:
I tried the patch but unfortunately that didn’t work as I still get the
same behaviour. With debugging turned on the following is an excerpt
from the error log:
There is in fact an error in there - “readv() failed (104: Connection
reset by peer)”. So is this the upstream server closing the connection
before it received the entire body, because it already responded?
On Fri, Jun 18, 2010 at 10:25:10AM -0400, woodsmar wrote:
I tried the patch but unfortunately that didn’t work as I still get the
same behaviour. With debugging turned on the following is an excerpt
from the error log:
Backend reseted connection instead of closing it, that’s the
reason why you don’t see reply.
In your particular case nginx actually got full reply but
discarded body as it got error from backend before it was able to
send it. This theoretically may be handled gracefully (and I even
posted patches for this a while ago). But note that with reset
there is a good chance that nginx won’t see data at all as they
will be discarded in socket buffer by OS.