Reported upload issues with website

Hello All,

We are running our site on a RHEL server. Nginx is setup to proxy
traffic
to Tomcat. We are hearing reports from some users of issues uploading
files. These issues are sporadically happening and it sounds like users
with flakey connections are the ones affected. Based on what’s
reported, I
think there is an issue with a users connection timing out during the
upload
process. This is causing uploads to fail instead of keeping the session
open until the timeout clears. The app (Grails) and Tomcat have a
pretty
long timeout set (30 min) so it doesn’t appear to be something there.
I’m
wondering if there are backend timeout issues with the proxy. Maybe a
problem where Nginx closes the connection to the user or Tomcat once the
user has timed out for a period of time. We’re running Nginx 0.7.64 and
I’ve added both configuration files to pastie.org.

nginx.conf = http://www.pastie.org/908712

virtual.conf = http://www.pastie.org/908714

I’ve looked into the following directives and implementing the “suggest”
values on staging tonight.

proxy_connect_timeout…
Default: proxy_connect_timeout 60;
suggest: proxy_connect_timeout 240;

proxy_ignore_client_abort…
default: proxy_ignore_client_abort off
suggest: proxy_ignore_client_abort on

proxy_max_temp_file_size
default: proxy_max_temp_file_size 1G;
suggested: proxy_max_temp_file_size 4G;

proxy_read_timeout
default: proxy_read_timeout 60
suggested: proxy_read_timeout 240

proxy_send_timeout
default: proxy_send_timeout 60
suggested: proxy_send_timeout 240

client_body_timeout
default: 60
suggested: client_body_timeout 240

[ Show » https://tracker.solutionset.com/browse/SAHMEDB-101 ]

Blake
Scrivnerhttps://tracker.solutionset.com/secure/ViewProfile.jspa?name=bscrivneradded
a comment - 07/Apr/10
04:05 PM I’m going to add those configurations above and below to stage
and
test to make sure everything is fine. If it is, then I’ll setup
production
to run with these new values. client_body_timeout default: 60 suggested:
client_body_timeout 240

The problem with this issue is I haven’t been able to reproduce it, nor
have
clear cut steps to reproduce. No precise timestamps have been given so
it
makes using the logs pretty difficult. Here is an interesting clip from
the
error.log that might help identify the issue. Most of my research has
shown
this panic happens due to perl but this Nginx was compiled without the
Perl
module.

panic: MUTEX_LOCK (22) [op.c:352].
2010/01/11 09:52:11 [error] 2250#0: *10 upstream timed out (110:
Connection
timed out) while reading response header from upstream, client:
12.228.184.2, server: fileblaze.net, request: “POST
/soundblaze/upload/fileUpload/ HTTP/1.1”, upstream: "
http://127.0.0.1:8080/soundblaze/upload/fileUpload/“, host: "
upload.fileblaze.net

Do you think those settings will help? Do I need all of these settings?
Any other suggestions I should try out? Please let me know if there is
any
more data you would like me to gather to help troubleshoot this.

Thanks in advance for all your help,

Blake

Hello!

On Fri, Apr 09, 2010 at 12:02:04PM -0700, Blake Scrivner wrote:

[…]

The problem with this issue is I haven’t been able to reproduce it, nor have
clear cut steps to reproduce. No precise timestamps have been given so it
makes using the logs pretty difficult. Here is an interesting clip from the
error.log that might help identify the issue. Most of my research has shown
this panic happens due to perl but this Nginx was compiled without the Perl
module.

panic: MUTEX_LOCK (22) [op.c:352].

This is indeed message from perl. If you think you have no perl
perl compiled in - you probably should re-investigate what you are
actually running. For on-disk binary you may run nginx -V to see
how nginx was configured.

This may not match running binary though, so you may want to
follow binary upgrade procedure as specified in docs (or just
restart nginx) to make sure your running binary match on-disk one.

2010/01/11 09:52:11 [error] 2250#0: *10 upstream timed out (110: Connection
timed out) while reading response header from upstream, client:
12.228.184.2, server: fileblaze.net, request: “POST
/soundblaze/upload/fileUpload/ HTTP/1.1”, upstream: "
http://127.0.0.1:8080/soundblaze/upload/fileUpload/“, host: "
upload.fileblaze.net

This message indicate that upstream wasn’t able to process request
in time. You should tune proxy_read_timeout if you want to take
upstream some more time to process request.

Maxim D.