Issues with my galaxy server

Server NGINX
1

Hi Team,

I am writing to gather some insight on how I may need to change my
configuration on Nginx. I have been running a galaxy server for 3+
years now and have begun experiencing issues with accessing several of
the largest libraries and get the following error: “The page you are
looking for is temporarily unavailable. Please try again later.” I
have read that this could possibly be a configuration issue with Nginx.
I am not confident that this is true, but can’t leave any resource
untouched. Has anyone reported this issue previously? Can you provide
me with insight? Additionally, how can I confirm what version of Nginx
I am using? I have it running on a linux server.

Thanks,

Iry T. Witham
Scientific Applications Administrator
Scientific Computing Group
Computational Sciences Dept.
The Jackson Laboratory
600 Main Street
Bar Harbor, ME 04609
Phone: 207-288-6744
email: [email protected]

[cid:B396D034-E6BC-42C0-A50E-0799B7C803A0]

The information in this email, including attachments, may be
confidential and is intended solely for the addressee(s). If you believe
you received this email by mistake, please notify the sender by return
email as soon as possible.

2

Hello!

On Thu, May 29, 2014 at 03:20:38PM +0000, Iry Witham wrote:

anyone reported this issue previously? Can you provide me with
insight? Additionally, how can I confirm what version of Nginx
I am using? I have it running on a linux server.

Try looking into logs.

–
Maxim D.
http://nginx.org/

3

“The page you are looking for is temporarily unavailable. Please try
again later.”
Can you provide me with insight?

The first thing would be to check what the actual HTTP error is (is it
some
5xx or 4xx etc) - some browsers (like IE also Chrome) tend to display a
“userfriendly” page without actually telling what the error is so it’s
hard
to help from technical aspect.

The second thing you should check are the error logs.
Usually there should be a reasonable (humanreadable) answer why a
particular
request has failed.

Additionally, how can I confirm what version of Nginx I am using? I have
it running on a linux server.

/path/to/nginx –v

or -V to see what compile time options were added

rr

4

I have checked the error.log and the following is what I am seeing:

2014/05/28 15:18:39 [info] 16146#0: *987 client closed prematurely
connection, so upstream connection is closed too while sending request
to
upstream, client: 10.40.42.12, server: localhost, request: “GET
/library_common/browse_library?sort=name&operation=browse&f-description=All
&f-name=All&f-deleted=False&cntrller=library_admin&async=false&show_item_ch
eckboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page=1 HTTP/1.1”,
upstream:
“http://127.0.0.1:8081/library_common/browse_library?sort=name&operation=br
owse&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&as
ync=false&show_item_checkboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page
=1”, host: “galaxy.jax.org”, referrer:
“http://galaxy.jax.org/library_admin/browse_libraries”
2014/05/28 15:18:45 [info] 16146#0: *1019 client closed prematurely
connection while reading client request line, client: 10.40.42.12,
server:
localhost
2014/05/28 15:27:03 [error] 16146#0: *1016 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client:
10.40.42.12, server: localhost, request: “GET
/library_common/browse_library?sort=name&operation=browse&f-description=All
&f-name=All&f-deleted=False&cntrller=library_admin&async=false&show_item_ch
eckboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page=1 HTTP/1.1”,
upstream:
“http://127.0.0.1:8080/library_common/browse_library?sort=name&operation=br
owse&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&as
ync=false&show_item_checkboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page
=1”, host: “galaxy.jax.org”, referrer:
“http://galaxy.jax.org/library_admin/browse_libraries”
2014/05/28 16:11:52 [error] 16146#0: *1604 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client:
10.20.23.189, server: localhost, request: “GET
/library_common/browse_library?sort=name&operation=browse&f-description=All
&f-name=All&f-deleted=False&cntrller=library_admin&async=false&show_item_ch
eckboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page=1 HTTP/1.1”,
upstream:
“http://127.0.0.1:8081/library_common/browse_library?sort=name&operation=br
owse&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&as
ync=false&show_item_checkboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page
=1”, host: “galaxy”, referrer:
“http://galaxy/library_admin/browse_libraries”
2014/05/28 16:12:12 [info] 16146#0: *1604 client 10.20.23.189 closed
keepalive connection
2014/05/28 16:18:20 [info] 16146#0: *1791 client closed prematurely
connection, so upstream connection is closed too while sending request
to
upstream, client: 10.1.101.207, server: localhost, request: “GET
/library_common/browse_library?sort=name&webapp=galaxy&f-description=All&f-
name=All&page=1&cntrller=library&show_item_checkboxes=false&async=false&ope
ration=browse&id=e49526e74f52ec39 HTTP/1.1”, upstream:
“http://127.0.0.1:8081/library_common/browse_library?sort=name&webapp=galax
y&f-description=All&f-name=All&page=1&cntrller=library&show_item_checkboxes
=false&async=false&operation=browse&id=e49526e74f52ec39”, host:
“galaxy.jax.org”, referrer:
“http://galaxy.jax.org/library/browse_libraries”
2014/05/28 16:22:52 [info] 16146#0: *1869 client 10.1.101.207 closed
keepalive connection
2014/05/28 16:23:52 [info] 16146#0: *1888 client 10.1.101.207 closed
keepalive connection
2014/05/28 16:24:16 [info] 16146#0: *1901 client 10.1.101.207 closed
keepalive connection
2014/05/28 16:26:37 [info] 16146#0: *1950 client closed prematurely
connection while reading client request line, client: 10.1.101.207,
server: localhost
2014/05/28 16:32:39 [info] 16146#0: *2047 client 209.222.197.239 closed
keepalive connection
2014/05/28 22:29:20 [info] 16146#0: *2082 client closed prematurely
connection while reading client request line, client: 192.168.253.157,
server: localhost
2014/05/28 22:29:20 [info] 16146#0: *2081 client closed prematurely
connection while reading client request line, client: 192.168.253.157,
server: localhost
2014/05/28 22:29:21 [info] 16146#0: *2083 client closed prematurely
connection while reading client request line, client: 192.168.253.157,
server: localhost
2014/05/28 22:29:21 [info] 16146#0: *2084 client closed prematurely
connection while reading client request line, client: 192.168.253.157,
server: localhost
2014/05/28 22:29:21 [info] 16146#0: *2085 client closed prematurely
connection while reading client request line, client: 192.168.253.157,
server: localhost
2014/05/29 08:53:40 [info] 16146#0: *2087 client closed prematurely
connection while reading client request line, client: 10.20.23.189,
server: localhost
2014/05/29 08:53:40 [info] 16146#0: *2088 client closed prematurely
connection while reading client request line, client: 10.20.23.189,
server: localhost
2014/05/29 08:53:40 [info] 16146#0: *2089 client closed prematurely
connection while reading client request line, client: 10.20.23.189,
server: localhost
2014/05/29 08:53:40 [info] 16146#0: *2090 client closed prematurely
connection while reading client request line, client: 10.20.23.189,
server: localhost
2014/05/29 08:53:40 [info] 16146#0: *2091 client closed prematurely
connection while reading client request line, client: 10.20.23.189,
server: localhost
2014/05/29 09:04:22 [error] 16146#0: *2099 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client:
10.20.23.189, server: localhost, request: “GET
/library_common/browse_library?sort=name&operation=browse&f-description=All
&f-name=All&f-deleted=False&cntrller=library_admin&async=false&show_item_ch
eckboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page=1 HTTP/1.1”,
upstream:
“http://127.0.0.1:8082/library_common/browse_library?sort=name&operation=br
owse&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&as
ync=false&show_item_checkboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page
=1”, host: “galaxy.jax.org”, referrer:
“http://galaxy.jax.org/library_admin/browse_libraries”
2014/05/29 09:20:30 [error] 16146#0: *2110 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client:
10.20.23.189, server: localhost, request: “GET
/library_common/browse_library?sort=name&operation=browse&f-description=All
&f-name=All&f-deleted=False&cntrller=library_admin&async=false&show_item_ch
eckboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page=1 HTTP/1.1”,
upstream:
“http://127.0.0.1:8082/library_common/browse_library?sort=name&operation=br
owse&f-description=All&f-name=All&f-deleted=False&cntrller=library_admin&as
ync=false&show_item_checkboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page
=1”, host: “galaxy”, referrer:
“http://galaxy/library_admin/browse_libraries”

I am currently running the following version of Nginx:

nginx version: nginx/0.8.53
built by gcc 4.3.4 [gcc-4_3-branch revision 152973] (SUSE Linux)
configure arguments: --prefix=/usr/local/nginx-0.8.53
–add-module=/usr/local/src/nginx_upload_module-2.2.0

I am planning an upgrade in the near future, but really need to get this
issue resolved.

Thanks,
Iry

On 5/29/14 11:36 AM, “Reinis R.” [email protected] wrote:

nginx Info Page
The information in this email, including attachments, may be
confidential and is intended solely for the addressee(s). If you believe
you received this email by mistake, please notify the sender by return
email as soon as possible.

5

2014/05/28 16:11:52 [error] 16146#0: *1604 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client:
10.20.23.189, server: localhost, request: “GET
/library_common/browse_library?sort=name&operation=browse&f-description=All
&f-name=All&f-deleted=False&cntrller=library_admin&async=false&show_item_ch
eckboxes=false&webapp=galaxy&id=ddd269e1ed2f8b5e&page=1 HTTP/1.1”,

In general the backend (galaxy) server can’t return the response in 60
seconds (default timeout).

You can either look at the backend (prefferably) or a quick workarround
(if
acceptable) would be to increase the proxy_read_timeout (
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
) - eg look into nginx.conf and check if any timeout directives are
present
and change or add them accordingly.

rr

6

I modified the proxy_read_timeout and that has resolved the issue.
Hopefully that will suffice until I upgrade.

Thanks,
Iry

On 5/29/14 12:16 PM, “Reinis R.” [email protected] wrote:

and change or add them accordingly.

rr

nginx mailing list
[email protected]
nginx Info Page

The information in this email, including attachments, may be
confidential and is intended solely for the addressee(s). If you believe
you received this email by mistake, please notify the sender by return
email as soon as possible.

7

Someone, perhaps you, changed something on your backend (your “galaxy”
server, which means absolutely nothing to anyone on this list, do be
aware), and they fucked it. Or “introduced an intermittent performance
issue which is resulting in a proportion of your proxied requests to
time out”, if you prefer. Personally, I don’t.