Upstream timed out

Hi.

I’m the new admin of a website and its servers. I have a frontend nginx
proxy server and an Apache backend.
I sometimes have errors :

2012/04/12 17:44:33 [error] 13442#0: *74193485 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client: 98.87.76.65, server: www.mysite.tld, request: “POST /my/url
HTTP/1.1”, upstream: “http://12.23.34.45:80/my/url”, host:
“www.mysite.tld”, referrer: “https://www.mysite.tld/another/url
2012/04/12 17:44:33 [error] 13442#0: *74193485 no live upstreams while
connecting to upstream, client: 98.87.76.65, server: www.mysite.tld,
request: “POST /my/url HTTP/1.1”, upstream: “http://backend/my/url”,
host: “www.mysite.tld”, referrer: “https://www.goeland.fr/another/url

(I’ve juste anonymized the urls and ip addresses)

Here are the configuration files :

conf.d/proxy.conf :

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
client_max_body_size 32M;
client_body_buffer_size 256k;
client_header_buffer_size 128k;
proxy_connect_timeout 30;
proxy_send_timeout 30;
proxy_read_timeout 30;
proxy_buffer_size 32k;
proxy_buffers 64 32k;
proxy_busy_buffers_size 128k;

conf.d/loadbalancer.conf :

upstream backend {
ip_hash;
# web1

server 127.0.0.1:8098 backup;

# web2
server 44.33.22.11;
#web3
server 11.22.33.44 down;

}

sites-enabled/mysite.conf :

server {
listen 80;
listen 443 ssl;
server_name www.mysite.tld www3.mysite.tld;

    access_log off; #/home/nginx/log/mysite.access.log;
    #error_log  /dev/null crit;

    # Possible level values : debug | info | notice | warn | error |

crit
error_log /home/nginx/log/mysite.error.log notice;
log_not_found off;

    #set $proto "http";
    #if ( $server_port = "443" ) {
     #set $proto "https";
    #}

    # serve static files directly
    location ~*

^.+.(jpg|jpeg|gif|css|png|js|ico|html|pdf|zip|tgz|gz|bz2|swf)$ {
expires 10d;
root /home/mysite/www;
}

    location / {
            #proxy_set_header X-Forwarded-Proto $proto;
            proxy_pass       http://backend;
            add_header       X-Whom [email protected];
    }

    error_page 405 = $uri;
    error_page   500 502 503 504  /50x.html;

    location = /50x.html {
            root /home/mysitewww;
    }

   location = /favicon.ico { access_log off; log_not_found off; }

}

The problem seems to be recent and sometimes happens immediatly after
clients have paid through the bank online credit card form which is
integrated (we do not redirect to the bank website). It’s a real problem
for us since when it happens at that moment, clients think their order
and paiement have not been saved and they order a second time (and also
pay a second time).

I don’t really know what i’m looking for : can it be a network
connection problem between frontend and backend servers ? Is it because
of the configuration of the proxy of the server ? or… ?

Thanks for any help.

Posted at Nginx Forum:

Have you tried to increase the proxy_connect_timeout,
proxy_send_timeout,
proxy_read_timeout values? Maybe to 60 or 180 seconds?

----- Original Message -----
From: “OlivierC” [email protected]
To: [email protected]
Sent: Friday, April 13, 2012 10:59 AM
Subject: upstream timed out

Hello!

On Fri, Apr 13, 2012 at 04:59:55AM -0400, OlivierC wrote:

“www.mysite.tld”, referrer: “https://www.mysite.tld/another/url
2012/04/12 17:44:33 [error] 13442#0: *74193485 no live upstreams while
connecting to upstream, client: 98.87.76.65, server: www.mysite.tld,
request: “POST /my/url HTTP/1.1”, upstream: “http://backend/my/url”,
host: “www.mysite.tld”, referrer: “https://www.goeland.fr/another/url

[…]

proxy_read_timeout 30;

[…]

The problem seems to be recent and sometimes happens immediatly after
clients have paid through the bank online credit card form which is
integrated (we do not redirect to the bank website). It’s a real problem
for us since when it happens at that moment, clients think their order
and paiement have not been saved and they order a second time (and also
pay a second time).

I don’t really know what i’m looking for : can it be a network
connection problem between frontend and backend servers ? Is it because
of the configuration of the proxy of the server ? or… ?

How long it takes for a backend server to generate response? The
30s read timeout might be just not enough.

Maxim D.