Connections time out before proxy_connect_timeout

Hi,

I’ve got Nginx set up to handle SSL connections, decode them, and
proxy them to a Java application. Every so often the Java stack pauses
while it does a GC, and the proxy connections time out. The odd thing
is that although proxy_connect_timeout is set to 70s, the connections
are timing out after around twenty seconds.

I tried setting proxy_connect_timeout to a smaller value (10s, say)
and Nginx does indeed time out the connections after that period, so
the setting is at least being read. But if you set it to anything
higher than 22s, it doesn’t make any difference - the connections are
still timed out after 22s with this error:

connect() failed (110: Connection timed out) while connecting to
upstream

I have also set proxy_read_timeout to a high value and it makes no
difference. I can reliably reproduce this issue on both Nginx 1.0.2
and 1.1.1. Is this a known bug or is there some other setting I should
look at? Could it be an issue with the backend server (Glassfish)? Or
related to the SSL listener? Any suggestions appreciated.

Here’s the full nginx.conf:

worker_processes 16;

#error_log /mnt/nginx-local/error.log debug;

set open fd limit to 30000

worker_rlimit_nofile 800000;
events {
worker_connections 8024;
}

http {
access_log /mnt/nginx-local/access.log combined;
include mime.types;
default_type application/octet-stream;

sendfile        on;

keepalive_timeout 100s;
proxy_read_timeout 100s;
# proxy_connect_timeout cannot be more than 75s
proxy_connect_timeout 70s;
proxy_send_timeout 300s;

server {
    listen       42;
    server_name  localhost;

    location / {
        root   html;
        index  index.html index.htm;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }

}

server {
    listen       7001 default ssl;
    server_name  localhost;

    ssl                  on;
    ssl_certificate      fullcert.crt;
    ssl_certificate_key  request.crt;

    ssl_session_timeout 7200m;

    ssl_protocols  SSLv2 SSLv3 TLSv1;
    ssl_ciphers 

ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers on;

    location / {
         client_max_body_size       10m;
         proxy_pass        http://localhost:79;
         proxy_set_header  X-Real-IP  $remote_addr;

    }

    location /controller {
            return 404;
    }
}
server {
    listen       7002 default ssl;
    server_name  localhost;

    ssl                  on;
    ssl_certificate      fullcert.crt;
    ssl_certificate_key  request.crt;

    ssl_session_timeout 7200m;

    ssl_protocols  SSLv2 SSLv3 TLSv1;
    ssl_ciphers 

ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers on;

    location / {
         client_max_body_size       10m;

         proxy_pass        http://localhost:79;
         proxy_set_header  X-Real-IP  $remote_addr;

    }
}

server {
    listen       80 default;
    server_name  localhost;

    location / {
         client_max_body_size       10m;
         proxy_pass        http://localhost:8080;
         proxy_set_header  X-Real-IP  $remote_addr;

    }
}
server {
    listen       443 default ssl;
    server_name  localhost;

    ssl                  on;
    ssl_certificate      fullcert.crt;
    ssl_certificate_key  request.crt;

    ssl_session_timeout 7200m;

    ssl_protocols  SSLv2 SSLv3 TLSv1;
    ssl_ciphers 

ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers on;

    location / {

         client_max_body_size       10m;
         proxy_pass        http://localhost:8080;
         proxy_set_header  X-Real-IP  $remote_addr;

    }
}

}

Regards,
John

Bitfield Consulting: we make software that makes things work

On Wed, Aug 31, 2011 at 6:07 PM, Maxim D. [email protected]
wrote:

Connect timeout is limited by your OS’s SYN retransmit count limit
and timeout. Under FreeBSD it’s 75s by default (3 retransmits
with exponential backoff: 3s + 6s + 12s + 24s), under Linux it’s
looks like something about 20s by default. Looks like you are
hitting this limit.

The only way to enlarge this limit is to tune OS, e.g. on Linux
try adjusting net.ipv4.tcp_syn_retries sysctl.

Yes, that fixed it. Thanks a lot!

Regards,
John

Bitfield Consulting: we make software that makes things work

Hello!

On Wed, Aug 31, 2011 at 05:31:15PM +0100, John Arundel wrote:

the setting is at least being read. But if you set it to anything
higher than 22s, it doesn’t make any difference - the connections are
still timed out after 22s with this error:

connect() failed (110: Connection timed out) while connecting to upstream

I have also set proxy_read_timeout to a high value and it makes no
difference. I can reliably reproduce this issue on both Nginx 1.0.2
and 1.1.1. Is this a known bug or is there some other setting I should
look at? Could it be an issue with the backend server (Glassfish)? Or
related to the SSL listener? Any suggestions appreciated.

Connect timeout is limited by your OS’s SYN retransmit count limit
and timeout. Under FreeBSD it’s 75s by default (3 retransmits
with exponential backoff: 3s + 6s + 12s + 24s), under Linux it’s
looks like something about 20s by default. Looks like you are
hitting this limit.

The only way to enlarge this limit is to tune OS, e.g. on Linux
try adjusting net.ipv4.tcp_syn_retries sysctl.

Maxim D.