# Monthly Gateway Timeout

**URL:** <https://www.ruby-forum.com/t/monthly-gateway-timeout/204075>\
**Category:** NGINX\
**Created:** [March 4, 2011, 10:14am UTC](https://www.ruby-forum.com/t/monthly-gateway-timeout/204075 "2011-03-04T10:14:36Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [March 4, 2011, 10:14am UTC](https://www.ruby-forum.com/t/monthly-gateway-timeout/204075/1 "2011-03-04T10:14:36Z")

</div>

Hi All

I’m experiencing a very strange network problem that occurs every four  
to six weeks and lasts for approximately one hour. I can not manually  
provoke the problem and, during this very hour, I can not resolve it  
even with rebooting the server. Even stranger, two physically separated  
servers suffer from the same problem at the same time. Both servers use  
nginx as SSL reverse proxy. Each server has disjunct domains to handle.

During one hour, we just see: Nginx - Gateway Timout. After one hour, it  
suddenly works again. It started around four months ago. Note that all  
other network traffic is not affected, only nginx HTTPS and HTTP.

So, what’s in common with both servers:

1. Hardware (UltraSPARC T2 Plus).
2. OS (Solaris 10 U9 latest patch level).
3. Time (both servers use the exact same NTP-controlled time).
4. Switch.
5. Firewall (I replaced the firewall four weeks ago, the error still  
just appeared).
6. Nginx 0.8.46-0.8.54, same configuration but for different domains  
internally hosted at different servers, compiled with Solaris OpenSSL.

I observed that nginx, during this mysterious hour, mistakenly proxies  
the requests back to the original IP on random ports instead of the  
proxy IP and that these requests are blocked by the firewall.

Because two different machines are affected at the same time and it  
cannot be resolved by a restart of nginx or a reboot of the whole  
server, and it resolves itself after approximately one hour, my guess is  
that some time-dependent error occurs in nginx.

I will replace nginx with apache to verify the problem actually is nginx  
and not the OS, switch are whatever and then wait and hope 🙂

Does anyone have an idea how to locate or investigate this problem?

Kind regards,  
Marc

Posted at Nginx Forum:

> **[Monthly Gateway Timeout](https://forum.nginx.org/read.php?2,180289,180289#msg-180289)**
>
> Hi All I'm experiencing a very strange network problem that occurs every four to six weeks and lasts for approximately one hour. I can not manually provoke the problem and, during this very hour, I can not resolve it even with rebooting the server....

---

<div class="post-metadata">

**Author:** ![Marc\_Kramis](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/d9b06d/32.png) [@Marc\_Kramis](https://www.ruby-forum.com/u/Marc_Kramis)\
**Post date:** [March 4, 2011, 11:04am UTC](https://www.ruby-forum.com/t/monthly-gateway-timeout/204075/2 "2011-03-04T11:04:51Z")

</div>

Hello!

On Fri, Mar 04, 2011 at 04:12:59AM -0500, Marc Kramis wrote:

> suddenly works again. It started around four months ago. Note that all  
> other network traffic is not affected, only nginx HTTPS and HTTP.
> 
> So, what’s in common with both servers:
> 
> 1. Hardware (UltraSPARC T2 Plus).

Some endianness issue?..

> proxy IP and that these requests are blocked by the firewall.
> 
> Because two different machines are affected at the same time and it  
> cannot be resolved by a restart of nginx or a reboot of the whole  
> server, and it resolves itself after approximately one hour, my guess is  
> that some time-dependent error occurs in nginx.
> 
> I will replace nginx with apache to verify the problem actually is nginx  
> and not the OS, switch are whatever and then wait and hope 🙂
> 
> Does anyone have an idea how to locate or investigate this problem?

You may want to follow [GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/Debugging) and provide  
config and debug log (if you are able to obtain one), as well as  
nginx -V output. This may help to investigate the problem if the  
issue is actually in nginx.

Maxim D.

---

<div class="post-metadata">

**Author:** ![Marc\_Kramis](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/d9b06d/32.png) [@Marc\_Kramis](https://www.ruby-forum.com/u/Marc_Kramis)\
**Post date:** [March 4, 2011, 11:34am UTC](https://www.ruby-forum.com/t/monthly-gateway-timeout/204075/3 "2011-03-04T11:34:21Z")

</div>

Hi Marc,  
can you reproduce this issue by changing system clock back to the  
“unfortunate hour”?

Also, is the wrong proxy address always the same or does nginx try to  
connect to random address and/or port?

Best regards,  
Piotr S. \< removed\_email\_address@domain.invalid \>

---

<div class="post-metadata">

**Author:** ![Marc\_Kramis](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/d9b06d/32.png) [@Marc\_Kramis](https://www.ruby-forum.com/u/Marc_Kramis)\
**Post date:** [March 5, 2011, 12:19pm UTC](https://www.ruby-forum.com/t/monthly-gateway-timeout/204075/4 "2011-03-05T12:19:30Z")

</div>

Hi Maxim, Hi Piotr

nginx -V:

nginx version: nginx/0.8.54  
built by Sun C 5.10 SunOS\_sparc Patch 141861-06 2010/07/28  
TLS SNI support disabled  
configure arguments: --with-cc=/opt/sunstudio12.1/bin/cc  
–with-cpp=/opt/sunstudio12.1/bin/cc --with-cc-opt=‘-xtarget=ultraT2plus  
-xO5 -I /usr/sfw/include’ --with-ld-opt=‘-R/usr/sfw/lib -L/usr/sfw/lib’  
–prefix=/nginx --user=daemon --group=daemon --with-http\_ssl\_module  
–with-pcre=../pcre-8.12 --with-zlib=../zlib-1.2.5

Note that the bug also appeared with optimization level O3.

nginx.conf:

# — Basic Configuration

* * *

user daemon daemon;  
error\_log /nginx/logs/error.log warn;  
ssl\_engine pkcs11;  
worker\_processes 16;

events {  
worker\_connections 256;  
}

# — HTTP Configuration

* * *

http {

log\_format LOG ‘$remote\_addr - $remote\_user  
[$time\_local] “$request” $status $body\_bytes\_sent “$http\_referer”  
“$http\_user\_agent”’;  
access\_log /nginx/logs/$host.access.log LOG;

server\_tokens off;

gzip on;  
gzip\_vary on;  
gzip\_proxied any;  
gzip\_types text/plain text/xml text/css text/javascript  
image/svg+xml application/xhtml+xml application/xml application/rss+xml  
application/atom+xml application/x-javascript application/json;

client\_body\_buffer\_size 128k;  
client\_max\_body\_size 256m;  
client\_body\_temp\_path /nginx/client\_body\_temp 1 2;

proxy\_read\_timeout 3600;  
proxy\_redirect off;  
proxy\_pass\_header Set-Cookie;  
proxy\_temp\_path /nginx/proxy\_temp;

# — [https://foo](https://foo) -------------------------------------------

server {

```
listen 446;
server_name foo;

ssl on;
ssl_certificate /nginx/ssl/foo.crt;
ssl_certificate_key /nginx/ssl/foo.key;
ssl_session_cache shared:SSL:8m;

location /bar {
  rewrite ^/(.*)$ https://foo/bar/ permanent;
}

location /bar/ {
  proxy_pass http://10.10.10.1:8080/bar/;
}

location / {
  rewrite ^/(.*)$ https://foo permanent;
}

```

}

server {

```
listen 80 default;
server_name _;
server_name_in_redirect off;

location / {
  rewrite ^/(.*)$ http://foo permanent;
}

```

}

}

The error log is full of the following error (only during the  
problematic hour):

2011/03/04 08:40:28 [error] 20062#0: \*507995 upstream timed out (145:  
Connection timed out) while reading response header from upstream,  
client: _ **IP** _, server: _ **SERVER** _, request: “GET _ **URL** _  
HTTP/1.1”, upstream: “_ **UPSTREAM** _”, host: “_ **HOST** _”, referrer:  
“\*\*\*REFERER”

I just realized that only during this hour, the firewall lists blocked  
outgoing traffic exactly to the client IPs of the error log at random  
ports, i.e., I assume that during this hour, nginx mistakenly sends the  
proxied request back to the client instead of the internal server.

Regards,  
Marc

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,180289,180623#msg-180623](http://forum.nginx.org/read.php?2,180289,180623#msg-180623)

---

<div class="post-metadata">

**Author:** ![Marc\_Kramis](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/d9b06d/32.png) [@Marc\_Kramis](https://www.ruby-forum.com/u/Marc_Kramis)\
**Post date:** [March 5, 2011, 1:45pm UTC](https://www.ruby-forum.com/t/monthly-gateway-timeout/204075/5 "2011-03-05T13:45:55Z")

</div>

Hello!

On Sat, Mar 05, 2011 at 06:18:00AM -0500, Marc Kramis wrote:

[…]

> log\_format LOG ‘$remote\_addr - $remote\_user  
> [$time\_local] “$request” $status $body\_bytes\_sent “$http\_referer”  
> “$http\_user\_agent”’;  
> access\_log /nginx/logs/$host.access.log LOG;

Just a side note: this, along with no root specified in server{}  
blocks, makes your system vulnerable to inode exhaustion attack  
(i.e. attacker may create arbitrary number of log files on your  
system, eventually bringing it down).

[…]

> proxy\_read\_timeout 3600;

Uhm… Its really big one. It’s unlikely that clients will wait  
for so long for an answer.

[…]

> ```
> location /bar/ {
> proxy_pass http://10.10.10.1:8080/bar/;
> 
> ```

With this configuration nginx will convert ip/port to binary form  
once (while reading config), and will use it (binary) while doing  
connect()'s. It is very unlikely that something bad will happen  
in nginx at this point.

[…]

> The error log is full of the following error (only during the  
> problematic hour):
> 
> 2011/03/04 08:40:28 [error] 20062#0: \*507995 upstream timed out (145:  
> Connection timed out) while reading response header from upstream,  
> client: _ **IP** _, server: _ **SERVER** _, request: “GET _ **URL** _  
> HTTP/1.1”, upstream: “_ **UPSTREAM** _”, host: “_ **HOST** _”, referrer:  
> “\*\*\*REFERER”

Words “while reading response header” in error\_log suggest that  
connection to upstream was established (and request was sent), but  
upstream failed to generate an answer in time.

> I just realized that only during this hour, the firewall lists blocked  
> outgoing traffic exactly to the client IPs of the error log at random  
> ports, i.e., I assume that during this hour, nginx mistakenly sends the  
> proxied request back to the client instead of the internal server.

With your proxy\_read\_timeout it’s very likely that appropriate  
firewall states for connections with clients were already expired,  
and that’s why your firewall complains about nginx trying to  
reply “504 Gateway timeout” back to clients once it detects  
timeout.

> From here it looks like it’s your backend problem, not nginx. And  
> strange firwall complains you see are due to timeouts  
> misconfiguration (you have to configure timeouts on your firewall  
> at least as big as ones in nginx).

Maxim D.

---

<div class="post-metadata">

**Author:** ![Marc\_Kramis](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/d9b06d/32.png) [@Marc\_Kramis](https://www.ruby-forum.com/u/Marc_Kramis)\
**Post date:** [March 5, 2011, 3:29pm UTC](https://www.ruby-forum.com/t/monthly-gateway-timeout/204075/6 "2011-03-05T15:29:12Z")

</div>

Hi Igor

It’s a 32-bit executable.

Regards,  
Marc

Posted at Nginx Forum:

> **[Re: Monthly Gateway Timeout](https://forum.nginx.org/read.php?2,180289,180656#msg-180656)**
>
> Hi All I'm experiencing a very strange network problem that occurs every four to six weeks and lasts for approximately one hour. I can not manually provoke the problem and, during this very hour, I can not resolve it even with rebooting the server....

---

<div class="post-metadata">

**Author:** ![Marc\_Kramis](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/d9b06d/32.png) [@Marc\_Kramis](https://www.ruby-forum.com/u/Marc_Kramis)\
**Post date:** [March 5, 2011, 1:50pm UTC](https://www.ruby-forum.com/t/monthly-gateway-timeout/204075/7 "2011-03-05T13:50:55Z")

</div>

On 04.03.2011, at 12:12, Marc Kramis wrote:

> UltraSPARC T2 Plus

How does nginx run: as 32-bit or 64-bit executable?

–  
Igor S.

> **[Igor Sysoev](http://sysoev.ru/en/)**

---

<div class="post-metadata">

**Author:** ![Marc\_Kramis](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/d9b06d/32.png) [@Marc\_Kramis](https://www.ruby-forum.com/u/Marc_Kramis)\
**Post date:** [March 5, 2011, 3:29pm UTC](https://www.ruby-forum.com/t/monthly-gateway-timeout/204075/8 "2011-03-05T15:29:13Z")

</div>

Hi Maxim

The timeout is required because some (old) webapps take quite some time  
to generate reports and stuff. I try to remove this global setting and  
just apply it for the old webapps.

Thanks for the hint about the logs. I will centralize that in a single  
file.

I’m now replacing one nginx by an apache and then wait a few weeks. If  
both go down the same time again, the problem is definitively not in the  
reverse proxy.

Best regards,  
Marc

Posted at Nginx Forum:

> **[Re: Monthly Gateway Timeout](https://forum.nginx.org/read.php?2,180289,180655#msg-180655)**
>
> Hi All I'm experiencing a very strange network problem that occurs every four to six weeks and lasts for approximately one hour. I can not manually provoke the problem and, during this very hour, I can not resolve it even with rebooting the server....