Lots of "No route to host" in Nginx error log

Hi All,

We got lots of “No route to host” in Nginx error log.

2010/01/05 22:28:01 [error] 29366#0: *26063920 connect() failed (113: No
route to host) while connecting to upstream, client: 60.217.152.205,
server: test.com, request: GET /herundong/index.php HTTP/1.1, upstream:
http://x.y.z.a:80/herundong/index.php, host: events.test.com
Using curl on the Nginx proxy server like
curl -v -H “Host: events.test.comhttp://x.y.z.a/herundong/index.php
works as expected.
I searched in Nginx source code, and find even no ‘No route to host’ in
its source code.

Nginx version: 0.7.63
Backend are Apache servers.

Current Nginx connections ~2k;
Current Apache Connections ~800(counting all active and inactive
connections)
I saw on some connections of apache, there was a high Send-Q on
established nginx-apache connections.
Current Apache load average is under 1;

Any suggestions?

Thanks in advance.

2010-01-06

xufengnju

xufengnju at 2010-1-6 10:04 wrote:

Hi All,

We got lots of “No route to host” in Nginx error log.

2010/01/05 22:28:01 [error] 29366#0: *26063920 connect() failed (113: No route to host) while connecting to upstream, client: 60.217.152.205, server: test.com, request: GET /herundong/index.php HTTP/1.1, upstream: http://x.y.z.a:80/herundong/index.php, host: events.test.com

“No route to host” is the error string of system call connect. It’s
error value is 113. It means the server can’t find the route to
the backend server. This’s the network problem, not nginx.


Weibin Y.

sometimes caused by upstreams’ firewall.

2010/1/6 Weibin Y. [email protected]:

[email protected]
http://nginx.org/mailman/listinfo/nginx


Ren Xiaolei

I find that error to , My Configuration is 4 upstreams, three upstreams
is 1 cpu、2G memory,one is 2cpu、4Gmemory,the 2cpu’s doesn’t have this
error,but the three 1cpus have this error,those server all in the same
LAN ,anyone have the suggestion?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,37426,66936#msg-66936

On Tue, 2010-03-23 at 08:26 -0400, zhangyun112112 wrote:

I find that error to , My Configuration is 4 upstreams, three
upstreams is 1 cpu、2G memory,one is 2cpu、4Gmemory,the 2cpu’s doesn’t
have this error,but the three 1cpus have this error,those server all
in the same LAN ,anyone have the suggestion?

As 崔玉松 said, this is a network issue, not a Nginx issue. You should
check the proper function and configuration of both your LAN and the
machines attached to it.

Regards,
Cliff

Hi,

“No route to host” is not nginx error, if you use linux, please man
route
for detail information
please make sure that the firewall does not block the port to your
backend

2010/1/6 xufengnju [email protected]

Hi,

We have that problem for awhile at our site. During the last two days,
I investigated it and I pretty sure that it is cause by the netfilter
firewall. Our backend servers are running RHEL5 and the 'iptables, rule
that accept new connections from the proxy looks like:

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp -s
iii.jjj.kkk.lll --dport 9005 -j ACCEPT

iii.jjj.kkk.lll is the IP address of the proxy server.

And the last rule of the chain ‘RH-Firewall-1-INPUT’ is:

-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited

Yesterday I inserted a new rule in between in order to log incoming
packets from the proxy that are going to be rejected. The rule looks
like:

-A RH-Firewall-1-INPUT -s iii.jjj.kkk.lll -j LOG

The result was that for each ‘No route to host’ error logged in the
error_log file on the proxy server, there is a corresponding rejected
packet logged on one of the backend servers. That means that our
‘iptables’ rules need a little bit of tuning.

I thought of two solutions:

  1. Remove the ‘state match’ test in the ACCEPT rule so the new rule
    looks like:

-A RH-Firewall-1-INPUT -m tcp -p tcp -s iii.jjj.kkk.lll --dport 9005 -j
ACCEPT

  1. Add a new rule to ‘iptables’ which issues a REJECT with options
    ‘tcp-reset’ for the problematics packets. I hope that the proxy will
    reissue the request while receiving the RST flag from the backend. The
    rule should be inserted just before the last one and it should look
    like:

-m tcp -p tcp -s iii.jjj.kkk.lll --dport 9005 -j REJECT --reject-with
tcp-reset

Tonight I will try the solution number 2. If the proxy react as I think
it will, That will be a better solution than the number 1 which I will
try eventually if the solution number 2 won’t work.

Hope that this message will help you.

Sorry for the poor english.

Paul

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,37426,67536#msg-67536

pacudes Wrote:

-p tcp -s iii.jjj.kkk.lll --dport 9005 -j ACCEPT
Yesterday I inserted a new rule in between in
tuning.
REJECT with options ‘tcp-reset’ for the
better solution than the number 1 which I will try
eventually if the solution number 2 won’t work.

Hope that this message will help you.

Sorry for the poor english.

Paul

Hi again,

Finally the solution #2 solves the problem. I just made a last minute
modification ( I restrict the rule to the -SYN packet) to the rule in
‘iptables’. The right rule looks like this:

-A RH-Firewall-1-INPUT -m tcp -p tcp -s iii.jjj.kkk.lll --dport 9005
–syn -j REJECT --reject-with tcp-reset

Don’t forget to adjust the source IP and destination port (-dport)
according to your installation.

Regards

Paul

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,37426,67798#msg-67798

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs