98: Address already in use

Server NGINX
1

Hi all,

Nginx has been working fine for months on my server. Recently, however,
I see that “service nginx configtest” reports the following:

the configuration file /etc/nginx/nginx.conf syntax is ok
[emerg]: listen() to 0.0.0.0:17942, backlog 511 failed (98: Address
already in use)
configuration file /etc/nginx/nginx.conf test failed

I believe nginx has been updated recently, but I’m not sure. It’s
currently at version 0.8.53. I’ve ran configtest before and have never
seen this error. I need to add a new site in conf.d, but I fear that
reloading or restarting afterwards will fail and my live sites will be
down.

netstat -tlnp lists 0.0.0.0:17942 as in use by Program name “nginx.conf”
(which seems weird, shouldn’t this just say “nginx”). Also, if I rename
the conf.d/site.conf file that includes the 17942 port entry, it just
fails on the next conf down the line on the following ip/port.

Now that I’ve typed out the entire problem, it occurs to me that the
problem could be the misnamed Program name. Maybe nginx thinks that a
different program is occupying those ports because the name is not
“nginx”. Does that seem likely? Also, was there a bug fixed somewhere
that resolved/caused the name change? I have a hunch that restarting
nginx will solve the problem, but I don’t want to risk it since it’s a
live server.

Any ideas? Thank you for reading.

Posted at Nginx Forum:

2

Hello!

On Thu, Apr 14, 2011 at 08:04:03AM -0400, slowgary wrote:

Hi all,

Nginx has been working fine for months on my server. Recently, however,
I see that “service nginx configtest” reports the following:

the configuration file /etc/nginx/nginx.conf syntax is ok
[emerg]: listen() to 0.0.0.0:17942, backlog 511 failed (98: Address
already in use)
configuration file /etc/nginx/nginx.conf test failed

Could you please provide details about OS you use?

EADDRINUSE should occur on bind() syscall earlier, and it’s
correctly handled there (ignored during configuration testing, and
no listen() call done).

I believe nginx has been updated recently, but I’m not sure. It’s
currently at version 0.8.53. I’ve ran configtest before and have never
seen this error. I need to add a new site in conf.d, but I fear that
reloading or restarting afterwards will fail and my live sites will be
down.

Reload/restart will likely succeed - listen() shouldn’t be even
called in your situation normally. It’s not clear why bind()
doesn’t report the same error though, most likely something
OS-related.

nginx will solve the problem, but I don’t want to risk it since it’s a
live server.

Name reported by netstat is completely unrelated (and it’s not
under nginx control).

Maxim D.

3

The OS is Centos 5.6 (Final)

Posted at Nginx Forum:

4

Hello,
could you stop nginx and check whats running on port 80?
by:*
lsof -i :80*
or
netstat -antp|grep 80

Hseyin Mert
http://www.hmert.com

5

“netstat -antp | grep 80” output can be found at:

Thanks.

Posted at Nginx Forum:

6

I don’t think it has anything to do with my configs, since things have
been working correctly for a long time. Just in case though, you can
see them here:

nginx.conf

mydomain.conf

Thanks again.

Posted at Nginx Forum:

7

On 14 Abr 2011 19h21 WEST, [email protected] wrote:

Sorry, I forgot to mention that I did not stop nginx. There are
live sites hosted on this server. My ultimate goal here is to avoid
downtime, otherwise I would definitely restart nginx to see what
happens.

There’s no downtime if you restart it properly: See

— appa

8

On Thu, Apr 14, 2011 at 08:04:03AM -0400, slowgary wrote:

Now that I’ve typed out the entire problem, it occurs to me that the
problem could be the misnamed Program name. Maybe nginx thinks that a
different program is occupying those ports because the name is not
“nginx”. Does that seem likely? Also, was there a bug fixed somewhere
that resolved/caused the name change? I have a hunch that restarting
nginx will solve the problem, but I don’t want to risk it since it’s a
live server.

Could you show output of
ps ax -o pid,ppid,%cpu,vsz,wchan,command|egrep ‘(nginx|PID)’

–
Igor S.

9

Sorry, I forgot to mention that I did not stop nginx. There are live
sites hosted on this server. My ultimate goal here is to avoid
downtime, otherwise I would definitely restart nginx to see what
happens.

Posted at Nginx Forum:

10

PID PPID %CPU VSZ WCHAN COMMAND
1867 1 0.0 14248 rt_sig nginx: master process /usr/sbin/nginx -c
/etc/nginx/nginx.conf
22231 1867 0.2 18704 - nginx: worker process
22232 1867 0.2 27852 - nginx: worker process
32653 32619 0.0 1932 pipe_w egrep (nginx|PID)

Thanks Igor.

Posted at Nginx Forum:

11

Hello!

On Thu, Apr 14, 2011 at 10:20:31AM -0400, slowgary wrote:

The OS is Centos 5.6 (Final)

Looking though linux code and testing on the exactly same centos -
it seems that linux may indeed return EADDRINUSE on listen().

But as far as I see the only situation when it may appear is two
bind()'s on conflicting ports followed by two listen()'s. Second
listen() call will fail with EADDRINUSE.

This shouldn’t happen in nginx unless you are running something
with 3rd party patches (or there is some other programm doing
bind()/listen() on the same ports, but this is racy and really
unlikely to be reproduceable).

Could you please show full config (with all listen’s - the only
you’ve provided seems not contain ports you see in complaints at
all…), “nginx -V” output and “nginx -t” output (this should be
identical to “service nginx configtest”, but just in case)?

Maxim D.

12

Last night I stopped and started nginx and it resolved the issue.

nginx -t
the configuration file /etc/nginx/nginx.conf syntax is ok
configuration file /etc/nginx/nginx.conf test is successful

It really seems that somehow nginx couldn’t realize that nginx itself
was bound to those addresses.

Posted at Nginx Forum:

13

Thanks Maxim.

I installed nginx from the EPEL repo, I don’t know of any 3rd party
patches. Is it common for 3rd party patched versions to exist in a
repo? Also, my current configuration has worked fine for months. I
believe this problem may have started after upgrading from CentOS 5.5 to
5.6.

=======================================================================
nginx -V output:

nginx version: nginx/0.8.53
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)
TLS SNI support disabled
configure arguments: --user=nginx --group=nginx
–prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log
–http-client-body-temp-path=/var/lib/nginx/tmp/client_body
–http-proxy-temp-path=/var/lib/nginx/tmp/proxy
–http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi
–http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi
–http-scgi-temp-path=/var/lib/nginx/tmp/scgi
–pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx
–with-http_ssl_module --with-http_realip_module
–with-http_addition_module --with-http_xslt_module
–with-http_image_filter_module --with-http_geoip_module
–with-http_sub_module --with-http_dav_module --with-http_flv_module
–with-http_gzip_static_module --with-http_random_index_module
–with-http_secure_link_module --with-http_degradation_module
–with-http_stub_status_module --with-http_perl_module --with-mail
–with-file-aio --with-mail_ssl_module --with-ipv6 --with-cc-opt=‘-O2 -g
-pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
–param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables’ --with-cc-opt=‘-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
–param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables’

=======================================================================
nginx -t output:

the configuration file /etc/nginx/nginx.conf syntax is ok
[emerg]: listen() to 0.0.0.0:17942, backlog 511 failed (98: Address
already in use)
configuration file /etc/nginx/nginx.conf test failed

=======================================================================
configs

fastcgi_params

nginx.conf

phpmyadmin.conf

mydomain1.conf
http://pastebin.com/r96E1Nmm

mydomain2.conf
http://pastebin.com/Gku1irY1

shared_rewrite_rules

Thank you for your help.

Posted at Nginx Forum:

14

I have the same problem and even if I stop nginx with /etc/init.d/nginx
stop
it is running on Debian Wheezy:

PID PPID %CPU VSZ WCHAN COMMAND
2709 1 0.0 127476 ep_pol nginx: worker process
2710 1 0.0 127716 ep_pol nginx: worker process
2711 1 0.0 127476 ep_pol nginx: worker process
2713 1 0.0 127444 ep_pol nginx: worker process
2714 1 0.0 125208 ep_pol nginx: cache manager process
3458 3218 0.0 6392 pipe_w egrep (nginx|PID)

Posted at Nginx Forum:

15

Hello!

On Tue, Apr 19, 2011 at 09:14:48AM -0400, slowgary wrote:

Last night I stopped and started nginx and it resolved the issue.

nginx -t
the configuration file /etc/nginx/nginx.conf syntax is ok
configuration file /etc/nginx/nginx.conf test is successful

It really seems that somehow nginx couldn’t realize that nginx itself
was bound to those addresses.

As I already said - the problem is that EADDRINUSE error was
returned on listen() call for some reason, not on bind() as it
should. This is itself POSIX violation, but I believe we don’t
mind adding Linux-specific workaround if the problem is common and
understood.

So the real question is “how this may happen?”. So far I wasn’t
able to reproduce it even on exactly the same OS. Please let us
know if you will be able to reproduce it.

Maxim D.