Socket CLOSE_WAIT problem and 404.html

Server NGINX
1

Hi All,

I have a website that hits by automatic scheduled job every hour from
thousands sources.

last time, my website is down because of thousands CLOSE_WAIT when I
check
using “netstat -na”
I already try to debug refer to Lots of CLOSE_WAIT sockets, nginx+php (WordPress site) - NGINX - Ruby-Forum
but I don’t know what to do with the debug log.

when I check the “lsof -p ”:
i found a lots of sockets (CLOSE_WAIT) and file open of my 404.html.
i think there’s a problem with my 404 configuration.

nginx 10500 nobody 357r REG 8,3 141 52396049
/home/j/html/404.html
nginx 10500 nobody 359r REG 8,3 141 52396049
/home/j/html/404.html
nginx 10500 nobody 360r REG 8,3 141 52396049
/home/j/html/404.html
nginx 10500 nobody 362u IPv4 174783995 TCP
70.36.100.46:http->y.net:49333 (CLOSE_WAIT)
nginx 10500 nobody 363r REG 8,3 141 52396049
/home/j/html/404.html
nginx 10500 nobody 365u IPv4 174497664 TCP
70.36.100.46:http->y.net:52096 (CLOSE_WAIT)
nginx 10500 nobody 366r REG 8,3 141 52396049
/home/j/html/404.html
nginx 10500 nobody 369u IPv4 174905532 TCP
70.36.100.46:http->y.net:50057 (CLOSE_WAIT)
nginx 10500 nobody 370r REG 8,3 141 52396049
/home/j/html/404.html
nginx 10500 nobody 371u IPv4 174729756 TCP
70.36.100.46:http->y.net:33731 (CLOSE_WAIT)

here is my “nginx -V”

nginx version: nginx/0.7.65
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-46)
TLS SNI support disabled
configure arguments: --with-debug --with-http_ssl_module
–pid-path=/var/run/nginx.pid --without-mail_smtp_module
–without-mail_imap_module --without-mail_pop3_module
–add-module=…/nginx_upload_module-2.0.10/

here is my config:

worker_processes 4;
error_log logs/error.log debug;
pid /var/run/nginx.pid;
events {
worker_connections 40960;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
include /usr/local/nginx/sites-enabled/*;
}

server {
listen 80;
server_name jj.x.y.z jj.x.y;

access_log /usr/local/nginx/logs/jj-access.log;
error_log  /usr/local/nginx/logs/jj-error.log;

location / {
    root   /home/j/html/;
    index  index.php index.html;
}

location ~ \.php$ {
    if (!-f /home/j/html/$fastcgi_script_name) { return 404; break;

}
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_read_timeout 270;
include /usr/local/nginx/conf/fastcgi_params;
fastcgi_param SCRIPT_FILENAME /home/j/html/$fastcgi_script_name;
}

error_page   500 502 503 504  /50x.html;
location = /50x.html {
    root   html;
}
error_page   404  /404.html;

}

does anyone have idea what’s the problem?

thank you.

2

Hello!

On Mon, Mar 08, 2010 at 11:16:25PM +0700, Oon Arfiandwi wrote:

i think there’s a problem with my 404 configuration.
/home/j/html/404.html

–add-module=…/nginx_upload_module-2.0.10/

Some questions in mostly random order:

Is nginx still answers requests or it’s stuck completely?

If it’s stuck - where? What nginx workers do? You should be able
to find it out by attaching to them with gdb and asking for
backtrace.

Which event method do you use? It should be logged at ‘notice’
level on nginx startup.

Are you able to reproduce problem? If yes, please try if you
are able to reproduce it without third party modules/patches.

here is my config:

worker_processes 4;
error_log logs/error.log debug;

Ok, you swithed on debug log here…

[…]

server {
listen 80;
server_name jj.x.y.z jj.x.y;

access_log /usr/local/nginx/logs/jj-access.log;
error_log  /usr/local/nginx/logs/jj-error.log;

… oops, you swithed it off for the server in question (default
logging level is ‘error’).

If you want to obtain usable debug log you should configure it at
global level and avoid overriding at other levels. Though in this
case I believe gdb will help better.

Maxim D.