Nginx and/or Passenger failing

Server NGINX
1

Hi,

Envirionment:
Passenger 3.0.9
nginx 1.0.6
Suse SLE 11
Rails 3.0.7
Ruby 1.9.2 p136
Quad core
8 GB
Single Rails application

nginx settings:

worker_processes 3;
worker_rlimit_nofile 8192;
worker_priority 0;
events {
multi_accept off;
worker_connections 4096;
}

pid /var/run/nginx.pid;
error_log logs/error.log debug;
http {
passenger_root
/usr/local/rvm/gems/ruby-1.9.2-p136/gems/passenger-3.0.9;
passenger_ruby /usr/local/rvm/wrappers/ruby-1.9.2-p136/ruby;
passenger_max_pool_size 4;
passenger_min_instances 1;
keepalive_timeout 180;
client_body_buffer_size 128k;
client_header_buffer_size 2k;
large_client_header_buffers 32 64k;
passenger_pool_idle_time 300;
passenger_log_level 2;
passenger_use_global_queue on;
include /opt/nginx/conf/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] ’
'“$request” $status $bytes_sent ’
'“$http_referer” “$http_user_agent” ’
‘“$gzip_ratio”’;
access_log logs/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay off;
gzip on;
gzip_http_version 1.0;
gzip_comp_level 2;
gzip_proxied any;
gzip_min_length 1000;
gzip_types text/plain text/css application/x-javascript
text/xml application/xml application/xml+rss text/javascript;
…

I’m not sure if the problem lies with passenger or nginx, after a period
of time the passenger as well as the nginx master process shuts down.
When I look in the log file I have the following:

2011/09/27 16:45:36 [info] 6408#0: *204 client 61.88.141.194 closed
keepalive connection
[ pid=6381 thr=140088026498848
file=ext/common/LoggingAgent/LoggingServer.h:829 time=2011-09-27
16:45:46.849 ]: Flushing all sinks (periodic action)
[ pid=6381 thr=140088026498848
file=ext/common/LoggingAgent/LoggingServer.h:829 time=2011-09-27
16:46:01.848 ]: Flushing all sinks (periodic action)
[ pid=6381 thr=140088026498848
file=ext/common/LoggingAgent/LoggingServer.h:829 time=2011-09-27
16:46:16.848 ]: Flushing all sinks (periodic action)
[ pid=6381 thr=140088026498848
file=ext/common/LoggingAgent/LoggingServer.h:829 time=2011-09-27
16:46:31.848 ]: Flushing all sinks (periodic action)
[ pid=6381 thr=140088026498848
file=ext/common/LoggingAgent/LoggingServer.h:829 time=2011-09-27
16:46:46.849 ]: Flushing all sinks (periodic action)
[ pid=6381 thr=140088026498848
file=ext/common/LoggingAgent/LoggingServer.h:829 time=2011-09-27
16:47:01.849 ]: Flushing all sinks (periodic action)
[ pid=6381 thr=140088026498848
file=ext/common/LoggingAgent/LoggingServer.h:829 time=2011-09-27
16:47:16.848 ]: Flushing all sinks (periodic action)
[ pid=6381 thr=140088026498848
file=ext/common/LoggingAgent/LoggingServer.h:829 time=2011-09-27
16:47:31.848 ]: Flushing all sinks (periodic action)
[ pid=6371 thr=139975134312192 file=ext/common/Watchdog.cpp:1057
time=2011-09-27 16:47:38.756 ]: Web server did not exit gracefully,
forcing shutdown of all service processes…
2011/09/27 16:49:00 [crit] 6410#0: *250 connect() to
unix:/passenger_helper_server failed (2: No such file or directory)
while connecting to upstream, client: 120.146.194.176, serv
er: localhost, request: “GET /candidates/new HTTP/1.1”, upstream:
“passenger:unix:/passenger_helper_server:”, host: “61.88.141.221”,
referrer: “http://61.88.141.221/home/view_repor
ts”

When I look at the running processes the nginx master is no longer
running but the workers are still running, I have to kill them before I
can restart. All passenger processes are not longer running.

As I said I’m not sure which process is causing the other to close and I
have no idea what the problem may be, so any help greatly appreciated.

Etienne.

Posted at Nginx Forum:

2

Sergey A. Osokin Wrote:

passenger_root
    root

0:00,09 PassengerHelperAgent

host.
nginx mailing list
[email protected]
nginx Info Page

Hi Sergey,

Thank you so much for the prompt reply.

Sorry I just can’t get this working, nginx fires up ok but nothing
happens when I telnet. I checked passenger using passenger-status
–verbose
and it shows the details for passenger but when I check for running
passenger processes it’s not running.

When I do GET / HTTP/1.0 nothing happens.

Am I missing something?

user …;
events {
multi_accept off;
worker_connections 4096;
}

http {
include mime.types;
default_type application/octet-stream;

passenger_root
/usr/local/rvm/gems/ruby-1.9.2-p136/gems/passenger-3.0.9;
passenger_ruby /usr/local/rvm/wrappers/ruby-1.9.2-p136/ruby;

passenger_default_user …;

server {
listen 80;
server_name localhost;

root /home/etienne/test/stub/rack/public;
passenger_enabled on;
passenger_use_global_queue on;
}
}

Posted at Nginx Forum:

3

On Tue, Sep 27, 2011 at 03:54:07AM -0400, springbok wrote:

Single Rails application

client_header_buffer_size 2k;
access_log  logs/access.log  main;

…
[ pid=6381 thr=140088026498848
16:46:46.849 ]: Flushing all sinks (periodic action)
time=2011-09-27 16:47:38.756 ]: Web server did not exit gracefully,
running but the workers are still running, I have to kill them before I
can restart. All passenger processes are not longer running.

As I said I’m not sure which process is causing the other to close and I
have no idea what the problem may be, so any help greatly appreciated.

Could you try very simple config with small application from passenger
distro, i.e. something like this.

http {
include mime.types;
default_type application/octet-stream;

passenger_root

/usr/local/rvm/gems/ruby-1.9.2-p136/gems/passenger-3.0.9;
passenger_ruby /usr/local/rvm/wrappers/ruby-1.9.2-p136/ruby;
passenger_default_user www;

server {
    listen       127.0.0.1:80;
    server_name  localhost;

    root

/usr/local/rvm/gems/ruby-1.9.2-p136/gems/passenger-3.0.9/test/stub/rack/public;
passenger_enabled on;
passenger_use_global_queue on;
}
}

After restart your nginx with passenger you should find nginx and
passenger
processes.

% ps auxww | grep nginx | grep -v nginx
root 18932 4,0 15,9 44464 39656 ?? Ss 8:46 0:00,01 nginx:
master process /usr/local/sbin/nginx
www 18933 4,0 16,0 44464 39904 ?? S 8:46 0:00,02 nginx:
worker process (nginx)
% ps uaxww | grep -i passenger | grep -v grep
root 18923 0,0 1,0 5760 2508 ?? Is 8:46 0:00,07
PassengerWatchdog
root 18926 0,0 1,5 15052 3820 ?? I 8:46 0:00,09
PassengerHelperAgent
root 18929 0,0 2,7 13176 6620 ?? I 8:46 0:01,75 ruby:
Passenger spawn server (ruby)
nobody 18930 0,0 1,4 6688 3600 ?? I 8:46 0:00,07
PassengerLoggingAgent

% telnet 127.0.0.1 80
Trying 127.0.0.1…
Connected to localhost.
Escape character is ‘^]’.
GET / HTTP/1.0

HTTP/1.1 200 OK
Content-Type: text/html
Connection: close
Status: 200
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 3.0.9
Server: nginx/1.0.6 + Phusion Passenger 3.0.9 (mod_rails/mod_rack)

hello worldConnection closed by foreign host.

Does it possible reproduce your problem for simple configuration?

–
Sergey A. Osokin
[email protected]
[email protected]

4

Ok, I’ve got passenger loading and appearing in the process list, I
created a index.html file and was able to display it doing a GET / but
when I try GET / HTTP/1.0 it just sits there and nothing happens. If I
remove the index.html I get “Hello world” returned, no mention of
passenger.

Posted at Nginx Forum:

5

springbok Wrote:

include       mime.types;

    passenger_use_global_queue on;

8:46
8:46

Connected to localhost.
(mod_rails/mod_rack)
[email protected]

server_name localhost;

root /home/etienne/test/stub/rack/public;
passenger_enabled on;
passenger_use_global_queue on;
}
}

Hi Sergey,

One other thing I’d mention is that our application works great for most
of the time but intermittently it fails with the symptoms I describe so
I’m not sure I can reproduce it with the simple setup if I did get it
going.

Regards,

Etienne.

Posted at Nginx Forum:

6

On Tue, Sep 27, 2011 at 05:35:57AM -0400, springbok wrote:

Ok, I’ve got passenger loading and appearing in the process list, I
created a index.html file and was able to display it doing a GET / but
when I try GET / HTTP/1.0 it just sits there and nothing happens. If I
remove the index.html I get “Hello world” returned, no mention of
passenger.

So, if you got "Hello, world this means passenger works well with
default installation. Could you please check your application or try to
use
previous version of nginx and/or passenger.

–
Sergey A. Osokin
[email protected]
[email protected]

7

On Tue, Sep 27, 2011 at 10:03:30AM -0400, springbok wrote:

remove the index.html I get “Hello world”
Sergey A. Osokin
I was using 0.8.53 when I started getting the errors and decided to
upgrade before posting my question.

Do you believe this is an nginx or passenger problem, is nginx causing
passenger to shutdown or the other way around?

I don’t know.

To me it looks like the
watchdog process is getting an error from nginx and closing the
passenger processes.

I think you should ask about your problem authors of passenger at first.

I just really don’t know where to start looking, could it be something
external for example the DB the app uses is running on a different
server maybe it’s a network problem, what about rogue messages is that
possible, the IT manager did say he removed the firewall to the machine.
I’m sure he has something else in place, but this app has been running
fine for the last 6 months, it only started happening after the weekend
when we went live and the DB was moved from the same machine to
another.

Sorry for all the questions but I’m really stumped and don’t know what
to try next.

–
Sergey A. Osokin
[email protected]
[email protected]

8

Sergey A. Osokin Wrote:

passenger.
[email protected]

nginx mailing list
[email protected]
nginx Info Page

Hi Sergey,

I was using 0.8.53 when I started getting the errors and decided to
upgrade before posting my question.

Do you believe this is an nginx or passenger problem, is nginx causing
passenger to shutdown or the other way around? To me it looks like the
watchdog process is getting an error from nginx and closing the
passenger processes.

We have 8GB of memory and 4 processors, do you think it could be memory
related/CPU load related, I know it’s difficult to say without seeing
our app and the load it places on the system, but if you had to guess?
It has plenty of memory available so I don’t think that’s a problem, the
IT manager for the client which I developed the app is convinced the
shutdown is caused by CPU load. When monitoring the load the passenger
processes spike up to around 50-80% but settle down to 10% of less when
idling, at the moment they are < 1% as the load is low. The shutdown has
happened when there is not a lot of load so I’m not convinced.

I just really don’t know where to start looking, could it be something
external for example the DB the app uses is running on a different
server maybe it’s a network problem, what about rogue messages is that
possible, the IT manager did say he removed the firewall to the machine.
I’m sure he has something else in place, but this app has been running
fine for the last 6 months, it only started happening after the weekend
when we went live and the DB was moved from the same machine to
another.

Sorry for all the questions but I’m really stumped and don’t know what
to try next.

Thanks again for your help.

Etienne.

Posted at Nginx Forum: