Mod_cgi for nginx - anyone?

mike · February 7, 2009, 12:07am

Igor, et al:

Is there a reason that there has not been an attempt to directly call
CGI programs (compiled CGI, perl CGI, whatever) in nginx?

I would love to take out proxying to apache if I could.

Is there a blocking nature in CGI or something else that makes it
impossible to function, or has there been no interest? I have a
handful of apps I have to support (bugzilla, mailman, etc.) that I
have to proxy to apache right now.

I’d be willing to put some money down for it, too. (and mod_svn !)

mike

mike · February 7, 2009, 12:35am

There is a fast-cgi interface.
Why would you really want CGI?

mike · February 7, 2009, 12:36am

If you want to write CGI script in Perl you can in-fact use the perl
module
and use
nginx::simple PERL module.

mike · February 7, 2009, 12:44am

afaik, the only way to do that is to alter the CGI application to use
FCGI.pm or whatever?

the point of this is to run the application unaltered.

mike · February 7, 2009, 12:43am

Because these do not support fastcgi.

Unless there is a php-fpm type daemon for CGI scripts, that talks
fastcgi to the webserver but CGI to the CGI scripts themselves?

mike · February 7, 2009, 11:29am

On Fri, Feb 06, 2009 at 02:55:25PM -0800, mike wrote:

have to proxy to apache right now.

I’d be willing to put some money down for it, too. (and mod_svn !)

nginx is not general purpose server, it’s rather highload server.
CGI is not compatible with highload: if you run Apache/CGI, then CGI
will became bottleneck much earlier than Apache.

There are two ways to implement CGI inside nginx:

simple one: just fork()ing worker process that has received a request
for CGI and exec() a CGI program. It’s simple enough, but has a lot
of overhead. Besides CGI programs will run with worker privilege only.
complex way: to run a special CGI manager (probably with root
privilege
as master process) and to pass it requests/sockets using Unix domain
sockets.
Then the manager can fork/exec CGI programs with required privileges
and with minimal overhead to nginx workers.

The second way will require some time to program, but the outcome will
be much similar like just proxying to mini_httpd
( mini_httpd ).

BTW, it seems that using Apache with several worker processes (2-5) for
bugzilla, mailman, etc. will not consume much CPU/memory: look top.

mike · February 7, 2009, 12:14pm

On Sat, Feb 07, 2009 at 02:33:09AM -0800, mike wrote:

of overhead. Besides CGI programs will run with worker privilege only.

and it requires the CGI program to be modified, yeah?

No.

into a small httpd?
No, the special CGI manager is a nginx process that fork()ed by master
nginx
process (like worker processes). The communication protocol between
workers and the manager is not HTTP: a worker starts to handle a
request,
then it sees that the request should be handled by CGI, something like

  location /mailman/ {
       cgi_script  ...;
       cgi_user    ...;
       ...
  }

and pass the request and client socket to the manager.

BTW, it seems that using Apache with several worker processes (2-5) for
bugzilla, mailman, etc. will not consume much CPU/memory: look top.

I’m just looking at it from simplifying the system administration.

I know it is not the most performant, those tools are not my preferred
ones, I just have to support hosting them right now…

I understand this, but currently nginx has more priority tasks.

mike · February 7, 2009, 12:31pm

On Feb 7, 2009, at 2:57 AM, Igor S. [email protected] wrote:

simple one: just fork()ing worker process that has received a
request
for CGI and exec() a CGI program. It’s simple enough, but has a lot
of overhead. Besides CGI programs will run with worker privilege
only.

and it requires the CGI program to be modified, yeah?

No.

Got any urls for examples then?

will
request,
BTW, it seems that using Apache with several worker processes

Sure. So it isn’t impossible if someone else was hired to do it. I was
wondering if there was a technical reason for it.

On a different note, I’ve paid someone to implement mod_auth_gss (more
or less) into nginx. So it might support kerberos tickets and single
sign on soon (at least SPNEGO)

mike · February 7, 2009, 9:46pm

mike wrote:

afaik, the only way to do that is to alter the CGI application to use
FCGI.pm or whatever?

You wrong, it’s just a point of using FastCGI wrapper. I have dozen
awstats installations running FastCGI wrapper from http://www.nginx.eu/

mike · February 7, 2009, 11:43am

On Sat, Feb 7, 2009 at 2:16 AM, Igor S. [email protected] wrote:

nginx is not general purpose server, it’s rather highload server.
CGI is not compatible with highload: if you run Apache/CGI, then CGI
will became bottleneck much earlier than Apache.

There are two ways to implement CGI inside nginx:

simple one: just fork()ing worker process that has received a request
for CGI and exec() a CGI program. It’s simple enough, but has a lot
of overhead. Besides CGI programs will run with worker privilege only.

and it requires the CGI program to be modified, yeah?

complex way: to run a special CGI manager (probably with root privilege
as master process) and to pass it requests/sockets using Unix domain sockets.
Then the manager can fork/exec CGI programs with required privileges
and with minimal overhead to nginx workers.

The second way will require some time to program, but the outcome will
be much similar like just proxying to mini_httpd
( mini_httpd ).

this is like starting out with something like php-fpm, and morphing
into a small httpd?

BTW, it seems that using Apache with several worker processes (2-5) for
bugzilla, mailman, etc. will not consume much CPU/memory: look top.

I’m just looking at it from simplifying the system administration.

I know it is not the most performant, those tools are not my preferred
ones, I just have to support hosting them right now…

mike · February 7, 2009, 9:48pm

Igor S. ha scritto:

[…]

nginx is not general purpose server, it’s rather highload server.
CGI is not compatible with highload: if you run Apache/CGI, then CGI
will became bottleneck much earlier than Apache.

This is not a problem from the Nginx point of view.
Nginx should however put a limit on the number of concurrent CGI
requests.
It can use a queue to store outgoing request, and if the queue grows too
much return a 503 Service Unavailable, setting Retry-After to a
reasonable (computed?) value.

No browser support 503 response, however.

There are two ways to implement CGI inside nginx:

simple one: just fork()ing worker process that has received a request
for CGI and exec() a CGI program. It’s simple enough, but has a lot
of overhead. Besides CGI programs will run with worker privilege only.

Nginx could use seteuid, instead of setuid.
So that it can reacquire root privileges.

As for CGI support, some time ago I was trying to implement it.
The idea was to use unix domain sockets (socketpair), and reusing the
http_upstream module.

However I gave up, a lot of code for connection/upstream setup must be
rewritten, and I hate to write boiler plate code; and expecially having
to maintain it :).

In Nginx one can also easily close all current opened file descriptors,
in child process; Nginx keeps all opened connection in the
ngx_cycle->free_connections variable so one can write a closefrom
function.

Regards Manlio P.

mike · February 7, 2009, 11:01pm

On piÄ…, lut 06, 2009 at 02:55:25 -0800, mike wrote:

I’d be willing to put some money down for it, too. (and mod_svn !)

Cool, how much will you pay for fcgiwrap then?

http://nginx.localdomain.pl/

or is there something fundamental missing from that? I’m running over a
hundred of those at the moment and they seem to do their job (talk
fastcgi on one end and cgi on the other), no cgi code modification
required,
pathinfo support, what’s not to like?

Best regards,
Grzegorz N.

mike · February 8, 2009, 11:24am

On Sat, Feb 07, 2009 at 09:38:41PM +0100, Manlio P. wrote:

It can use a queue to store outgoing request, and if the queue grows too
much return a 503 Service Unavailable, setting Retry-After to a
reasonable (computed?) value.

No browser support 503 response, however.

I mean that using CGI as main response handler on highload site is
useless.
CGI may be used for lowload tasks: monitoring, administration, etc.
those
can be easly handled by Apache or by proxying to Apache.

There are two ways to implement CGI inside nginx:

simple one: just fork()ing worker process that has received a request
for CGI and exec() a CGI program. It’s simple enough, but has a lot
of overhead. Besides CGI programs will run with worker privilege only.

Nginx could use seteuid, instead of setuid.
So that it can reacquire root privileges.

Yes, but it’s not so safe as setuid().

As for CGI support, some time ago I was trying to implement it.
The idea was to use unix domain sockets (socketpair), and reusing the
http_upstream module.

However I gave up, a lot of code for connection/upstream setup must be
rewritten, and I hate to write boiler plate code; and expecially having
to maintain it :).

Yes, CGI module should use modified upstream module.

In Nginx one can also easily close all current opened file descriptors,
in child process; Nginx keeps all opened connection in the
ngx_cycle->free_connections variable so one can write a closefrom function.

This can be resolved using fcntl(F_SETFD, FD_CLOEXEC) too.

mike · February 8, 2009, 10:32am

I googled around originally for nginx + CGI stuff (specifically
bugzilla, mailman) and could not find anything.

This might work just fine! I will have to try it out. Looks like it is
just like php-fpm, but for CGI?

mike · February 8, 2009, 4:03am

I’m currently testing this with PHPmotion and it works perfectly using
spawn-fcgi to get it started. Now I need to update the code to get Nginx
upload progress bar in there as the one that comes with the scrip is
meant for (cr)apache.

I was wondering, Grzegorz, if there is some “algorithm” as to how to
determine the optimal number of children to run.

Jim

mike · February 8, 2009, 11:31am

On Sat, Feb 07, 2009 at 03:21:23AM -0800, mike wrote:

There are two ways to implement CGI inside nginx:

Got any urls for examples then?

No, as I know there is no attempts to implemnet this in nginx.

This CGI support is the same as in lighttpd/thttpd/etc: they have
single process that fork()/exec() CGI program.

( mini_httpd ).

bugzilla, mailman, etc. will not consume much CPU/memory: look top.
Sure. So it isn’t impossible if someone else was hired to do it. I was
wondering if there was a technical reason for it.

No, there is no technical reason that does not allow to implement CGI
in nginx, this is wasting time only (from my point of view :).

mike · February 8, 2009, 2:10pm

On Sat, Feb 07, 2009 at 09:50:47PM -0500, Jim O. wrote:

I’m currently testing this with PHPmotion and it works perfectly using spawn-fcgi to get it started. Now I need to update the code to get Nginx upload progress bar in there as the one that comes with the scrip is meant for (cr)apache.

I was wondering, Grzegorz, if there is some “algorithm” as to how to determine the optimal number of children to run.

Basically, the number of concurrent CGI requests you wish to serve and
then some (a single fcgiwrap does not multiplex multiple requests yet,
sorry). It’s actually better to have a few slack ones (they are really
very cheap, especially after the first one – double-digit
_kilo_bytes).

Best regards,
Grzegorz N.

mike · February 8, 2009, 2:13pm

On Sun, Feb 08, 2009 at 01:19:12AM -0800, mike wrote:

I googled around originally for nginx + CGI stuff (specifically
bugzilla, mailman) and could not find anything.

I’ve been spamming this list with fcgiwrap ever since I wrote it many
months ago, you must’ve been asleep It’s even been on the wiki for
some time now.

This might work just fine! I will have to try it out. Looks like it is
just like php-fpm, but for CGI?

Roughly. It still needs a process manager of some sort. Have a look at
my website, there’s a very simple launcher script there that you might
find suitable.

Best regards,
Grzegorz N.

mike · February 8, 2009, 6:02pm

OK, I’m sure that I’m missing something, but I’m having trouble getting
more than one instance going.

I tried

/usr/local/bin/spawn-fcgi -C 5 -a 127.0.0.1 -p 10000 -u nobody -g

nobody -f /usr/local/bin/fcgiwrap but only saw one instance generated.

Jim

Do you have a sample script to start it with “X” number of children

mike · February 8, 2009, 8:16pm

On Sun, Feb 08, 2009 at 11:48:32AM -0500, Jim O. wrote:

OK, I’m sure that I’m missing something, but I’m having trouble getting more than one instance going.

I tried

/usr/local/bin/spawn-fcgi -C 5 -a 127.0.0.1 -p 10000 -u nobody -g nobody -f /usr/local/bin/fcgiwrap but only saw one instance generated.

-C only sets an environment variable for php-fcgi, it doesn’t actually
run multiple instances of anything.

Do you have a sample script to start it with “X” number of children

See http://nginx.localdomain.pl/ for a simple Perl launcher.

Best regards,
Grzegorz N.