Forum: NGINX mod_cgi for nginx - anyone?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
mike (Guest)
on 2009-02-07 01:07
(Received via mailing list)
Igor, et al:

Is there a reason that there has not been an attempt to directly call
CGI programs (compiled CGI, perl CGI, whatever) in nginx?

I would love to take out proxying to apache if I could.

Is there a blocking nature in CGI or something else that makes it
impossible to function, or has there been no interest? I have a
handful of apps I have to support (bugzilla, mailman, etc.) that I
have to proxy to apache right now.

I'd be willing to put some money down for it, too. (and mod_svn !)

- mike
Atif G. (Guest)
on 2009-02-07 01:35
(Received via mailing list)
There is a fast-cgi interface.
Why would you really want CGI?
Atif G. (Guest)
on 2009-02-07 01:36
(Received via mailing list)
If you want to write CGI script in Perl you can in-fact use the perl
module
and use
nginx::simple PERL  module.
mike (Guest)
on 2009-02-07 01:43
(Received via mailing list)
Because these do not support fastcgi.

Unless there is a php-fpm type daemon for CGI scripts, that talks
fastcgi to the webserver but CGI to the CGI scripts themselves?
mike (Guest)
on 2009-02-07 01:44
(Received via mailing list)
afaik, the only way to do that is to alter the CGI application to use
FCGI.pm or whatever?

the point of this is to run the application unaltered.
Igor S. (Guest)
on 2009-02-07 12:29
(Received via mailing list)
On Fri, Feb 06, 2009 at 02:55:25PM -0800, mike wrote:

> have to proxy to apache right now.
>
> I'd be willing to put some money down for it, too. (and mod_svn !)

nginx is not general purpose server, it's rather highload server.
CGI is not compatible with highload: if you run Apache/CGI, then CGI
will became bottleneck much earlier than Apache.

There are two ways to implement CGI inside nginx:

1) simple one: just fork()ing worker process that has received a request
for CGI and exec() a CGI program. It's simple enough, but has a lot
of overhead. Besides CGI programs will run with worker privilege only.

2) complex way: to run a special CGI manager (probably with root
privilege
as master process) and to pass it requests/sockets using Unix domain
sockets.
Then the manager can fork/exec CGI programs with required privileges
and with minimal overhead to nginx workers.

The second way will require some time to program, but the outcome will
be much similar like just proxying to mini_httpd
( http://acme.com/software/mini_httpd/ ).

BTW, it seems that using Apache with several worker processes (2-5) for
bugzilla, mailman, etc. will not consume much CPU/memory: look top.
mike (Guest)
on 2009-02-07 12:43
(Received via mailing list)
On Sat, Feb 7, 2009 at 2:16 AM, Igor S. <removed_email_address@domain.invalid> 
wrote:

> nginx is not general purpose server, it's rather highload server.
> CGI is not compatible with highload: if you run Apache/CGI, then CGI
> will became bottleneck much earlier than Apache.
>
> There are two ways to implement CGI inside nginx:
>
> 1) simple one: just fork()ing worker process that has received a request
> for CGI and exec() a CGI program. It's simple enough, but has a lot
> of overhead. Besides CGI programs will run with worker privilege only.

and it requires the CGI program to be modified, yeah?

> 2) complex way: to run a special CGI manager (probably with root privilege
> as master process) and to pass it requests/sockets using Unix domain sockets.
> Then the manager can fork/exec CGI programs with required privileges
> and with minimal overhead to nginx workers.
>
> The second way will require some time to program, but the outcome will
> be much similar like just proxying to mini_httpd
> ( http://acme.com/software/mini_httpd/ ).

this is like starting out with something like php-fpm, and morphing
into a small httpd?

> BTW, it seems that using Apache with several worker processes (2-5) for
> bugzilla, mailman, etc. will not consume much CPU/memory: look top.

I'm just looking at it from simplifying the system administration.

I know it is not the most performant, those tools are not my preferred
ones, I just have to support hosting them right now...
Igor S. (Guest)
on 2009-02-07 13:14
(Received via mailing list)
On Sat, Feb 07, 2009 at 02:33:09AM -0800, mike wrote:

> > of overhead. Besides CGI programs will run with worker privilege only.
>
> and it requires the CGI program to be modified, yeah?

No.

> into a small httpd?
No, the special CGI manager is a nginx process that fork()ed by master
nginx
process (like worker processes). The communication protocol between
workers and the manager is not HTTP: a worker starts to handle a
request,
then it sees that the request should be handled by CGI, something like

      location /mailman/ {
           cgi_script  ...;
           cgi_user    ...;
           ...
      }

and pass the request and client socket to the manager.

> > BTW, it seems that using Apache with several worker processes (2-5) for
> > bugzilla, mailman, etc. will not consume much CPU/memory: look top.
>
> I'm just looking at it from simplifying the system administration.
>
> I know it is not the most performant, those tools are not my preferred
> ones, I just have to support hosting them right now...

I understand this, but currently nginx has more priority tasks.
mike (Guest)
on 2009-02-07 13:31
(Received via mailing list)
On Feb 7, 2009, at 2:57 AM, Igor S. <removed_email_address@domain.invalid> 
wrote:

>>> 1) simple one: just fork()ing worker process that has received a
>>> request
>>> for CGI and exec() a CGI program. It's simple enough, but has a lot
>>> of overhead. Besides CGI programs will run with worker privilege
>>> only.
>>
>> and it requires the CGI program to be modified, yeah?
>
> No.

Got any urls for examples then?


>>> will
> request,
>>> BTW, it seems that using Apache with several worker processes
>
Sure. So it isn't impossible if someone else was hired to do it. I was
wondering if there was a technical reason for it.

On a different note, I've paid someone to implement mod_auth_gss (more
or less) into nginx. So it might support kerberos tickets and single
sign on soon (at least SPNEGO)
Volodymyr K. (Guest)
on 2009-02-07 22:46
(Received via mailing list)
mike wrote:
> afaik, the only way to do that is to alter the CGI application to use
> FCGI.pm or whatever?

You wrong, it's just a point of using FastCGI wrapper. I have dozen
awstats installations running FastCGI wrapper from http://www.nginx.eu/
Manlio P. (Guest)
on 2009-02-07 22:48
(Received via mailing list)
Igor S. ha scritto:
> [...]
>
> nginx is not general purpose server, it's rather highload server.
> CGI is not compatible with highload: if you run Apache/CGI, then CGI
> will became bottleneck much earlier than Apache.
>

This is not a problem from the Nginx point of view.
Nginx should however put a limit on the number of concurrent CGI
requests.
It can use a queue to store outgoing request, and if the queue grows too
much return a 503 Service Unavailable, setting Retry-After to a
reasonable (computed?) value.

No browser support 503 response, however.

> There are two ways to implement CGI inside nginx:
>
> 1) simple one: just fork()ing worker process that has received a request
> for CGI and exec() a CGI program. It's simple enough, but has a lot
> of overhead. Besides CGI programs will run with worker privilege only.
>

Nginx could use seteuid, instead of setuid.
So that it can reacquire root privileges.

As for CGI support, some time ago I was trying to implement it.
The idea was to use unix domain sockets (socketpair), and reusing the
http_upstream module.

However I gave up, a lot of code for connection/upstream setup must be
rewritten, and I hate to write boiler plate code; and expecially having
to maintain it :).

In Nginx one can also easily close all current opened file descriptors,
in child process; Nginx keeps all opened connection in the
ngx_cycle->free_connections variable so one can write a closefrom
function.


Regards  Manlio P.
Grzegorz N. (Guest)
on 2009-02-08 00:01
(Received via mailing list)
On piÄ…, lut 06, 2009 at 02:55:25 -0800, mike wrote:
> I'd be willing to put some money down for it, too. (and mod_svn !)

Cool, how much will you pay for fcgiwrap then? :D

http://nginx.localdomain.pl/

or is there something fundamental missing from that? I'm running over a
hundred of those at the moment and they seem to do their job (talk
fastcgi on one end and cgi on the other), no cgi code modification
required,
pathinfo support, what's not to like? :)

Best regards,
 Grzegorz N.
Jim O. (Guest)
on 2009-02-08 05:03
(Received via mailing list)
I'm currently testing this with PHPmotion and it works perfectly using
spawn-fcgi to get it started. Now I need to update the code to get Nginx
upload progress bar in there as the one that comes with the scrip is
meant for (cr)apache.

I was wondering, Grzegorz, if there is some "algorithm" as to how to
determine the optimal number of children to run.

Jim
mike (Guest)
on 2009-02-08 11:32
(Received via mailing list)
I googled around originally for nginx + CGI stuff (specifically
bugzilla, mailman) and could not find anything.

This might work just fine! I will have to try it out. Looks like it is
just like php-fpm, but for CGI?
Igor S. (Guest)
on 2009-02-08 12:24
(Received via mailing list)
On Sat, Feb 07, 2009 at 09:38:41PM +0100, Manlio P. wrote:

> It can use a queue to store outgoing request, and if the queue grows too
> much return a 503 Service Unavailable, setting Retry-After to a
> reasonable (computed?) value.
>
> No browser support 503 response, however.

I mean that using CGI as main response handler on highload site is
useless.
CGI may be used for lowload tasks: monitoring, administration, etc.
those
can be easly handled by Apache or by proxying to Apache.

> >There are two ways to implement CGI inside nginx:
> >
> >1) simple one: just fork()ing worker process that has received a request
> >for CGI and exec() a CGI program. It's simple enough, but has a lot
> >of overhead. Besides CGI programs will run with worker privilege only.
> >
>
> Nginx could use seteuid, instead of setuid.
> So that it can reacquire root privileges.

Yes, but it's not so safe as setuid().

> As for CGI support, some time ago I was trying to implement it.
> The idea was to use unix domain sockets (socketpair), and reusing the
> http_upstream module.
>
> However I gave up, a lot of code for connection/upstream setup must be
> rewritten, and I hate to write boiler plate code; and expecially having
> to maintain it :).

Yes, CGI module should use modified upstream module.

> In Nginx one can also easily close all current opened file descriptors,
> in child process; Nginx keeps all opened connection in the
> ngx_cycle->free_connections variable so one can write a closefrom function.

This can be resolved using fcntl(F_SETFD, FD_CLOEXEC) too.
Igor S. (Guest)
on 2009-02-08 12:31
(Received via mailing list)
On Sat, Feb 07, 2009 at 03:21:23AM -0800, mike wrote:

> >>>There are two ways to implement CGI inside nginx:
>
> Got any urls for examples then?

No, as I know there is no attempts to implemnet this in nginx.

This CGI support is the same as in lighttpd/thttpd/etc: they have
single process that fork()/exec() CGI program.

> >>>( http://acme.com/software/mini_httpd/ ).
> >
> >>>bugzilla, mailman, etc. will not consume much CPU/memory: look top.
> Sure. So it isn't impossible if someone else was hired to do it. I was
> wondering if there was a technical reason for it.

No, there is no technical reason that does not allow to implement CGI
in nginx, this is wasting time only (from my point of view :).
Grzegorz N. (Guest)
on 2009-02-08 15:10
(Received via mailing list)
On Sat, Feb 07, 2009 at 09:50:47PM -0500, Jim O. wrote:
> I'm currently testing this with PHPmotion and it works perfectly using spawn-fcgi to get 
it started. Now I need to update the code to get Nginx upload progress bar in there as the 
one that comes with the scrip is meant for (cr)apache.
>
> I was wondering, Grzegorz, if there is some "algorithm" as to how to determine the 
optimal number of children to run.

Basically, the number of concurrent CGI requests you wish to serve and
then some (a single fcgiwrap does not multiplex multiple requests yet,
sorry). It's actually better to have a few slack ones (they are really
_very_ cheap, especially after the first one -- double-digit
_kilo_bytes).

Best regards,
 Grzegorz N.
Grzegorz N. (Guest)
on 2009-02-08 15:13
(Received via mailing list)
On Sun, Feb 08, 2009 at 01:19:12AM -0800, mike wrote:
> I googled around originally for nginx + CGI stuff (specifically
> bugzilla, mailman) and could not find anything.

I've been spamming this list with fcgiwrap ever since I wrote it many
months ago, you must've been asleep ;) It's even been on the wiki for
some time now.

> This might work just fine! I will have to try it out. Looks like it is
> just like php-fpm, but for CGI?

Roughly. It still needs a process manager of some sort. Have a look at
my website, there's a very simple launcher script there that you might
find suitable.

Best regards,
 Grzegorz N.
Jim O. (Guest)
on 2009-02-08 19:02
(Received via mailing list)
OK, I'm sure that I'm missing something, but I'm having trouble getting
more than one instance going.

I tried

# /usr/local/bin/spawn-fcgi -C 5 -a 127.0.0.1 -p 10000 -u nobody -g
nobody -f /usr/local/bin/fcgiwrap but only saw one instance generated.

Jim

Do you have a sample script to start it with "X" number of children
Grzegorz N. (Guest)
on 2009-02-08 21:16
(Received via mailing list)
On Sun, Feb 08, 2009 at 11:48:32AM -0500, Jim O. wrote:
> OK, I'm sure that I'm missing something, but I'm having trouble getting more than one 
instance going.
>
> I tried
>
> # /usr/local/bin/spawn-fcgi -C 5 -a 127.0.0.1 -p 10000 -u nobody -g nobody -f 
/usr/local/bin/fcgiwrap but only saw one instance generated.

-C only sets an environment variable for php-fcgi, it doesn't actually
run multiple instances of anything.

> Do you have a sample script to start it with "X" number of children

See http://nginx.localdomain.pl/ for a simple Perl launcher.

Best regards,
 Grzegorz N.
Andrius Semionovas (Guest)
on 2009-02-08 21:18
(Received via mailing list)
this simple script not worked in my server

On Sun, 08 Feb 2009 20:33:40 +0200, Grzegorz N.
Jim O. (Guest)
on 2009-02-08 21:19
(Received via mailing list)
I tried that also and while it starts multiple instances, if I run them
using user "nobody" I get a 502 Bad Gateway response from Nginx.

This is how I edited the script:


#!/usr/bin/perl

use strict;
use warnings FATAL => qw( all );

use IO::Socket::UNIX;

my $bin_path = '/usr/local/bin/fcgiwrap';
my $socket_path = $ARGV[0] || '/tmp/cgi.sock';
my $num_children = $ARGV[1] || 5;

close STDIN;

unlink $socket_path;
my $socket = IO::Socket::UNIX->new(
    Local => $socket_path,
    Listen => 10000,
);

die "Cannot create socket at $socket_path: $!\n" unless $socket;

for (1 .. $num_children) {
    my $pid = fork;
    die "Cannot fork: $!" unless defined $pid;
    next if $pid;

    exec "sudo -u nobody $bin_path";
    die "Failed to exec


Jim
Grzegorz N. (Guest)
on 2009-02-09 02:44
(Received via mailing list)
On Sun, Feb 08, 2009 at 09:09:44PM +0200, Andrius Semionovas wrote:
> this simple script not worked in my server

Works for me (or at least it did when I last checked). What isn't
working for you? Please provide some details because a "it doesn't work"
report definitely won't get you a solution (my crystal ball had to be
RMA'd).

Best regards,
 Grzegorz N.
Grzegorz N. (Guest)
on 2009-02-09 02:46
(Received via mailing list)
On Sun, Feb 08, 2009 at 02:13:08PM -0500, Jim O. wrote:
> I tried that also and while it starts multiple instances, if I run them using user 
"nobody" I get a 502 Bad Gateway response from Nginx.

Did you see fcgiwrap instances start? On the socket Nginx connects to?
Did you try to collect a strace run from Nginx and/or fcgiwrap?
If you're still stuck, shoot me an e-mail.

Best regards,
 Grzegorz N.
Andrius Semionovas (Guest)
on 2009-02-09 15:10
(Received via mailing list)
I not remember. But i think, my centos 4.6 miss some libraries or
something else

On Mon, 09 Feb 2009 02:30:27 +0200, Grzegorz N.
This topic is locked and can not be replied to.