FCGI.pm?

Server NGINX
21

On wto, mar 03, 2009 at 08:09:12 -0800, Roger H. wrote:

For dynamically spawned processes, if someone were willing to write an nginx
module to call supervisord’s XML-RPC api, we’d have a complete,
language-agnostic replacement solution for mod_fastcgi.

XMLRPC sounds evil :wink: My checklist for required features looks like
below, how far is supervisord from meeting that?

  • killing idle processes after some timeout (with a configurable
    signal, preferably)
  • static pool size management (keep 5 of those running at all times)
  • dynamic pool size management (keep 1-5 running depending on load;
    this will require congestion notifications from the web server, like
    you said)
  • process status notifications (“foo failed to run, what now?”, with
    some mildly intelligent retry logic) – I need to alert an external
    entity somehow.
  • easily generated configuration file

If supervisord can do that, I’d be willing to write an Nginx module to
interface with that, sometime in this millenium.

Best regards,
Grzegorz N.

22

Why not just make a php-fpm style thing like we talked about?

Define pools of fastcgi resources, with various configuration options,
timeout thresholds, error logging/capturing perhaps, ability to run as
different uid/gid per pool, adaptive process spawning (hard limits
etc.) …

On Wed, Mar 4, 2009 at 12:40 AM, Grzegorz N.

23

Supervisord can do all this and more. It’s a full service process
manager
that can manage daemons in any language, including pools of FastCGI
processes.

http://supervisord.org/

It has an XML-RPC interface and cmd line tool (supervisorctl) so you can
send commands to the process manager as it’s running (for example,
starting
and stopping a given process or process pool). It even has a
language-agnostic event listening API so that you can monitor process
manager events and take your own custom actions (such as sending
yourselves
alerts if a process dies).

Here’s a sample config for managing a pool of 5 foo.pl processes and 2
bar.pl processes.

[fcgi-program:foo]
socket=tcp://127.0.0.1:4000
process_name = %(program_name)s_%(process_num)s
command = /path/to/foo.pl
numprocs = 5

[fcgi-program:bar]
socket=unix:///path/to/fcgi/socket
process_name = %(program_name)s_%(process_num)s
command = /path/to/bar.pl
numprocs = 2

The FCGI spawning has not been officially released but it’s well unit
tested
in the SVN repo. Just run easy_install

24

On śro, mar 04, 2009 at 12:55:01 -0800, mike wrote:

Why not just make a php-fpm style thing like we talked about?

Define pools of fastcgi resources, with various configuration options,
timeout thresholds, error logging/capturing perhaps, ability to run as
different uid/gid per pool, adaptive process spawning (hard limits
etc.) …

[disclaimer: I haven’t worked with supervisord yet, so I’m possibly
wrong about its capabilities]

Well, this is the foundation, which AIUI supervisord provides. However,
it has no possibility to know how busy the managed processes are and
when to spawn/kill (again, php-fpm has it too easy, it’s not
implementable
this way in a generic process manager). Thus it needs some notification
from Nginx when it’s overloaded (and when backends are idle – but this
is a harder task).

So, to recap: supervisord alone allows you to run a statically-sized
pool of processes (with some nifty features like you mentioned above).

If you need dynamic pool sizing, you need some external help, either
from
Nginx, or possibly the kernel. What we’re discussing here is not an
alternative to a dynamic process pool, it’s one of the few sensible ways
to implement it in a language-agnostic way.

(BTW, I wonder whether supervisord can interface with PAM)

Best regards,
Grzegorz N.

25

i am all for someone taking Grzegorz’s work, and instead using the
perl launcher or spawn-fcgi, maybe add a bit more intelligence behind
it and make it simple for launching pools under different uid/gid with
different child counts, etc.

On Wed, Mar 4, 2009 at 1:50 AM, Jean-Philippe M.

26

Grzegorz N. a écrit :

  • dynamic pool size management (keep 1-5 running depending on load;
    this will require congestion notifications from the web server, like
    you said)
  • process status notifications (“foo failed to run, what now?”, with
    some mildly intelligent retry logic) – I need to alert an external
    entity somehow.
  • easily generated configuration file

I don’t use it but God (http://god.rubyforge.org/) seems to respond to
your needs.

27

On Wed, Mar 4, 2009 at 12:40 AM, Grzegorz N.
[email protected]wrote:

On wto, mar 03, 2009 at 08:09:12 -0800, Roger H. wrote:

For dynamically spawned processes, if someone were willing to write an
nginx
module to call supervisord’s XML-RPC api, we’d have a complete,
language-agnostic replacement solution for mod_fastcgi.

XMLRPC sounds evil :wink: My checklist for required features looks like
below, how far is supervisord from meeting that?

I’m not a huge XMLRPC fan either but Python makes it easy for the
project
implementers. I suppose some other data format such as JSON is possible
if
there’s a will.

  • killing idle processes after some timeout (with a configurable
    signal, preferably)

I think it’s difficult to kill individual processes that have timed out
when
a pool of processes are all listening on the same FCGI socket because
nginx
would not know which process had accepted the request that timed out
(unless
there is some mechanism in the FCGI protocol to do this). However, the
entire process pool could be killed/restarted if that’s the behavior you
desire.

  • static pool size management (keep 5 of those running at all times)

This is already available in supervisord

  • dynamic pool size management (keep 1-5 running depending on load;
    this will require congestion notifications from the web server, like
    you said)

Functionality was recently added to supervisord to modify it’s
configuration
dynamically through the XML-RPC api so this is matter of implementing
the
load logic in an nginx plugin and making calls to supervisord to add and
subtract from the pool.

  • process status notifications (“foo failed to run, what now?”, with
    some mildly intelligent retry logic) – I need to alert an external
    entity somehow.

Depends on what kind of failure. If process foo exits with an
unexpected
exit code, supervisord has an event listening API (a simple text
protocol
listening on a pipe) that allows you to write an event handler in any
language to deal with those events. However, if process foo fails in
the
sense of returning a 500 status code or not returning any data at all,
that
would have to be handled by nginx module.

  • easily generated configuration file

I think the config is pretty simple but I’ll let you be the judge (
http://supervisord.org/manual/current/configuration.html). I generate
supervisord configs dynamically in my current company using Perl
Template
Toolkit.

28

On Wed, Mar 4, 2009 at 9:03 AM, Roger H. [email protected]
wrote:

 - dynamic pool size management (keep 1-5 running depending on load;
 this will require congestion notifications from the web server, like
 you said)

Functionality was recently added to supervisord to modify it’s configuration
dynamically through the XML-RPC api so this is matter of implementing the
load logic in an nginx plugin and making calls to supervisord to add and
subtract from the pool.

While I would like to keep my software stack low, this sounds like a
neat benefit. Would just need to define hard upper limits, and how
long to wait or whatever to kill spare/unused children (like apache, I
suppose)

Personally I would like to see a daemon that does this in itself.
Leverages the fcgiwrap code + adds on features. I suppose it would
have to be ‘aware’ of how many connections it was servicing per pool
which Grzegorz makes it sound like can be very hard… but then it
could manage things dynamically.

request comes in → depending on what port/socket/etc. it checks the
pool, determines if any children are open (if more needed, spawn like
apache, maybe log a notice in the log), changes to proper uid/gid if
configured, then executes the fastcgi stuff, if it gets back an error,
determine whether or not to log it, pass it back with the same http
code, do both, etc…

etc.

I don’t understand enough about sockets, C, threading/forking/event
models/etc. to see if that is even an option but it seems like it
could be done, just not sure if it would be way too slow or not?

29

Precisely. With regard to PAM, I think the answer is no.
Authentication/Authorization for the XMLRPC API are done using
configured
username/password. The risk is somewhat limited by the fact that it can
use
a unix domain socket so that at least exposure is limited to the
machine.

30

On Wed, Mar 4, 2009 at 11:51 AM, mike [email protected] wrote:

load logic in an nginx plugin and making calls to supervisord to add and
which Grzegorz makes it sound like can be very hard… but then it
could manage things dynamically.

request comes in → depending on what port/socket/etc. it checks the
pool, determines if any children are open (if more needed, spawn like
apache, maybe log a notice in the log), changes to proper uid/gid if
configured, then executes the fastcgi stuff, if it gets back an error,
determine whether or not to log it, pass it back with the same http
code, do both, etc…

etc.

The approach you describe assumes that the parent process can intercept
socket connections as they come in. I don’t think this is possible
within
the constraints of the FastCGI spec. Each FastCGI process is forked
with
file descriptor 0 pointing to a shared FastCGI socket and each child
process
just calls accept() on that socket. The OS is responsible to
determining
which process in the pool accepts each request so there’s no way for the
parent process to keep track of which child is taking which request.
Unless
that information can be retrieved from the kernel, I think the only
place
that load logic can be implemented is in an nginx module.

31

I am going to start a cgi-fpm project. The goals will be aligned like
php-fpm except slightly modified for the differences with cgi and
fastcgi. It might not be able to support adaptive spawning without
some sort of api or tools but at least will make management easier and
not require third party tools. It will basically enhance fcgiwrap with
php-fpm style configuration and hooks for external control for things
like an nginx module.

The annoyances with cgi and fastcgi can be discussed and hopefully
addressed.

Thoughts?

Also this could just be an nginx module too. But it would add some
weight and require suexec type stuff. So probably not a good idea.

Let me throw together a quick list of ideas.

On Mar 4, 2009, at 12:34 PM, Roger H. [email protected]

32

This is an interesting idea. I’d like to see a generic process manager
come
out of the effort though, not just one that only works for wrapping CGI.
It
would be a great separation of concerns. The CGI wrapper could focus
on a
single task: accepting FastCGI requests and forking CGI processes to
handle
them. Process pool management could be handled by a generic FastCGI
process
manager, which could manage fcgiwrap pools and any other type of FastCGI
processes.

33

Sure - you mean anything that nginx (or any other server) speaks fastcgi
to

webserver <-> this process manager <-> code (perl cgi, or anything else)
?

Wouldn’t this also be able to replace php-fpm then?

It seems like there are already ways using Tomcat/etc. for java,
php-fpm for PHP, ruby and python have managers, but CGI does not. Can
you give an example of what other things you’d want this to manage?

34

Hmm. Well

On Mar 4, 2009, at 12:34 PM, Roger H. [email protected]

35

I guess I would be more than happy then to stop using php-fpm and use
this new tool.

However I do like the benefits of php-fpm like the ability to override
some php.ini directives.

Perhaps we should get Grzegorz (fcgiwrap), Andrei (php-fpm) and others
together for this.

A generic fastcgi to (php|cgi|ruby|python|whatever) interface ?

36

On Thu, Mar 5, 2009 at 9:56 AM, mike [email protected] wrote:

Sure - you mean anything that nginx (or any other server) speaks fastcgi to

Yes!

webserver ↔ this process manager ↔ code (perl cgi, or anything else) ?

Wouldn’t this also be able to replace php-fpm then?

Ideally, yes. Even with PHP’s unique lifecycle (all objects discarded
after
each request), I don’t see a reason why process pool management needs to
be
built in to PHP FastCGI handling. Each PHP process could manage the
request
lifecycle for itself while the process pool management could be handled
generically, just like any other language.

It seems like there are already ways using Tomcat/etc. for java,
php-fpm for PHP, ruby and python have managers, but CGI does not. Can
you give an example of what other things you’d want this to manage?

In short, I’d be happy if it exactly matched the process management
features
of mod_fastcgi for Apache. mod_fastcgi does both FastCGI proxying and
process management. Nginx + other event driven webservers implement the
FastCGI proxying piece but not the process management piece.

You’re right that each language seems to have a different way of
managing
processes. This is appealing when all the infrastructure you run is in
a
single language. If you run ruby, you do it “the ruby way”. However,
the
organizations I’ve worked for need to run code in different languages,
for
various reasons including aquisition or migration. I think everyone
would
benefit from a language agnostic process manage similar to mod_fastcgi
but
not tied to Apache.

37

On Thu, Mar 5, 2009 at 4:15 PM, mike [email protected] wrote:

I guess I would be more than happy then to stop using php-fpm and use
this new tool.

However I do like the benefits of php-fpm like the ability to override
some php.ini directives.

This seems like something that still appropriate for PHP’s FCGI wrapper,
even if that wrapper no longer needs to manage pools of processes.

Perhaps we should get Grzegorz (fcgiwrap), Andrei (php-fpm) and others
together for this.

I’d love to see if there’s interest in this idea.

A generic fastcgi to (php|cgi|ruby|python|whatever) interface ?

A language-agnostic FastCGI process manager? One process manager to
rule
them all and in darkness bind them. :slight_smile:

38

On Fri, Mar 6, 2009 at 5:22 PM, Roger H. [email protected]
wrote:

This seems like something that still appropriate for PHP’s FCGI wrapper,
even if that wrapper no longer needs to manage pools of processes.

I want management of pools so I can separate by uid/gid, limits of
children, and if applicable adaptive spawning limits

I believe php-fpm uses libevent, there’s gotta be a way to make a
middleware style daemon for this that leverages a large amount of
php-fpm style concepts and even daemonizing code…

Perhaps we should get Grzegorz (fcgiwrap), Andrei (php-fpm) and others
together for this.

I’d love to see if there’s interest in this idea.