RFC: Feasibility of a "dynamic module loader" built in to nginx?

Server NGINX
1

It seems (at least to me) that the past year has seen explosive growth
in nginx usage and third party module development.

I believe that for package management, as well as module adoption, it
would be a definite move in the right direction to be able to load
modules up on demand instead of having to custom compile nginx with
the modules you want.

That allows for module development to be done completely independently
of nginx, modules to be added/removed, etc, without needing to
recompile nginx every time.

I find that the majority of people would most likely say this is a
good idea, a few may think it is not necessary, but everyone can
benefit, and for those who don’t, they can still maintain their nginx
however they want, by recompiling each time.

I don’t expect much overhead being added, as the modules would still
only be loaded when the master process (or perhaps child processes?)
startup.

Ideally it would be as simple as

load /usr/lib/nginx/mod_drizzle.so;

Or something in the configuration. This would allow plugins to
flourish without complex build instructions or having to touch the
nginx core; module developers could be informed to make version checks
and issue a notice when trying to startup, something along the lines
of “mod_drizzle is not compatible with this version of nginx (=>
0.8.43 needed)” or something of that nature.

I do not know if nginx could support this or not, it seems like it
should be able to, if it is able to accept modules at compile time,
just some more magic could be added to dynamically load something else
instead.

I’m sure some people would offer to help sponsor this if needed…

As usual, this is open for discussion, I’d really like to hear
Igor/Maxim’s take, or what the FRiCKLE guys or agentzh would have to
say about how useful this would be or how easy they think it could be
accomplished.

Personally I have not played with many third party modules; I have
passenger being built as part of my build script but I only use it on
one machine, all the other ones issue an annoying notice (that I’ve
been trying to get removed) - but being able to just install a module
and then enable it quickly to play with it, that would allow me to try
out all these third party modules out there.

The future seems to be moving to ultimate modularity. Having to
recompile the core each time for a third party module seems archaic
still. This should allow for module developers to focus a lot more on
purely releasing updates to modules, and not instructions on compiling
core with them, etc.

Thanks,
mike

2

Hello!

On Thu, Aug 26, 2010 at 04:14:10AM -0700, Michael S. wrote:

recompile nginx every time.
[…]

This was already discussed several times. Just to name a few
reasons why no:

  1. Using dynamically linked libraries implies speed penalty.

  2. It is likely to cause problems like DLL hell. Even with small
    number of carefully selected dynamic libraries nginx currently use
    (e.g. openssl) - there are periodically reported problems.

  3. It will require to maintain ABI, not something we currently do.
    E.g. right now ngx_http_request_s structure changes depending on:
    whether pcre library used, http cache enabled, gzip support
    enabled, stub status compiled in. Not even talking about local
    patches and changes between versions. So things is much more
    complex than “0.8.43 needed”.

  4. Most of us are happy enough with static linkage. Especially
    keeping in mind that binary may be upgraded with zero downtime.

Maxim D.

3

On Thu, Aug 26, 2010 at 7:14 PM, Michael S. [email protected]
wrote:

I believe that for package management, as well as module adoption, it
would be a definite move in the right direction to be able to load
modules up on demand instead of having to custom compile nginx with
the modules you want.

I totally agree here :slight_smile: We wrote 15 nginx C modules in the last year
(just one or two are opensourced yet) and we found rpm packaging
especially hard in a corporation environment. For every different kind
of business, we have to provide a variation of nginx rpm package just
because they use a different set of modules. (Yeah, we could compile
all those modules into a giant nginx but it hurts performance.)

That allows for module development to be done completely independently
of nginx, modules to be added/removed, etc, without needing to
recompile nginx every time.

I personally spent a lot of CPU cycles on recompiling nginx on my
laptop during nginx development. Fortunately I have a fast machine,
but still it’s far from ideal :slight_smile:

I find that the majority of people would most likely say this is a
good idea, a few may think it is not necessary, but everyone can
benefit, and for those who don’t, they can still maintain their nginx
however they want, by recompiling each time.

Having more choices is always a Good Thing ™ :wink:

I don’t expect much overhead being added, as the modules would still
only be loaded when the master process (or perhaps child processes?)
startup.

chaoslawful and I have talked about the implementation details several
times and dynamic loader support for the nginx core is one of our TODO
items. And yeah, it also means that it’s very likely that we’ll
maintain our own patches to the core or even our own fork of nginx
(given the historic records in the nginx world) :slight_smile:

Ideally it would be as simple as

load /usr/lib/nginx/mod_drizzle.so;

Cool, and by that way, we can re-arrange the order of nginx modules
just in the config file! Like so:

load /path/to/ngx_rds_json.so
load /path/to/ngx_srcache.so
load /path/to/ngx_lua.so

or make it even more elegant and more portable for other OS’s:

use ngx_rds_json;
use ngx_srcache;
use ngx_lua;

And we can control the search paths for the module .so files from the
environment. Quite perl-ish, isn’t it? :wink:

Or something in the configuration. This would allow plugins to
flourish without complex build instructions or having to touch the
nginx core; module developers could be informed to make version checks
and issue a notice when trying to startup, something along the lines
of “mod_drizzle is not compatible with this version of nginx (=>
0.8.43 needed)” or something of that nature.

nod

There we go!

Cheers,
-agentzh

4

On Fri, Aug 27, 2010 at 2:03 PM, agentzh [email protected] wrote:

I totally agree here :slight_smile: We wrote 15 nginx C modules in the last year
(just one or two are opensourced yet)

Sorry, typo here. It should have been “just one or two are not
opensourced yet” :stuck_out_tongue:

Cheers,
-agentzh

5

Hi,
I was thinking about this about two weeks, but (as Maxim already pointed
out) currently there is no ABI, which means that you would need to build
modules against your target nginx version on your target operating
system.

This renders this idea viable only for 3 use cases that I can think of
right
now:

  1. operating system’s pre-compiled binary packages,
  2. corporate environments / clusters with staging server(s),
  3. closed-source modules.

Having this probably wouldn’t hurt, but it won’t be useful for as many
people as you would hope to.

Best regards,
Piotr S. < [email protected] >

6

Hey, I just found you via Google as I was trying to find out why …
Nginx
is not supporting dynamic loading of modules, for the very obvious
issues
the other posters seem to have too.

However, Maxim, you claim, that one reason is performance. While
performance
of course matters, I’d like to know why you say so.
Can you please go a little bit more into the technical details why a
dynamic
loaded module does not perform as well as a statically linked “module”?

Of course, a compiler could use global optimization techniques to
perform
interprocedual optimizations, but I don’t believe (yet), that the impact
shall be that high.
2nd) statically linked libraries speed up process bootup, but this is
neglect-able for a long-running process.
3rd) of course, at least for the core modules, they could use #ifdef’s
inside the request structs and friends (just as you stated) in order to
further optimize resource (ie. memory) usage.

So, I’m only interested in the performance arguments (not zero-downtime
upgrade,…), so I can understand a little better.

Many thanks in advance,
Christian.

Posted at Nginx Forum:

7

Hello!

On Tue, Apr 15, 2014 at 03:52:04AM -0400, trapni wrote:

interprocedual optimizations, but I don’t believe (yet), that the impact
shall be that high.
2nd) statically linked libraries speed up process bootup, but this is
neglect-able for a long-running process.
3rd) of course, at least for the core modules, they could use #ifdef’s
inside the request structs and friends (just as you stated) in order to
further optimize resource (ie. memory) usage.

So, I’m only interested in the performance arguments (not zero-downtime
upgrade,…), so I can understand a little better.

In addition to the above, calling functions from dynamically
linked libraries implies additional indirection, hence it’s
expected to be slower.

I don’t think that speed difference is a major problem though,
most likely it will be small. It’s just one of the reasons in the
list.

–
Maxim D.
http://nginx.org/

8

Have you looked @ Dynamic Module Loading Support (DSO) - The Tengine Web Server

Posted at Nginx Forum:

9

On Tuesday 15 April 2014 14:37:17 nxspeed wrote:

Have you looked @ Dynamic Module Loading Support (DSO) - The Tengine Web Server

http://mailman.nginx.org/pipermail/nginx/2012-September/035405.html

wbr, Valentin V. Bartenev

10

Hi,

What might work just as well, and wouldn’t suffer the API / performance
issues would be making it easier to compile optional modules into the
source without recompiling the whole Nginx source code.

When I’m doing module development, I just build the object code of the
module I’m writing, and combined it with the pre-compiled object files
of all the other parts of Nginx, thereby typically taking less than a
second to compile (I’m assuming most developers do this too).

It would be nice to be able to have an alternative build script that
makes it easier to do this in a customizable way, allowing the reuse of
existing object code if it’s available, compiling it if it’s not and
forcing re-compiling if so desired.

It could be set up to accept the normal configuration options, then
automatically work out which object files needed to be re-generated and
which ones could use the pre-compiled versions, and could be extended to
include options and 3rd party modules too.

It wouldn’t be particularly difficult to do this, but I’m really busy
with other stuff at the moment. If someone wants to write such a script,
I’ll add it to the ngx_devel_kit
(GitHub - simpl-it/ngx_devel_kit). Otherwise I’ll write it at
some point later on, but I can’t give any timelines as of yet.

Cheers,

Marcus.