The highlight of this release is the dynamic module loading support
(Dynamic Module Loading Support (DSO) - The Tengine Web Server). We now can load modules
upon demand instead of having to recompile nginx with the individual
modules. It is very convenient for module developers and corporations.
And as this release, Tengine is based on Nginx-1.2.3.
Complete changelog is as follows:
*) Feature: added the dynamic module loading support (a.k.a. DSO),
so we don’t have to recompile tengine when we want to add a new
module. (monadbobo)
*) Feature: updated the Lua module to the latest stable version.
(chaoslawful, agentzh, jinglong)
*) Feature: added json and csv format output for the
upstream_check module. (yaoweibin)
*) Feature: added the ‘log_empty_request’ directive which could be
used to turn off logs from a connection without HTTP data.
(zhuzhaoyuan)
*) Feature: added the ‘concat_delimiter’ directive to the concat
module to allow adding delimiter between each file. (dinic)
*) Feature: added the ‘concat_ignore_file_error’ directive to
ignore file errors and the syntax of concat is less strict now.
(dinic)
*) Feature: added the ‘default’ option to the ‘error_page’
directive to set all error pages to default values. (jinglong)
*) Feature: added the ‘priority’ directive of the procs module.
(yzprofile)
*) Feature: added the ‘delay_start’ directive of the procs module.
(yzprofile)
*) Change: merged changes from nginx-1.2.3. (zhuzhaoyuan)
*) Bugfix: fixed a segmentation fault bug of the geo module when
‘range’ was set without default value. (yzprofile)
*) Bugfix: fixed a segmentation fault bug with the procs module.
(yzprofile)
*) Bugfix: fixed a socket leak bug when upstream_check was
enabled. (yaoweibin)
*) Bugfix: fixed some bugs of the limit_req module. (monadbobo)
*) Bugfix: fixed wrong format types with error logs. (yaoweibin)
*) Bugfix: fixed a compile error if the perl module was used with
procs. (yzprofile)
For those who don’t know Tengine, it is a free and open source
distribution of Nginx with some advanced features. See our website for
more details: http://tengine.taobao.org
Have fun!
Regards,
–
Joshua Z.
Senior Software Engineer
Server Platforms Team at Taobao
upon demand instead of having to recompile nginx with the individual
*) Feature: added the ‘log_empty_request’ directive which could be
(yzprofile)
*) Bugfix: fixed wrong format types with error logs. (yaoweibin)
*) Bugfix: fixed a compile error if the perl module was used with
procs. (yzprofile)
For those who don’t know Tengine, it is a free and open source
distribution of Nginx with some advanced features. See our website for
more details: http://tengine.taobao.org
Have fun!
Very nice! If you have a benchmark comparing performance with compiled
modules vs DSO, please do share.
I’m expecting the speed difference to be too small to notice.
On Thu, Sep 6, 2012 at 12:01 PM, Christopher M. [email protected]
wrote:
Does nginx support DSO now?
Not yet, as far as I know. And that’s why we developed this feature
(Dynamic Module Loading) by ourselves.
BTW, we’ll pick out the patches against standard Nginx and post them
on this mailing list later.
Regards,
–
Joshua Z.
Senior Software Engineer
Server Platforms Team at Taobao
IMO,nginx should apply this patch therefore people can load or unload
modules like apache.
My servers use epel to install nginx, however this may be difficult for
users to compile and use a 3rd party module which they want,so DSO
REALLY
solve my problem.
And I’ve taken a look at nginx mailinglist,it seems the upstream doesn’t
want to patch this feature into master branch.Right?
if so, that’s a pity…
–
Yours sincerely, Christopher M.
Ambassador/Contributor of Fedora Project and many others.
Very nice! If you have a benchmark comparing performance with compiled
modules vs DSO, please do share.
I’m expecting the speed difference to be too small to notice.
Yeah we have compared the performance though I have no formal report
on hand. The performance difference was too trivial to notice.
Regards,
–
Joshua Z.
Senior Software Engineer
Server Platforms Team at Taobao
I would love to see and even pledge a bounty (if Igor, Maxim and team even
need any money anymore! :p) to get DSO into nginx mainline…
Sorry to hijack Tengine release thread but whats the gain/advantages of
having dynamic modules (potentially having a performance loss) in nginx
versus a single executable?
On a moderate system it takes like 3-5 seconds to recompile full source
and
seamlessly rotate to the new version.
the basic concept of “modules” are pluggable instruments to extend
something else.
having to compile it in makes it less modular. DSO support would make
package management in distributions seamless.
minor differences in any one single module means you need to recompile
the entire binary for that small change instead of a single .so for
example being changed and the server gracefully reloading.
But it’s useful of us, we are serving more than 500 applications in
our company and we use redhat RPM package system to manager our
servers. Different applications need different nginx modules. We have
more than dozens of different nginx rpm packages for the applications.
If one module appears a bug, we have to upgrade all the packages and
production servers. It’s not easy task every time for us.
I knew the ABI reasoning was behind it as this has came up at least once
before. Just wanted to express it again. Curious how tengine does it
with very small performance loss still based off the nginx code base.
Perhaps something to look at? that’s all.
On Thu, Sep 06, 2012 at 12:10:46AM -0700, Michael S. wrote:
the basic concept of “modules” are pluggable instruments to extend
something else.
having to compile it in makes it less modular. DSO support would make
package management in distributions seamless.
minor differences in any one single module means you need to recompile
the entire binary for that small change instead of a single .so for
example being changed and the server gracefully reloading.
The main problem with dynamically loaded modules is ABI, and
that’s why we don’t support it now. With even minor difference in
nginx core you’ll need to recompile all modules - else you’ll see
unexpected behaviour here and there due to ABI changes.
Moreover, right now we don’t maintain ABI even within single
source code base - various structure layout may change depending
on various configure options and test results. That is, if one
needs to compile loadable modules - the only valid option now is
to compile them with nginx itself, else expect problems.
With this in mind it doesn’t really make sense to support dynamic
modules. Especially if you also consider resulting performance
loss (and nginx’s seamless upgrade process as mentioned by Reinis).
So the question isn’t really about dynamic modules by itself.
Question is about (at least tracking of) ABI provided for modules.
For now it’s something mostly nonexistent.
Not even talking about resulting problems due to use of other
libraries n loaded modules. I’ve seen Apache segfaulting due to
incompatible libraries used in modules more than once, and I don’t
really want to see the same with nginx.
On Fri, Sep 7, 2012 at 12:27 AM, Michael S. [email protected]
wrote:
Maxim,
I knew the ABI reasoning was behind it as this has came up at least once before.
Just wanted to express it again. Curious how tengine does it with very small
performance loss still based off the nginx code base. Perhaps something to look
at? that’s all.
We already know the limitations pointed out by Maxim and thank Maxim
for making it clear several times. Definitely it’s difficult to solve
them, especially the ABI problem. The approach we’ll take is keeping
the ABI of the stable versions as conservative as we can, i.e. we’ll
not change the ABI of a stable branch. And the ABI changes will just
happen in a development branch.
In such a way it may reduce the value of dynamically loaded modules,
but it’s better than nothing.
Regards,
–
Joshua Z.
Senior Software Engineer
Server Platforms Team at Taobao
The question isn’t really about “keeping ABI in the stable
branch”, checking version signature is more or less enough here
(unless you apply patches which change the ABI within the same
version).
It’s more about what happens once you’ll try to load a module
compiled with --with-ipv6 into nginx compiled without this flag
(or vice versa). Layout of the ngx_connection_t structure will be
different in nginx itself and the module, and the resulting code
will happily do various unexpected thing.
Thank you very much for your valuable comments
I think there should be solutions, though it’s difficult to fix the
conditional compilation problem. For example, we can use the same
configure options (–with-ipv6, and etc) when compiling a dynamically
loaded module. And nginx must be recompiled once such a configure
option is changed. For a module which introduces conditional
compilations, we can make it compiled in forcibly if the DSO feature
is enabled. Plus, IMHO, some conditional compilations are not
necessary hence could be removed.
And the same thing may even happen without any changes in
configure arguments - e.g. if OS was upgraded and nginx
now detects some new feature it want to use.
It’s not a problem caused by dynamically loaded modules. It exists
even in statically compiled nginx. The accept4() problem (yeah it has
been fixed) is an example
Regards,
–
Joshua Z.
Senior Software Engineer
Server Platforms Team at Taobao
On Tue, Sep 11, 2012 at 12:16:55PM +0800, Joshua Z. wrote:
[…]
And the same thing may even happen without any changes in
configure arguments - e.g. if OS was upgraded and nginx
now detects some new feature it want to use.
It’s not a problem caused by dynamically loaded modules. It exists
even in statically compiled nginx. The accept4() problem (yeah it has
been fixed) is an example
Problems with nonexistent syscalls is completely different thing:
By upgrading hosts in correct order (i.e. build host last) you
just don’t see any problems.
If there is no syscall at all on the host you are running,
you’ll see ENOSYS returned and it’s up to your code to handle it
(either just fail or fallback to something else).
This is because ABI is actually provided by OS here. And
it’s completely different from “no ABI” case where you’ll just get
random unexpected behaviour and segfaults if you are lucky or
just corrupted data if you aren’t.
On Fri, Sep 07, 2012 at 10:07:08AM +0800, Joshua Z. wrote:
them, especially the ABI problem. The approach we’ll take is keeping
the ABI of the stable versions as conservative as we can, i.e. we’ll
not change the ABI of a stable branch. And the ABI changes will just
happen in a development branch.
In such a way it may reduce the value of dynamically loaded modules,
but it’s better than nothing.
The question isn’t really about “keeping ABI in the stable
branch”, checking version signature is more or less enough here
(unless you apply patches which change the ABI within the same
version).
It’s more about what happens once you’ll try to load a module
compiled with --with-ipv6 into nginx compiled without this flag
(or vice versa). Layout of the ngx_connection_t structure will be
different in nginx itself and the module, and the resulting code
will happily do various unexpected thing.
And the same thing may even happen without any changes in
configure arguments - e.g. if OS was upgraded and nginx
now detects some new feature it want to use.
Maxim D.
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
that’s all.
that’s all.