Ngx_drizzle: make nginx talk directly to mysql, drizzle, and sqlite3 by libdrizzle

Hi, guys!

This is the last day in 2009 and I’m too impatient to hold the first
release (version v0.0.1) of our ngx_drizzle module, an upstream module
that talks directly to RDBMS backends like mysql, drizzle, and the
drizzle server shipped with libdrizzle for sqlite3.

This module is initially started by my friend and colleague,
chaoslawful. His Chinese name is ÍõÏþÕÜ :slight_smile: He did the initial (most
difficult) work all in his own time :slight_smile: I also shamelessly borrowed a
lot of code from Igor’s ngx_http_upstream.c and
ngx_http_memcached_module.c in the nginx 0.8.30 core, as well as Maxim
Dounin’s excellent upstream_keepalive module. These parts of code are
copyrighted by these authors, respectively.

This module is still at its very early phase of development and
considered highly experimental. But you’re encouraged to test it out
on your side and report any quirks that you experience :slight_smile:

Here’s some sample configurations:

http {
    ...

    upstream cluster {
        # simple round-robin
        drizzle_server 127.0.0.1:3306 dbname=test
             password=some_pass user=monty protocol=mysql;
        drizzle_server 127.0.0.1:1234 dbname=test2
             password=pass user=bob protocol=drizzle;
    }

    upstream backend {
        drizzle_server 127.0.0.1:3306 dbname=test
             password=some_pass user=monty protocol=mysql;
    }

    server {
        location /mysql {
            set $my_sql 'select * from cats';
            drizzle_query $my_sql;

            drizzle_pass backend;
        }
        ...
    }
}

Essentially it provides a very efficient and flexible way for nginx
internals to access mysql, drizzle, sqlite3, as well as other RDBMS’s
that support the drizzle protocol or mysql protocol. Also it can serve
as a direct REST interface to those RDBMS backends.

It also has a builtin per-worker connection pool mechanism borrowed
from Maxim D.'s upstream_keepalive module.

Here's a sample configuration:

    upstream backend {
        drizzle_server 127.0.0.1:3306 dbname=test
             password=some_pass user=monty protocol=mysql;
        drizzle_keepalive max=100 mode=single overflow=reject;
    }

You may wonder why it will be useful for your PHP/Python/Perl/Java
applications fastcgi’d or proxied by nginx. Well, we’ll work out an
ngx_accel_subrequest module some time in the future to allow these
backend apps directly issue subrequests by means of the special
X-Accel-Subrequest header and continuation passing style.

Unlike the current X-Accel-Redirect trick we’re already familiar with,
X-Accel-Subrequest is more like a function invocation that issues one
(or multiple parallel subrequests), and eventually returns back when
the resulting data the subrequest obtains is ready and gives control
back to your backend apps to go on processing the data.

Sadly this mdoule does not have a wiki page yet, just the source
repository on GitHub:

http://github.com/chaoslawful/drizzle-nginx-module

I’ve done some work in the README file there. I promise I’ll work on
the wiki doc in another day :wink:

Release tarballs can be downloaded from the page below

http://github.com/chaoslawful/drizzle-nginx-module/downloads

I’ve also cc’d the nginx-devel mailing list in hope to get interested
developers to join this project’s development. YOU are always
welcomed!

Happily, we’ve already got a working framework for integrating
third-party libraries like libdrizzle and libpq into nginx’s upstream
system as long as the libraries meet the following prerequisites:

  1. It exposes the underlying socket fd.
  2. It has a truly non-blocking API for its operations.

libdrizzle and libpq meet these conditions while oracle’s OCI library
needs some hack for the requirement #1. So, you can expect ngx_oracle
and even ngx_pgsql to be announced here in the next few weeks or so,
and you’ll see even more! If you’d join the fun, please don’t hesitate
to drop us a line :slight_smile:

Happy new year!

Enjoy!
-agentzh

P.S. I was hoping to release v0.0.1 by this Christmas, but missed the
deadline because I had spent too much time on fixing weird bugs in the
ngx_chunkin module :stuck_out_tongue:

The year 2009 is my first year in the nginx community. Thank you all
for creating such a live community and thanks Igor for creating nginx
in the first place :wink: I’m going to do more nginx C hacking in the next
year! :smiley:

Looks great! Thanks for sharing!

Happy New Year!

Best regards,
Piotr S. < [email protected] >

This is really wonderful thing! Thx for you work.

libdrizzle and libpq meet these conditions while oracle’s OCI library
needs some hack for the requirement #1. So, you can expect ngx_oracle
and even ngx_pgsql to be announced here in the next few weeks or so,
and you’ll see even more! If you’d join the fun, please don’t hesitate
to drop us a line :slight_smile:

Maybe this code helps:
http://www.total-knowledge.com/progs/mod_plsql/
http://www.sharemation.com/~dmcmahon/modowa.htm

Here is a two opensource apache modules to run pl/sql database
procedures via the web.
Alternative to oracle modplsql, dont know about its licence, guess
propietary one.

Posted at Nginx Forum:

Here is one more piece of code, in fact ready module for nginx to work
with OCI:

at the end of post.

Posted at Nginx Forum:

Hello!

On Fri, Feb 12, 2010 at 08:00:10AM -0500, jch wrote:

Here is one more piece of code, in fact ready module for nginx to work with OCI:
http://habrahabr.ru/blogs/nginx/69974/

at the end of post.

It’s not ready module, it’s prototype which shouldn’t be used in
production due to simple (and really common for nginx modules from
unaware authors) reason: the code blocks nginx worker on database
requests.

See catap’s (Kirill Korinskiy) comments there for details.

Maxim D.

On Sat, Feb 13, 2010 at 6:20 PM, agentzh [email protected] wrote:

We hope to kick ngx_oracle out of the door in the near future. And any
sponsorship will be highly appreciated, because our $department’s
business no longer heavily replies on Oracle.

And…we’d like ngx_oracle to share the same upstream mechanism as our
ngx_drizzle, as well as the ngx_postgres module that Piotr S. will
probably start to work on in the near future :wink: And we’d also like
them to emit standard RDS outputs so that the RDS toolchain (including
ngx_rds_json) can work automagically across various RDBMS’s.

Yay for ngx_dbd or ngx_databases!

-agentzh

On Fri, Feb 12, 2010 at 10:36 PM, Maxim D. [email protected]
wrote:

It’s not ready module, it’s prototype which shouldn’t be used in
production due to simple (and really common for nginx modules from
unaware authors) reason: the code blocks nginx worker on database
requests.

See catap’s (Kirill Korinskiy) comments there for details.

We (chaoslawful and I) do have something runnable for ngx_oracle in
our private repository. Note that not every part of OCI can be made
non-blocking; notable exceptions are connection/session creation
operations. So we’re using internal threads to workaround these
blocking parts and nginx’s event model for other parts.

We hope to kick ngx_oracle out of the door in the near future. And any
sponsorship will be highly appreciated, because our $department’s
business no longer heavily replies on Oracle.

Cheers,
-agentzh