Forum: NGINX Nginx, Lua and blocking libraries

8b4249ca3bb8c123da9f7aca63a652e1?d=identicon&s=25 Andre Nathan (Guest)
on 2014-01-09 14:36
(Received via mailing list)
Attachment: signature.asc (554 Bytes)
Hello

I'm considering the possibility of implementing a project using Nginx
and the Lua module. One of the requirements of the project is that the
code must use an embedded database such as SQLite. However, as known,
using the lua-sqlite3 library directly is not optimal because it would
block the Nginx worker process.

My question is, is there a way to work around this in any way? For
example, creating a coroutine to run the lua-sqlite3 calls?

If not, does anyone know of some other embedded database that works well
with the Lua Nginx module?

Thank you in advance,
Andre
37f3ea777f96500b332a1a89d6027897?d=identicon&s=25 Yichun Zhang (agentzh) (Guest)
on 2014-01-09 18:33
(Received via mailing list)
Hello!

On Thu, Jan 9, 2014 at 5:35 AM, Andre Nathan wrote:
> However, as known,
> using the lua-sqlite3 library directly is not optimal because it would
> block the Nginx worker process.
>

Well, I suggest you benchmark the actual performance and measure the
actual blocking effect (We actually have systemtap-based tools to
measure the epoll loop blocking effect:
https://github.com/agentzh/stapxx#epoll-loop-blocking-distr and the
off-CPU flamegraph tool is very useful for determining the
contributors:
https://github.com/agentzh/nginx-systemtap-toolkit...
).

Basically if your database is in-memory, then the blocking effect
should be much smaller because no disk IO involved.

Even if you're using on-disk database, the blocking effect should be
quite small if your disks are fast (like modern SSD cards) and/or the
kernel's page cache's hit rate is high enough.

Nginx core's popular in-file http cache (used by proxy_cache,
fastcgi_cache, and etc) also involves blocking file IO system calls
and people are still enjoying it a lot ;)

It's worth noting that when you're using ngx_lua to embed a database
like SQLite, you should always cache the file descriptor to save
`open` and `close` system calls.

> My question is, is there a way to work around this in any way?

Ideally you should use (or write) a pthread based network service
frontend (be it TCP or just unix domain sockets) for SQLite and let
your Nginx talk to this network interface in a 100% nonblocking way.

> For
> example, creating a coroutine to run the lua-sqlite3 calls?
>

Basically Lua coroutines cannot work around blocking system calls. You
need real OS threads for that.

Regards,
-agentzh
8b4249ca3bb8c123da9f7aca63a652e1?d=identicon&s=25 Andre Nathan (Guest)
on 2014-01-09 19:23
(Received via mailing list)
Attachment: signature.asc (554 Bytes)
Thanks a lot for the detailed answer, Yichun! I'll try to benchmark it,
estimate the db size, see if it fits in memory, etc.

Cheers,
Andre
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.