Ruby Forum NGINX > OT: 'best' dynamic language

Hi,

due the fact that here are a lot of peoples who care about fast and
light environments so I just ask ;-)

What do YOU think is the 'best (smallest/fastest/easiest)' language to
develop a dynamic website?

The requirements are:

MySQL requests (insert/update/delete)
Generate HTML-Files with templates => static files
Work with nginx ;-)
As small as possible mem and cpu usage => efficient interpreter

I have thought about the followings (no order):

perl
python
ruby
php
java
haxe (http://haxe.org/)
lua (http://www.lua.org/)
io (http://www.iolanguage.com/)
.
.
.

I know about

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all

but this page don't tell me how the language behave underload or when
there is a problem with the IO (network/disk/...)

Thank you for your opinions ;-)

Cheers

Aleks

go with django it's a web framework based on python with excellent
documentation and super ease of use.

they abstract database interface for you,

built-in template  system

ready to use admin interface (to manage users) and comment system

at djangoproject.com

-Bedros

Aleksandar Lazic ha scritto:
> MySQL requests (insert/update/delete)
> Generate HTML-Files with templates => static files
> Work with nginx ;-)

Well, *any* language will work with Nginx, as long as it has an
HTTP/FastCGI/SCGI server implementation.

As for languages embedded inside Nginx, the only choices at the moment
are Python and Perl.

> As small as possible mem and cpu usage => efficient interpreter
>

The memory usage usually depends on the package you use.
The python interpreter in mod_wsgi takes about 3616 KB in the master
process and 1908 KB for each worker.


> .
io seems interesting, I did not knew it.

Lua seems the best choice, however it does not have all the packages
available for Python and Perl.


I'm very interested in developing a module for lua (or io), mostly for
testing the best possible integration with the Nginx event module.

For mod_wsgi I have implemented the ngx.poll extension:
http://hg.mperillo.ath.cx/nginx/mod_wsgi/file/tip/examples/nginx-postgres-async.py

but it's not very usable yet.


For really scalable applications you can also consider Erlang;
unfortunayely its syntax is not good, but many use it since there are no
better choices (Haskell is interesting too, but it is not as mature as
Erlang for network applications - and this is very unfortunate)

> .
> .
> 
> I know about
> 
> http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all
> 
> but this page don't tell me how the language behave underload or when
> there is a problem with the IO (network/disk/...)
> 

You can test it with Nginx mod_perl and mod_wsgi.
However if the application is IO bound, then the performance are not the
best.

> Thank you for your opinions ;-)
> 
> Cheers
> 
> Aleks
> 


Regards  Manlio Perillo

On Sun, 2008-04-20 at 23:35 +0200, Aleksandar Lazic wrote:
> MySQL requests (insert/update/delete)
This is OT, but I'd highly recommend PostgreSQL over MySQL for any
application.

> Generate HTML-Files with templates => static files
> Work with nginx ;-)

I don't think these aspects will be affected by language choice.

> As small as possible mem and cpu usage => efficient interpreter

I personally use Python.  I don't consider Python the "best" language by
any stretch of the imagination (in fact, it's a really bad language in a
couple key aspects), but it's a *practical* language.  It's mature, it
has extensive libraries, the interpreter is rock-solid.  There's several
nice web frameworks to select from.  If you care about getting work done
versus doing the coolest thing possible at every moment, you can't go
wrong with Python.

As far as "light", I'd consider Python about average in this regard.

Regards,
Cliff

On Mon, 2008-04-21 at 00:26 +0200, Manlio Perillo wrote:

> 
> I'm very interested in developing a module for lua (or io), mostly for 
> testing the best possible integration with the Nginx event module.

I'd like to see Lua as a replacement/alternative for Nginx's Perl
module.  Unlike most dynamic languages, Lua was originally designed to
be embedded.  I find its syntax much cleaner than Perl and it
undoubtedly has a much smaller memory footprint.

Regards,
Cliff

On Son 20.04.2008 15:22, Bedros Hanounik wrote:
>go with django it's a web framework based on python with excellent
>documentation and super ease of use.
>
>they abstract database interface for you,
>
>built-in template  system
>
>ready to use admin interface (to manage users) and comment system
>
>at djangoproject.com

Jep, it looks nice, but there are a lot of them for different languages:

ruby (merb, nitro, rails, ...)
php (symfony, Akelos, ...)
python (django, cherrypy, ...)
.
.
http://en.wikipedia.org/wiki/Comparison_of_web_application_frameworks

Due the fact that I have not the time and the knowledge to evalute all
of the ;-)), that's the reason why I ask the experts ;-)

Cheers

Aleks

On Son 20.04.2008 15:38, Cliff Wells wrote:
>be embedded.  I find its syntax much cleaner than Perl and it
>undoubtedly has a much smaller memory footprint.

I prefer nekovm (http://www.nekovm.org/) for this ;-)

Look:

http://lists.motion-twin.com/pipermail/neko/2008-April/thread.html

=> NGinx and NekoVM

Cheers

Aleks

On Mon 21.04.2008 00:26, Manlio Perillo wrote:
>
> Well, *any* language will work with Nginx, as long as it has an
> HTTP/FastCGI/SCGI server implementation.

Jep ;-)

> As for languages embedded inside Nginx, the only choices at the moment
> are Python and Perl.
>
>> As small as possible mem and cpu usage => efficient interpreter
>>
>
> The memory usage usually depends on the package you use.
> The python interpreter in mod_wsgi takes about 3616 KB in the master
> process and 1908 KB for each worker.

Thanks for this info ;-)

>
> io seems interesting, I did not knew it.

Jep.

> I'm very interested in developing a module for lua (or io), mostly for
> testing the best possible integration with the Nginx event module.

Cool, sounds very interesting ;-)

> You can test it with Nginx mod_perl and mod_wsgi.
> However if the application is IO bound, then the performance are not
> the best.

Thanks ;-)

Cheers

Aleks

On Son 20.04.2008 15:35, Cliff Wells wrote:
>
>On Sun, 2008-04-20 at 23:35 +0200, Aleksandar Lazic wrote:
>> 
>> The requirements are:
>> 
>> MySQL requests (insert/update/delete)
>
>This is OT, but I'd highly recommend PostgreSQL over MySQL for any
>application.

Ok.

>As far as "light", I'd consider Python about average in this regard. 
Thanks for your opinion ;-)

Cheers

Aleks

I can suggest you take a look at Kepler Project
(http://kepler-project.org/) which is trying to build a web framework
on top of lua and it works well with fastcgi so it will be no problem
to run it under Nginx. They are almost ready with their 1.0 release
and they took a lot of testing and tweaking to make sure it works on
OS X, Linux and also Windows. I was on their mailing list for quite
some time and these people are persistent :) And in the last months
they had so much traffic on the list that I had to unsubscribe. But if
I even need to do anything low memory and fast I would use Lua. The
language is fast and the syntax is very easy to pickup and use.

Kiril

Manlio,

I would be all over a module for nginx which has the same
functionality as mod_magnet on Lighttpd and I can see already you are
very much into Nginx so I am sure it will be a no brainer for you :)


On Sun, Apr 20, 2008 at 6:26 PM, Manlio Perillo

Ah, http://www.keplerproject.org/

On Mon, Apr 21, 2008 at 09:50:58AM +0200, Aleksandar Lazic wrote:

> >module.  Unlike most dynamic languages, Lua was originally designed to
> >be embedded.  I find its syntax much cleaner than Perl and it
> >undoubtedly has a much smaller memory footprint.
> 
> I prefer nekovm (http://www.nekovm.org/) for this ;-)
> 
> Look:
> 
> http://lists.motion-twin.com/pipermail/neko/2008-April/thread.html
> 
> => NGinx and NekoVM

It seems that Neko as well as Lua, perl, etc do the same in memory
allocation failure case: exit() or nothing, i.e., segfault.

> What do YOU think is the 'best (smallest/fastest/easiest)' language to
> develop a dynamic website?

Well, smallest =/= fastest =/= easiest.

And ... picking correct framework or appserver may have greater impact
here than picking the languaga as such.

My current preference for such a tasks is Python, Mako and ... any
framework which allows SQLAlchemy ;-)

Regarding nginx support: currently I tend to believe that using
nginx as reverse proxy (and static file server) may be the best
runtime configuration....

Igor Sysoev ha scritto:
>>> be embedded.  I find its syntax much cleaner than Perl and it
> allocation failure case: exit() or nothing, i.e., segfault.
> 

 From what I can see, Lua (as Python) throws an exception in case of
memory allocation failure.

Python uses a statically allocated object for the Memory Error 
exception.

Lua uses _longjmp/_setjmp, with integer representing error codes.
It only calls exit if no exception handler is installed.



Manlio Perillo

On Tue, Apr 22, 2008 at 12:23:26PM +0200, Manlio Perillo wrote:

> >>>module.  Unlike most dynamic languages, Lua was originally designed to
> >It seems that Neko as well as Lua, perl, etc do the same in memory
> >allocation failure case: exit() or nothing, i.e., segfault.
> >
> 
> From what I can see, Lua (as Python) throws an exception in case of 
> memory allocation failure.
> 
> Python uses a statically allocated object for the Memory Error exception.
> 
> Lua uses _longjmp/_setjmp, with integer representing error codes.
> It only calls exit if no exception handler is installed.

Well, but what can I do in exception handler ?
Destroy a whole interpreter, leaving various leaks ?

Igor Sysoev ha scritto:
>>>>>> module.
>>>> => NGinx and NekoVM
> 
> Well, but what can I do in exception handler ?
> Destroy a whole interpreter, leaving various leaks ?
> 

With Lua you can supply you allocator function.



Manlio Perillo

Marcin Kasperski ha scritto:
> 
The same for me ;-)
But lately I'm trying to avoid to use the SQLAlchemy ORM.

> Regarding nginx support: currently I tend to believe that using
> nginx as reverse proxy (and static file server) may be the best
> runtime configuration....
> 

The problem with reverse proxy is that you need another server, and this
server is usually written in Python, and most of the time it uses
threads for concurrency, so it's not the best solution (unless, of
course, you don't run a whole "cluster" of servers, like Ruby
programmers like to do)


IMHO, a better solution is:
1) Use Nginx as main server + reverse proxy
2) Run your application embedded in Apache or
    Nginx (but the application should be written with care)



Manlio Perillo

On Tue, Apr 22, 2008 at 12:55:17PM +0200, Manlio Perillo wrote:

> >>>>>>for testing the best possible integration with the Nginx event
> >>>>
> >>It only calls exit if no exception handler is installed.
> >
> >Well, but what can I do in exception handler ?
> >Destroy a whole interpreter, leaving various leaks ?
> >
> 
> With Lua you can supply you allocator function.

It does not resolve the problem. The interpreter internally must test
result of EVERY function that may fail on memory allocations:

void *
interpreter_function0()
{
    p = interpreter_function1();
    if (p == NULL) {
        some rollback operations
        return NULL;
    }
}

>> Regarding nginx support: currently I tend to believe that using
>> nginx as reverse proxy (and static file server) may be the best
>> runtime configuration....
>
> The problem with reverse proxy is that you need another server,

FastCGI/SCGI/etc are also another server, just less explicit ;-)

> and this server is usually written in Python, and most of the time
> it uses threads for concurrency,

If you are hardcore, you can use twisted. But IMO it is overkill for
web.

Also, threads are not that bad in such context. Note that here you have
non-trivial code, which makes some database operations, dynamically
formats pages using templates, etc. The cost of some context switches
is far less noticeable than in case of static file serving.

> 1) Use Nginx as main server + reverse proxy
> 2) Run your application embedded in Apache or
>    Nginx (but the application should be written with care)

I do not see any advantages of apache here (unless you want to
use existing permissions, authorization etc infrastructure already
defined for apache).

Igor Sysoev ha scritto:
>>>> It only calls exit if no exception handler is installed.
>>> Well, but what can I do in exception handler ?
>>> Destroy a whole interpreter, leaving various leaks ?
>>>
>> With Lua you can supply you allocator function.
> 
> It does not resolve the problem. The interpreter internally 

The Lua interpreter?

>     }
> }
> 

Not sure to understand, but since memory allocation failure is signaled
via an exception, the code flow is quite different.

 From http://www.lua.org/source/5.1/ldo.c.html#luaD_rawrunprotected:

   struct lua_longjmp lj;
   lj.status = 0;
   lj.previous = L->errorJmp;  /* chain new error handler */
   L->errorJmp = &lj;

   LUAI_TRY(L, &lj,
     (*f)(L, ud);
   );

   L->errorJmp = lj.previous;  /* restore old error handler */
   if (lj.status != 0) {
      ... if memory error, destroy the interpreter
   }


and:
#define LUAI_TRY(L,c,a) if (setjmp((c)->b) == 0) { a }






Manlio Perillo

Kiril Angov ha scritto:
> Manlio,
> 
> I would be all over a module for nginx which has the same
> functionality as mod_magnet on Lighttpd and I can see already you are
> very much into Nginx so I am sure it will be a no brainer for you :)
> 

Yes, but the problem is time :)

I'm already working (among other projects and work) on mod_wsgi and
mod_scgi.
In future I plan to "revive" the mod_pg project, and to write a
digest_auth module (if it is still not in the Igor's schedule).


 > [...]


Manlio Perillo

On Tue, Apr 22, 2008 at 03:18:48PM +0200, Manlio Perillo wrote:

> >>>>
> >>>>Lua uses _longjmp/_setjmp, with integer representing error codes.
> >>>>It only calls exit if no exception handler is installed.
> >>>Well, but what can I do in exception handler ?
> >>>Destroy a whole interpreter, leaving various leaks ?
> >>>
> >>With Lua you can supply you allocator function.
> >
> >It does not resolve the problem. The interpreter internally 
> 
> The Lua interpreter?

Yes.

> >    }
>   lj.previous = L->errorJmp;  /* chain new error handler */
> 
> 
> and:
> #define LUAI_TRY(L,c,a) if (setjmp((c)->b) == 0) { a }

I understand the exception mechanic, this way is easy to program -
you do not need to worry about all these tests, but I'm not sure, that
it is safe way.

Marcin Kasperski ha scritto:
>>> Regarding nginx support: currently I tend to believe that using
>>> nginx as reverse proxy (and static file server) may be the best
>>> runtime configuration....
>> The problem with reverse proxy is that you need another server,
> 
> FastCGI/SCGI/etc are also another server, just less explicit ;-)
> 

Of course :)

>> and this server is usually written in Python, and most of the time
>> it uses threads for concurrency,
> 
> If you are hardcore, you can use twisted. But IMO it is overkill for
> web.
> 

By the way I'm a Twisted programmer :).
This how I came to know and low asynchronous programming!

And I have also used Twisted Web (and Nevow) for a rather big web
application.

However I think that Nginx + mod_wsgi + templating system like Mako
produces applications that are more maintainable and modular.

> Also, threads are not that bad in such context. Note that here you have
> non-trivial code, which makes some database operations, dynamically
> formats pages using templates, etc. The cost of some context switches
> is far less noticeable than in case of static file serving.
> 

The problem is not with context switches, but with the Python Global
Interpreter Lock.

>> 1) Use Nginx as main server + reverse proxy
>> 2) Run your application embedded in Apache or
>>    Nginx (but the application should be written with care)
> 
> I do not see any advantages of apache here (unless you want to
> use existing permissions, authorization etc infrastructure already
> defined for apache).
> 

No, Apache *has* advantages.
You have a very robust server for your application, and it's wsgi module
allows much more control then any of the "custom" servers written in 
Python.




Manlio Perillo

Of course!

I will be happy if you put it on your todo list and whenever you can
get to it. I will help with testing when the time comes.

Regards,
Kiril

On Tue, Apr 22, 2008 at 9:50 AM, Manlio Perillo

On Tue, 2008-04-22 at 13:07 +0200, Manlio Perillo wrote:

> The problem with reverse proxy is that you need another server, and this 
> server is usually written in Python, and most of the time it uses 
> threads for concurrency, so it's not the best solution (unless, of 
> course, you don't run a whole "cluster" of servers, like Ruby 
> programmers like to do)

FAPWS is a non-threaded, libevent-based WSGI server:

http://william-os4y.livejournal.com/

I've tested it and it is quite fast (~5000 req/s is achievable on decent
hardware).

Regards,
Cliff

Igor Sysoev ha scritto:
>>> void *
>> via an exception, the code flow is quite different.
>>   );
> I understand the exception mechanic, this way is easy to program -
> you do not need to worry about all these tests, but I'm not sure, that
> it is safe way.
> 

Ah, ok.
But do you have doubts only about handling memory allocation failures
via exceptions, about exception handling in general or about exception
handling as implemented in Lua?

> 



Manlio Perillo

Cliff Wells ha scritto:
> http://william-os4y.livejournal.com/
> 
> I've tested it and it is quite fast (~5000 req/s is achievable on decent
> hardware). 
> 

What type of application have you tested?

One limitation of FAPWS (like Twisted) is that it is single process.

> Regards,
> Cliff
> 


Regards  Manlio Perillo

Hi Folks

Currently I don't see problem using one or other Programming Language, 
nginx
can work very fine with any PL.

I'm using both Ruby and perl with optimal results in both Apache and 
nginx;
I prefer nginx for hosting RoR apps (like Mephisto, Typo, Radiant, or my
own's). Perl also work with FastCGI and work fine, maybe it requiere a 
bit
of work but it's functional.

I think, that you can use any language, whenever you are satisfied with 
it.

Best Regards.

On Tue, 2008-04-22 at 22:26 +0200, Manlio Perillo wrote:
> > 
> > http://william-os4y.livejournal.com/
> > 
> > I've tested it and it is quite fast (~5000 req/s is achievable on decent
> > hardware). 
> > 
> 
> What type of application have you tested?

I've only done testing with minimal "hello, world" type apps.  The
author claims to have run a simple Django app (wiki) under it with a
significant performance increase over other methods.

> One limitation of FAPWS (like Twisted) is that it is single process.

The author is working on this aspect.  Personally I'd simply
load-balance several instances behind Nginx, but he wants to do it
within FAPWS.

Regards,
Cliff

>
>
> IMHO, a better solution is:
> 1) Use Nginx as main server + reverse proxy
> 2) Run your application embedded in Apache or
>   Nginx (but the application should be written with care)
>
this methodology works really well for us, esp when you throw in
memcache for data that is only semi volatile.

I too would be really into an embedded Lua module.  Lua and nginx seem 
like
they're a very good match, we just need to make the introduction.

I've only written a couple of nginx modules (both pretty simple and 
specific
for my weird needs) so I don't think I'm quite up to this task yet.  But 
I'd
love to help out if someone with more expertise would be willing to 
point me
in the right direction.

>j.

On Tue, Apr 22, 2008 at 6:50 AM, Manlio Perillo 
<manlio_perillo@libero.it>

On Mon 21.04.2008 22:57, Kiril Angov wrote:
>Ah, http://www.keplerproject.org/

Thank you I will take a look ;-)

On Die 22.04.2008 16:31, Manlio Perillo wrote:
> Marcin Kasperski ha scritto:
>>>> Regarding nginx support: currently I tend to believe that using
>>>> nginx as reverse proxy (and static file server) may be the best
>>>> runtime configuration....
>>> The problem with reverse proxy is that you need another server,
>> FastCGI/SCGI/etc are also another server, just less explicit ;-)
>
> Of course :)

Full ack ;-)

>>> and this server is usually written in Python, and most of the time
>>> it uses threads for concurrency,
>> If you are hardcore, you can use twisted. But IMO it is overkill for
>> web.
>
> By the way I'm a Twisted programmer :).

Cool ;-)

cheers

Aleks

Hi all,

On Son 20.04.2008 23:35, Aleksandar Lazic wrote:
> Hi,
>
> due the fact that here are a lot of peoples who care about fast and
> light environments so I just ask ;-)
>
> What do YOU think is the 'best (smallest/fastest/easiest)' language to
> develop a dynamic website?

Thank you for all your input ;-).

Asa I have a decision I will communicate it ;-)

Cheers

Aleks

Hi Igor,

On Die 22.04.2008 10:14, Igor Sysoev wrote:
>It seems that Neko as well as Lua, perl, etc do the same in memory
>allocation failure case: exit() or nothing, i.e., segfault.

The developer of neko (Nicolas Cannasse) have agreed that he can add a
memory management hook at init state so that it is possible to use the
mm of nginx.

Request:
http://lists.motion-twin.com/pipermail/neko/2008-April/002194.htm

Response:
http://lists.motion-twin.com/pipermail/neko/2008-April/002194.html

what do you think is this enough for the integration or do you need
something more from neko?!

Cheers

Aleks ;-)

On Sat, Apr 26, 2008 at 07:56:45AM +0200, Aleksandar Lazic wrote:

> >>
> http://lists.motion-twin.com/pipermail/neko/2008-April/002194.htm
> 
> Response:
> http://lists.motion-twin.com/pipermail/neko/2008-April/002194.html
> 
> what do you think is this enough for the integration or do you need
> something more from neko?!

The problem is not in hooks/etc.
The problem is that interpreter MUST TEST EVERY operation result that
may fail on memory allocation. And it MUST return an error to all higher
levels, closing and freeing all allocated resources on the back way.

The existent interpreters either do not test result in most cases 
(perl),
or simply exit(), or in best case they throw exception. Exceptions are 
easy
way to program (you not need to test most operations) and cheap way to 
test
results (for the same reason), but they may lead to socket/file 
descriptor/etc
leak.

Igor Sysoev ha scritto:
> 
I understand the problem, however I think that Lua is still usable.

I'm reading the source code of Lua io library, and any opened file is
closed when reached by the gc.

This means that when Nginx detects an error, it can just force a full gc
cycle.

If this still does not sounds safe, Nginx can just create a Lua state
(interpreter) for each request, finalizing it when the request is 
finalized.

This is both feasible and efficient (but better is one of Lua language
developer can confirm it).



Regards   Manlio Perillo

On Sat, Apr 26, 2008 at 11:59:38AM +0200, Manlio Perillo wrote:

> >descriptor/etc
> 
> If this still does not sounds safe, Nginx can just create a Lua state 
> (interpreter) for each request, finalizing it when the request is finalized.
> 
> This is both feasible and efficient (but better is one of Lua language 
> developer can confirm it).

Can this per request interpreter run precompiled code, or will it 
compile
it on its creation ?

Igor Sysoev ha scritto:
>>> way to program (you not need to test most operations) and cheap way to test
>> cycle.
>>
>> If this still does not sounds safe, Nginx can just create a Lua state 
>> (interpreter) for each request, finalizing it when the request is finalized.
>>
>> This is both feasible and efficient (but better is one of Lua language 
>> developer can confirm it).
> 
> Can this per request interpreter run precompiled code, or will it compile
> it on its creation ?
> 

You can precompile all the Lua code at Nginx configuration time:
http://www.lua.org/manual/5.1/manual.html#lua_load


The lua_load function reads and parses a Lua code chunk and return (in
the Lua stack) the compiled code (or an error code).


So, if I'm not wrong, you need to:
1) Create a Lua state at the begin of the configuration phase
2) Use this Lua state for precompiling all the Lua code in Nginx
3) Finalize this Lua State at the end of configuration phase
4) For each request create a new Lua state
5) Use this Lua state for executing the precompiled script code
6) Finalize the Lua state at the end of the request



But, again, I have never used Lua, I'm just reading the reference manual
and the source code, so better if a Lua developer can confirm this.



Regards  Manlio Perillo

Manlio Perillo ha scritto:
>> test
>> results (for the same reason), but they may lead to socket/file 
>> descriptor/etc
>> leak.
>>
> 
> I understand the problem, however I think that Lua is still usable.
> 
> I'm reading the source code of Lua io library, and any opened file is 
> closed when reached by the gc.
> 

By the way, I'm reading the source code for the NekoVM `file` library,
and it *seems* that an opened file object must be explicitly closed, it
is not garbage collected.

Moreover, this piece of code (gc_alloc_block function in vm/gc.c) is
quite impressive:

#define ASSERT()    *(char*)NULL = 0

...

p = malloc(sizeof(gc_page));
if( p == NULL )
     ASSERT();





Another unhandled memory error is in vm/context.c:

_context *context_new() {
  _context *ctx = malloc(sizeof(_context));
  pthread_key_create( &ctx->key, NULL );
  return ctx;
}


So, unless I'm wrong, NekoVM does seems to meets Igor requirements.

 > [...]



Regards  Manlio Perillo

Manlio Perillo ha scritto:
>> test
> This means that when Nginx detects an error, it can just force a full gc 
> cycle.
> 
> If this still does not sounds safe, Nginx can just create a Lua state 
> (interpreter) for each request, finalizing it when the request is 
> finalized.
> 
> This is both feasible and efficient (but better is one of Lua language 
> developer can confirm it).
> 

I have written a small program and I can confirm that opened files are
correctly closed when the Lua interpreter is finalized.

The program is attached: the Lua code simply open a big file and read
all its content in memory.

There is a custom memory allocator that make sure to fail when a large
amount of memory is required.


There is a strange behaviour, however.
The status code returned by luaL_dostring should be
4 (LUA_ERRMEM), instead 1 is returned.



Regards   Manlio Perillo

Manlio Perillo ha scritto:
> 
> There is a strange behaviour, however.
> The status code returned by luaL_dostring should be
> 4 (LUA_ERRMEM), instead 1 is returned.
> 

Ok, solved.

The last luaL_dostring should be replaced with:

   status = luaL_loadstring(L, CHUNCK_2);
   if (status != 0) {
     printf("syntax error in CHUNK_2\n");
     return EXIT_FAILURE;
   }

   status = lua_pcall(L, 0, 0, 0);
   msg = lua_tolstring(L, 1, &len);
   printf("execution status 2: %i:%s\n", status, msg);


since luaL_dostring always returns 1 in case of errors.


Updated code is here: http://rafb.net/p/yOVhjw87.html



Regards  Manlio Perillo

On Sat, Apr 26, 2008 at 05:28:38PM +0200, Manlio Perillo wrote:

> >
>   status = luaL_loadstring(L, CHUNCK_2);
> since luaL_dostring always returns 1 in case of errors.
> 
> 
> Updated code is here: http://rafb.net/p/yOVhjw87.html

Thank you for your investigation. I will look Lua.

On Sat, 26 Apr 2008 12:46:00 +0200
Manlio Perillo <manlio_perillo@libero.it> wrote:

> The lua_load function reads and parses a Lua code chunk and return
> (in the Lua stack) the compiled code (or an error code).

Sure, and you can even byte-compile code into memory buffers by using
luaU_dump (along with a custom writer) and then loading them into the VM
by using lua_load (the luac compiler works by those functions, which
are present in the Lua library, its source code is in luac.c and can be
read easily.)

> this.
I am using Lua in a World Domination™ project I can't talk about (and
because it's fun! :-P)

That method will work, but I think it would be a bit faster not
creating an entire VM on each request. Using lua_newthread you can fork
off a new lua_State from an existing one, which will share the same
globals but will have its own execution stack. You could do:

1. Create a lua_State before configuring (call it "main" state)
2. Load all Lua code into the "main" state.
3. Fork off a new "thread" with lua_newthread using the "main"
   lua_State as starting point when a request arrives.
4. Run code into the forked "thread"
5. Let Lua garbage-collect the finished threads (or force a collection
   at the end of the request, or every n-th request, or whatever).

I wrote "thread" with quotes because they are coroutines, not operating
system "full-blown threads", they are very light and are implemented in
the VM.

One advantage of using coroutines is that they can yield and pass
control asynchronously to Nginx. Coroutines can be resumed from C by
using lua_resume. Also, you can set up a fake global environment on a
coroutine by using lua_setfenv (e.g. to only allow read access to
globals).

Lua is wonderful in many ways, but what I like most is that it gives
you a small set of features which can be used to achieve nearly
everything, hehe.

Just my two cents :D

Cheers,

-Adrian

Adrian Perez ha scritto:
>> manual and the source code, so better if a Lua developer can confirm
>> this.
> 
> I am using Lua in a World Domination™ project I can't talk about (and
> because it's fun! :-P)
> 


May be it a game? ;-).

> That method will work, but I think it would be a bit faster not
> creating an entire VM on each request. Using lua_newthread you can fork
> off a new lua_State from an existing one, which will share the same
> globals but will have its own execution stack. 
> You could do:
> 

I was suggesting to use one VM per request since you can be sure that
there are no leaks, and you have more control.

As an example in the code I have posted, the file is not closed by just
running a cycle of the gc (but probabily I'm doing it in the wrong way).

Of course this should be a configurable parameter.
Using one VM per request means that one can not have persistent
connections to a database, as an example.

> 1. Create a lua_State before configuring (call it "main" state)
> 2. Load all Lua code into the "main" state.
> 3. Fork off a new "thread" with lua_newthread using the "main"
>    lua_State as starting point when a request arrives.

How is this possible?

> using lua_resume. 
Right, this is the main reason why I'm interested in Lua inside Nginx!

> Cheers,
> 
> -Adrian
> 


Regards  Manlio Perillo

On Sat, 26 Apr 2008 19:42:35 +0200
Manlio Perillo <manlio_perillo@libero.it> wrote:

> Adrian Perez ha scritto:
> > [...]
> >
> > I am using Lua in a World Domination™ project I can't talk about
> > (and because it's fun! :-P)
> > 
> 
> May be it a game? ;-).

Not precisely... but it is fun to code :D

> Of course this should be a configurable parameter.
> Using one VM per request means that one can not have persistent 
> connections to a database, as an example.

Well, one can always store persistent values in a hash table in the C
side of the world, and provide a view of it to the Lua side by adding
e.g. an userdata with table metamethds named "persistend" to every VM
state, so one could handle requests the following way:

  if persistent.db_conn ~= nil then
     db_conn = persistent.db_conn
  else
     db_conn = persistent.db_conn = open_database_connection()
  fi

  handle_request(db_conn)


Have a nice day!


-Adrian