Nginx feature request

Server NGINX
1

Hi all,

Discovered Nginx a few weeks back. After some experimenting I have come
to the conclusion that Nginx really is very very good.

I have a small feature request:

I
have a set of static files (html and others) which I want to make
available only from a certain date/time. Inspired by the rewrite
module, this morning I thought of a simple method to control access to
such files.

My idea was to set the file creation/modification
time of the static files in the future, i.e. the date/time from which
they can be made available (e.g. I could use the touch tool to set the
appropriate date/time). I could then do something like

if (!-f $request_filename) {return 404;}
if ($date_gmt < fct($request_filename)) {return 403;}

Unfortunately,
I haven’t found a way to get at the file creation/modification time
(please do
let me know if I overlooked something). The ‘fct’ in my example is an
imaginary function which returns the file creation time in the same
format as $date_gmt (presumably unix epoch timestamp)

An even
better way to handle this would be to have a directive in the core
module which disallows serving files which have a creation/modification
time in the future. I could then use:

location /ftc/ {
filetime_check on;
}

to disallow serving of files with a filetime in the future for that
location. Files would then automatically become available once the
request time >= filetime. This way access to certain files could be
controlled in a very straightforward and transparent way - and with a
simple ‘touch’.

Though
I’d definitely prefer a directive for the above purpose, access to file
creation/modification time (through variable or function) could still
be useful in the rewriting or ssi module.

Anyone any other suggestions ?

Thanks,

Hendrik H.

2

check out secdownload module in lighttpd

http://trac.lighttpd.net/trac/wiki/Docs:ModSecDownload

I’ve already made a feature request for something like that in nginx;
but I
believe the developers of nginx did not think building such a module is
of
high priority.

lighttpd has a memory leak, and I’m not sure if they actually solved it
(probably not). I personally prefer nginx, but I wish they build a
module
like secdownload.

On Feb 10, 2008 12:56 AM, Hendrik H.
[email protected]

3

Hendrik H. ha scritto:

something).
You can use the embedded Perl module.

[…]

Manlio P.

4

Manlio,

Thanks for the suggestion, but when I propose this feature, it’s exactly
because I want to avoid something on top of Nginx (whether Perl,
Python, PHP or anything else) !

My need is very simple. All I want is to be able to disallow Nginx to
serve certain files based on filetime. This could be done in two ways:

A directive which is off by default but which can be switched on in any
of the standard places - most likely place would be in ‘location’. Let
me tentatively name this directive ‘filetime_check’. If the directive is
on, nginx would not serve any files with a filetime later than current
server/request time. Instead it would return a 403 error so that we can
distinguish between ‘file not found’ and ‘access not allowed’.
You could then define something like:

location /ftc/ {filetime_check on;}

Any request for a file in location /ftc/ with a filetime later than
request time (i.e. in the future) would then return 403.

Independent from the above, it might be useful to be able to access the
filetime of requested file for defining rewriting rules or for ssi. This
could either be through a variable or through a function:

if ($date_gmt < fct($request_filename)) {do something, e.g. return 403;}

Nginx already has to check the last modification time of requested
files, so adding a directive as proposed in 1 shouldn’t put a lot of
extra burden on it. Shouldn’t be too hard to implement this myself
privately, but I thought that others might find this useful as well and
in that case an official feature is probably better.

Hendrik H.

5

Hendrik H. ha scritto:

Manlio,

Thanks for the suggestion, but when I propose this feature, it’s exactly
because I want to avoid something on top of Nginx (whether Perl,
Python, PHP or anything else) !

Well, you need scripting support, so I don’t understand why you should
avoid to use Perl.

nginx scripting module is very limited, and Igor has expressed
intentions to improve it.

Maybe a better language to embed in nginx is LUA, but this is another
question.

My need is very simple. All I want is to be able to disallow Nginx to
serve certain files based on filetime. This could be done in two ways:

A directive which is off by default but which can be switched on in any
of the standard places - most likely place would be in ‘location’. Let
me tentatively name this directive ‘filetime_check’.

This should be easy to implement with a post access handler module, as
an example.

Independent from the above, it might be useful to be able to access the
filetime of requested file for defining rewriting rules or for ssi. This
could either be through a variable or through a function:

if ($date_gmt < fct($request_filename)) {do something, e.g. return 403;}

This too is easy to implement patching the rewrite module, but nginx
will not be able to do the comparison (it only support boolean and regex
opoerators).

Better to add a condition that will return true if the last modification
time is in the future.

[…]

Manlio P.

6

Hi Bedros,

The module you refer to is not exactly what I have requested.

The feature I propose is very simple: a method / directive to disallow
serving a file which has a filetime (last modified time) set in the
future. E.g. if I set the last modified time for a certain file to 4
March 2008 14:30, then any request for the file before that time would
result in a 403. For any request made after that time the file will be
served as usual.

Such a directive would make life a lot simpler for me since I have
thousands of small static data files which should become available only
from a certain time. I could pregenerate lots of files, place them in
the corresponding directories (for which I switched on the directive
filetime_check), set any wanted filetime (e.g. with touch) and Nginx
would do all the hard :slight_smile: work by returning 403 instead of the file
whenever the request is made before the last modification time of the
file. As simple as that ! Perl, databases, etc. are too much overhead
and I would not want to use any of those for this purpose.

Hendrik H.

Date: Sun, 10 Feb 2008 08:54:59 -0800
From: [email protected]
To: [email protected]
Subject: Re: Nginx feature request

check out secdownload module in lighttpd

I’ve already made a feature request for something like that in nginx;
but I believe the developers of nginx did not think building such a
module is of high priority.

lighttpd has a memory leak, and I’m not sure if they actually solved it
(probably not). I personally prefer nginx, but I wish they build a
module like secdownload.

On Feb 10, 2008 12:56 AM, Hendrik H.
[email protected] wrote:

Hi all,

Discovered Nginx a few weeks back. After some experimenting I have come
to the conclusion that Nginx really is very very good.

I have a small feature request:

I
have a set of static files (html and others) which I want to make
available only from a certain date/time. Inspired by the rewrite
module, this morning I thought of a simple method to control access to
such files.

My idea was to set the file creation/modification
time of the static files in the future, i.e. the date/time from which
they can be made available (e.g. I could use the touch tool to set the
appropriate date/time). I could then do something like

if (!-f $request_filename) {return 404;}
if ($date_gmt < fct($request_filename)) {return 403;}

Unfortunately,
I haven’t found a way to get at the file creation/modification time
(please do
let me know if I overlooked something). The ‘fct’ in my example is an
imaginary function which returns the file creation time in the same
format as $date_gmt (presumably unix epoch timestamp)

An even
better way to handle this would be to have a directive in the core
module which disallows serving files which have a creation/modification
time in the future. I could then use:

location /ftc/ {
filetime_check on;
}

to disallow serving of files with a filetime in the future for that
location. Files would then automatically become available once the
request time >= filetime. This way access to certain files could be
controlled in a very straightforward and transparent way - and with a
simple ‘touch’.

Though
I’d definitely prefer a directive for the above purpose, access to file
creation/modification time (through variable or function) could still
be useful in the rewriting or ssi module.

Anyone any other suggestions ?

Thanks,

Hendrik H.

Post free auto ads on Yello Classifieds now! Try it now!

7

Manlio,

Thanks a lot for your thoughts. I will look into your suggestions for
implementing points 1 and/or 2 below.

As for scripting, for this particular purpose (serving static files) I
don’t require any scripting support. So adding Perl, or any other
language for that matter, on top would be overkill.

To be frank, I’m personally also not a big fan of Perl. I’d prefer
Python over Perl (I’m sorry about that :-).
Embedded LUA – anything else with the smallest possible overhead - could
be nice though for some purposes.

For dynamic content I myself use Nginx upstream to several
purpose-written asynchronous servers, written in Python on top of the
libevent module (libevent). I use
Libevent as the actual server engine (probably quite close in
performance to Nginx) which takes care of receiving / sending. I don’t
even bother about FastCGI. Each Python server listens on a particular
port (behind Nginx upstream), is written to handle particular requests
and has only the Python code required for that to keep overhead as small
as possible.

I also use SSI wherever possible - I wish Nginx had a few more options
in this area.

By pregenerating static files with content that should become available
only after a certain time I can exploit Nginx even better - that is, if
one day it has the feature I propose in this thread !

Hope this provides a better insight into my reasoning for proposing a
filetime check option / directive.

Hendrik

8

Manlio P. ha scritto:

This too is easy to implement patching the rewrite module, but nginx
will not be able to do the comparison (it only support boolean and regex
opoerators).

A correction: this feature requires to patch the http script module too.

Manlio P.

9

Manlio,

Thanks for the correction. Will try to look into this when I find some
time. So far, I haven’t even looked at any Nginx code, let alone tried
to understand how it works.

Anyway, option 1 (directive) seems to be the easiest to implement. So
will look at that first.

Hendrik

10

Hendrik H. ha scritto:

Python over Perl (I’m sorry about that :-).
Well, I’m a Python programmer and the author of the WSGI module for
nginx :).

Embedded LUA – anything else with the smallest possible overhead - could
be nice though for some purposes.

For dynamic content I myself use Nginx upstream to several
purpose-written asynchronous servers, written in Python on top of the
libevent module (libevent). I use
Libevent as the actual server engine (probably quite close in
performance to Nginx) which takes care of receiving / sending.

Interesting, but I prefer to use Python embedded in Nginx, since it is
more robust.
In future I hope to be able to add support for asynchronous Python
applications.

I think that it would be interesting to have some benchmarks of mod_wsgi
against libevent based Python servers.

I don’t
even bother about FastCGI. Each Python server listens on a particular
port (behind Nginx upstream), is written to handle particular requests
and has only the Python code required for that to keep overhead as small
as possible.

This is a Goog Thing!

I also use SSI wherever possible - I wish Nginx had a few more options
in this area.

By pregenerating static files with content that should become available
only after a certain time I can exploit Nginx even better - that is, if
one day it has the feature I propose in this thread !

Hope this provides a better insight into my reasoning for proposing a
filetime check option / directive.

The problem is that this is a very specialized need, so you should
implement it by yourself, of rent a coder that can work on the nginx
codebase for you.

Hendrik

Manlio P.

11

Manlio,

Interesting, but I prefer to use Python embedded in Nginx, since it is
more robust.
In future I hope to be able to add support for asynchronous Python
applications.

I agree that embedded Python would be more robust, but then I would
probably still use standard Nginx in front with perhaps one or more
bare-bones Nginx servers upstream with embedded Python listening on
particular ports. With embedded Python I don’t see a need for some of
the standard modules, e.g. the rewriting module. That’s something the
Nginx server in front could take care of. I’d rather see something like
Libevent (i.e. an asynchronous HTTP server engine) with embedded Python.
The core of Nginx is probably similar to Libevent with its HTTP layer.
Would be interesting to find out how Nginx core compares to Libevent
with HTTP layer in terms of performance.

A core Nginx engine with embedded Python would definitely interest me !
That’s for sure more robust than Python on top of Libevent. Already
binding Python to the latest version of Libevent was quite tricky. Both
existing Python modules for libevent (pyevent / libevent-python) are
outdated and I had to use the ctypes library to bind to Libevent.
Wouldn’t mind replacing Python->ctypes->libevent by an Nginx core with
embedded Python.

I think that it would be interesting to have some benchmarks of mod_wsgi
against libevent based Python servers.

I believe it would be difficult to compare a more general purpose
mod_wsgi with a purpose-written Python server based on libevent. I
myself have no experience with wsgi and am not in a position to compare,
but you can try.

The problem is that this is a very specialized need, so you should
implement it by yourself, of rent a coder that can work on the nginx
codebase for you.

Will probably implement it myself, unless there’s someone else out there
who would be interested to give it a try. For the time being I have
myself no time whatsoever, at least not in the next two months.

Hendrik

12

Just a quick blurb for libevent users: you may want to check out libev.

From http://software.schmorp.de/pkg/libev.html:
“A full-featured and high-performance (see
benchmarkhttp://libev.schmorp.de/bench.html)
event loop that is loosely modelled after libevent, but without its
limitations and bugs. It is used, among others, in the GNU Virtual
Private
Ethernet http://savannah.gnu.org/projects/gvpe and
rxvt-unicodehttp://software.schmorp.de/pkg/rxvt-unicode.htmlpackages,
and in the Deliantra MORPG Server and Client.”

On Feb 10, 2008 1:39 PM, Hendrik H. [email protected]

13

How about a cron job that runs every 5 minutes, and sets a file with a
future mtime as non readable, and a past mtime as readable?

14

Hendrik H. ha scritto:

bare-bones Nginx servers upstream with embedded Python listening on
particular ports.

Yes, this is the recommended deployment.

With embedded Python I don’t see a need for some of
the standard modules, e.g. the rewriting module. That’s something the
Nginx server in front could take care of.

Right.

I’d rather see something like
Libevent (i.e. an asynchronous HTTP server engine) with embedded Python.
The core of Nginx is probably similar to Libevent with its HTTP layer.
Would be interesting to find out how Nginx core compares to Libevent
with HTTP layer in terms of performance.

A core Nginx engine with embedded Python would definitely interest me !

Then you should check http://hg.mperillo.ath.cx/nginx/mod_wsgi/!

[…]

I believe it would be difficult to compare a more general purpose
mod_wsgi with a purpose-written Python server based on libevent. I
myself have no experience with wsgi and am not in a position to compare,
but you can try.

Not sure about this, WSGI is a low level interface, well designed.
I think it is better then any other specific purpose API.

The problem is that this is a very specialized need, so you should
implement it by yourself, of rent a coder that can work on the nginx
codebase for you.

Will probably implement it myself, unless there’s someone else out there
who would be interested to give it a try. For the time being I have
myself no time whatsoever, at least not in the next two months.

I can write it, but not for free, sorry, since I don’t need this feature
and I too don’t have very free time.
It will take, for me, only a few hours of coding, however.

Also, you should give the elliot suggestion a change.
Removing the read permission bit from a file is a good solution, and
with the help of cron it should resolve your problem.

Hendrik

Manlio P.

15

Manlio,

Then you should check http://hg.mperillo.ath.cx/nginx/mod_wsgi/!

Will check it out !

Not sure about this, WSGI is a low level interface, well designed.
I think it is better then any other specific purpose API.

As I said, I have no experience with WSGI. Will try and see if I can
compare with my existing libevent-based solution.

Also, you should give the elliot suggestion a change.
Removing the read permission bit from a file is a good solution, and
with the help of cron it should resolve your problem.

Have replied to eliott’s suggestion. As I mentioned there, the solution
seems workable, but has its drawbacks. Is definitely an option for a
temporary solution though.

Hendrik

16

On Sun, 2008-02-10 at 23:14 +0100, Manlio P. wrote:

Also, you should give the elliot suggestion a change.
Removing the read permission bit from a file is a good solution, and
with the help of cron it should resolve your problem.

I think using “at” would scale better than a cron script if there are
thousands of files.

Regards,
Cliff

17

eliott,

Thanks for the suggestion.

The solution you offer seems workable for a smaller number of files, but
I’m not sure how such a cron job would scale with lots of files
(potentially thousands) added and deleted on a regular basis. Cron jobs
also require cpu time and harddisk access - even more so if I’d want a
finer resolution than 5 minutes.

I’d rather rely entirely on the filesystem itself. Just create a file
with the appropriate mtime (with a resolution of seconds rather than
minutes) and then forget about it - as nginx could take care of the
rest. Replicating / scaling would also be very straightforward.

Well, just an idea. Is probably not very useful for anyone else, so I
don’t expect much support for my feature request :slight_smile: Will probably end up
implementing it myself.

Hendrik

18

On Tue, 2008-02-12 at 07:59 +0530, Hendrik H. wrote:

Setting the read permission bit with “at” does look like a solution
worth investigating. “at” does indeed seem better than cron here, but
it still isn’t as transparent as a future file modification time. E.g.
checking or resetting the time at which a particular file should
become accessible seems rather complicated with atq / atrm / at.

While I’m still keen on setting up a filesystem solution with a
directive in Nginx, the combination of eliott’s and your suggestion
seems to be the best interim solution.

One more option (assuming you’re on Linux) would be to write a FUSE
filesystem that exposes the files you want and hides those with a future
timestamp. You could mount this over the top (or rather, alongside) of
your real FS.

This would probably be pretty simple (assuming you start with an
existing FUSE FS and modify it to your needs) although I’m sure how much
of a performance impact you’d see.

Regards,
Cliff

19

Cliff,

Thanks a lot for yet another interesting suggestion.

One more option (assuming you’re on Linux) would be to write a FUSE
filesystem that exposes the files you want and hides those with a future
timestamp. You could mount this over the top (or rather, alongside) of
your real FS.

Yes, I’m on debian. Had never looked into FUSE. Sounds like a really
interesting option. If I could use FUSE to hide files with a future
timestamp and automatically expose them at the appropriate time, then
that would be an ideal solution since it would work with nginx and
anything else that should or should not have access.

This would probably be pretty simple (assuming you start with an
existing FUSE FS and modify it to your needs) although I’m sure how much
of a performance impact you’d see.

There will obviously be some performance impact, but not a huge one I
would imagine. I can’t spare time for experiments with FUSE right now,
but will definitely look at this promising option at a later stage.

Regards,

Hendrik

20

Cliff,

Thanks for your valuable suggestion.

Setting the read permission bit with “at” does look like a solution
worth investigating. “at” does indeed seem better than cron here, but it
still isn’t as transparent as a future file modification time. E.g.
checking or resetting the time at which a particular file should become
accessible seems rather complicated with atq / atrm / at.

While I’m still keen on setting up a filesystem solution with a
directive in Nginx, the combination of eliott’s and your suggestion
seems to be the best interim solution.

Regards,
Hendrik