Nginx_uploadprogress_module v0.1

Brice_F · October 3, 2007, 6:01pm

Hi,

I’m proud to announce the availability of the first version of
nginx_uploadprogress_module (v0.1).

This module implements a progress upload ala Lighttpd mod_uploadprogress
for nginx. It monitors RFC1867 uploads sent to upstream servers and can
serve special JSON response containing information about the upload
progress on request.

This module doesn’t implement a RFC1867 parser and client code. As such
I consider this module as a temporary hack until Igor S. implements
a full client body filter chain into Nginx.

This module is delivered as is (no guarantee of any kind) under the BSD
license with the following caveats and warnings:

this software has never been tested under load. It only passed a few
lab tests.
it is not intended to be deployed on production systems.
when compiled with --with-debug, this module will produce high number
of log messages.

More information on the wiki page I just created:
http://wiki.codemongers.com/NginxHttpUploadProgressModule

The code is available as an attachment of the aforementionned page.

Installation instructions:

nginx_uploadprogress_module has been compiled/tested with Nginx 0.6.1,
0.6.12 and 0.6.13.

download the Nginx sources from http://nginx.net/ and unpack it.
build Nginx: change to the directory which contains the Nginx
sources, and run the configuration script making sure to add the path
to the nginx_uploadprogress_module sources using the --add-module
option:
$ ./configure --add-module=/path/to/nginx_uploadprogress_module/

Now you can build and install the software:
$ make

and as root:
$ make install

For usage, configuration and examples, please see the wiki page or the
README file.

There are still some missing functionnalities, like retaining
information about a finished upload for a few seconds to be able to
serve to report upload request that the upload has finished…

Anyway enjoy

Brice_F · October 3, 2007, 9:07pm

Brice- This is pretty sweet, thanks for sharing! I’ll give this some
testing and let you know any feedback I have.

Thanks
-ezra

On Oct 3, 2007, at 8:49 AM, Brice F. wrote:

progress on request.

this software has never been tested under load. It only passed a
The code is available as an attachment of the aforementionned page.
to the nginx_uploadprogress_module sources using the --add-module
README file.

There are still some missing functionnalities, like retaining
information about a finished upload for a few seconds to be able to
serve to report upload request that the upload has finished…

Anyway enjoy

Brice F. [email protected]

– Ezra Z.
– Founder & Ruby Hacker
– [email protected]
– Engine Y., Serious Rails Hosting
– (866) 518-YARD (9273)

Brice_F · October 9, 2007, 2:45pm

Brice F. wrote:

Hi,

I’m proud to announce the availability of the first version of
nginx_uploadprogress_module (v0.1).

Very cool, I’ve been waiting for a module like this one. Today I tried
to compile it but I get errors like these:
/nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:264:38:
macro “ngx_log_debug2” passed 7 arguments, but takes just 6
/nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:548:73:
macro “ngx_log_debug2” requires 6 arguments, but only 5 given

/nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:850:34:
macro “ngx_log_debug1” passed 6 arguments, but takes just 5
/nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:878:34:
macro “ngx_log_debug1” passed 6 arguments, but takes just 5

Any idea what’s going wrong? I’ve tried to build on both OpenBSD and
OSX, same errors. The configure scripts detects variadic macros:
checking for gcc variadic macros … found
checking for C99 variadic macros … found
and nginx compiles fine if I disable the uploadprogress module.

cheers,
Otto

Brice_F · October 9, 2007, 4:14pm

On Tue, 2007-10-09 at 14:34 +0200, Otto Bretz wrote:

/nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:548:73:
checking for C99 variadic macros … found
and nginx compiles fine if I disable the uploadprogress module.

I’ll check tonight (sorry I’m at work atm) what could be the cause, but
these are debug messages so you can remove the lines 264, 548, 850 and
878 of ngx_http_uploadprogress_module.c without any problem.

When I released this version last week it compiled fine on OSX (10.4,
stock Apple GCC) and a Debian Linux box (gcc 4.2.1).

I’m almost ready to ship version 0.2 with a few enhancements, so I’ll
check twice this specific issue.

Thanks for notifying me,

Brice_F · October 9, 2007, 4:14pm

On Tue, 2007-10-09 at 14:34 +0200, Otto Bretz wrote:

Brice F. wrote:

Hi,

I’m proud to announce the availability of the first version of
nginx_uploadprogress_module (v0.1).

[snip]

Any idea what’s going wrong? I’ve tried to build on both OpenBSD and
OSX, same errors. The configure scripts detects variadic macros:
checking for gcc variadic macros … found
checking for C99 variadic macros … found
and nginx compiles fine if I disable the uploadprogress module.

I fixed this issue, if you don’t want to wait the upcoming version 0.2
(available later this week), here is a patch to apply.

You can apply it by using:
patch -p0 < ngx_uploadprogress-fixes-ngx_debug_log.patch
in the nginx_uploadprogress_module directory…

Hope this helps,

Brice_F · October 10, 2007, 11:32am

Brice F. wrote:

I fixed this issue, if you don’t want to wait the upcoming version 0.2
(available later this week), here is a patch to apply.

Thanks for the patch, I tried it but now I get this:
/nginx_uploadprogress_module/ngx_http_uploadprogress_module.c: In
function ngx_http_reportuploads_handler': /home/wedge/temp/software/nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:190: warning:lz’ might be used uninitialized in this function
*** Error code 1

But I can wait for the 0.2 release, no hurry.

thanks,
Otto

Brice_F · October 9, 2007, 8:21pm

This bit me before. Passing the wrong # of arguments to a debug function
will cause the compile to fail unless “–with-debug” was passed to
./configure. Authors of modules should be sure to compile both with and
without the debug module enabled.

Evan

Brice_F · October 10, 2007, 3:26pm

On Wed, 2007-10-10 at 11:21 +0200, Otto Bretz wrote:

But I can wait for the 0.2 release, no hurry.

And it is now available
This time it should compile fine with nginx 0.6.x.

More information on the module wiki page:
http://wiki.codemongers.com/NginxHttpUploadProgressModule
The source code is still available on the wiki page.

Changes since v0.1:

fixed compilation without --with-debug. Thanks to Otto Bretz for the
original report, and thanks to Evan M. for pointing the real
culprit.
added the new paramater ‘timeout’ to directive track_upload. The
timeout parameter controls the time upload information are kept after
the upload request has finished. This allows the upload progress probe
to be able to get the “upload done” information or when an error as
occurred. Usual value is 30 seconds.
added handling of HTTP error 413 (request entity too large). Now, the
right status is sent to upload progress probe if the original upload
connection errored with this error code.

For usage, configuration and examples, please see the wiki page or the
README file.

Thanks,

Brice_F · October 11, 2007, 2:07pm

Brice F. wrote:

And it is now available
This time it should compile fine with nginx 0.6.x.

And it did! Though when I try it with the attached config I get this
message:
2007/10/11 12:05:40 [error] 1347#0: *138 open()
“/etc/nginx/html/progress” failed (2: No such file or directory),
client: my.client.tld, server: my.server.tld, URL: “/progress”, host:
“my.server.tld”, referrer: “http://my.server.tld/otto/upload.html”

To try it out I used a copy of the lighttpd upload page[1]. Is that ok?
I compiled the module with-debug as well and have attached the log of an
upload attempt.

cheers,
Otto

[1]http://upload.lighttpd.net/index.html

Brice_F · October 11, 2007, 2:54pm

On Thu, 2007-10-11 at 13:55 +0200, Otto Bretz wrote:

To try it out I used a copy of the lighttpd upload page[1]. Is that ok?
I compiled the module with-debug as well and have attached the log of an
upload attempt.

The progress report handler doesn’t yet support case-insentive progress
id parameter (I know it sucks, but that’s on my todo list for v0.3).

Apply this patch to your upload.html (provided it is the same as the
file on the webpage you referenced):

— upload.html.orig 2007-10-11 14:21:26.000000000 +0200
+++ upload.html 2007-10-11 14:21:36.000000000 +0200
@@ -40,7 +40,7 @@
if(!req) return;

 req.open("GET", "/progress", 1);

req.setRequestHeader(“X-Progress-Id”, uuid);

req.setRequestHeader(“X-Progress-ID”, uuid);
req.onreadystatechange = function () {
if (!req) {
window.clearTimeout(interval);

And then it will work fine.

If you are testing locally (ie the server is on the same computer or LAN
as the client), you won’t see the progress bar evolving, because the
bandwidth is too high.
If you try on a distant server with limited bandwidth it works fine, the
progress bar is slowly increasing.
If you want to see the progress evolving on localhost, I suggest you
reduce interval_msec in upload.html to something like 150 or 200,
otherwise the first /progress probe is sent after the upload has
finished

To test that the upload progress works fine, I’m using curl:
To initiate the upload:
curl -v -Ffilename=@file-to-upload --limit-rate 1k
“http://localhost/upload.html?X-Progress-ID=1234567890123456789019”

And to see the probe in action:
curl -v -H’X-Progress-ID: 1234567890123456789019’
“http://localhost/progress”

If you want to send me debug material, please send them privately and
not on the list, that might annoy people not interested in our
discussion…

Anyway, let me know if that’s working for you, and do not hesitate to
report any issues or improvements,

Brice_F · February 5, 2008, 4:40pm

Hi,

On Tue, 2008-02-05 at 13:14 +0100, Joerg D. wrote:

Any ideas. Any manual overrides I need to perform on IE so that it
works?

I think that’s because the upload progress module is returning the
response with a javascript mime-type. Internet Explorer which always
tries to be smarter than everyone just says it can’t do anything with
it, and can’t display it.

I don’t think it’s an issue if you just programmatically (I mean from
javascript)
try to access to the progress url.

I never tested the whole setup with IE, but if it doesn’t work let me
know, and I’ll try to reproduce it and fix it (if necessary).

Thanks,

Brice_F · February 5, 2008, 4:54pm

Hi Brice,

I have set it up according to this article:

In it I call that URL using an AJAX method. What I get back from that
are 403 errors(?) from nginx when using Internet Explorer; and then
Internet Explorer freaks out and responds with a 500, but works fine
when I use FF or Safari.

Any ideas on that?
Joerg

I don’t think it’s an issue if you just programmatically (I mean from
javascript)
try to access to the progress url.

Brice_F · February 5, 2008, 6:20pm

On Tue, 2008-02-05 at 16:54 +0100, Joerg D. wrote:

Hi Brice,

I have set it up according to this article:

http://blog.new-bamboo.co.uk/2007/11/23/upload-progress-with-nginx

Oh, I wasn’t aware of this blog article.

In it I call that URL using an AJAX method. What I get back from that
are 403 errors(?) from nginx when using Internet Explorer; and then
Internet Explorer freaks out and responds with a 500, but works fine
when I use FF or Safari.

Please activate debug (compile nginx with --with-debug, and put the
error log in debug mode) and send me privately the gzipped logs for one
request.
I’ll try to understand what happens from the log.
Also, if you are using exactly what is outlined in the article, you
might want to use a somewhat simpler example (like the one I provide in
the readme) to isolate javascript issues.

Thanks,

Brice_F · February 5, 2008, 1:14pm

Does this work for Internet Explorer?

When I do this in my FF:

http://www.mydomain.com/progress?X-Progress-ID=4a9d9c3264ccfabd2bce1aaf919cfbdd

I get back - as expexted - the following plain text:

new Object({ ‘state’ : ‘starting’ })

But, if I do this in Internet Explorer:
I get back an error alert box saying:

Internet Explorer cannot download … 9d9c3264ccfabd2bce1aaf919cfbdd
from www.mydomain.com.
Internet Explorer was not able to open this Internet site. The requested
site is either unavailable or cannot be found. Please try again later.

When I look into my nginx log files, I can see that the request returned
a 200 status.

Any ideas. Any manual overrides I need to perform on IE so that it
works?

Thanks
Joerg

Brice F. wrote:

On Thu, 2007-10-11 at 13:55 +0200, Otto Bretz wrote:

To try it out I used a copy of the lighttpd upload page[1]. Is that ok?
I compiled the module with-debug as well and have attached the log of an
upload attempt.

The progress report handler doesn’t yet support case-insentive progress
id parameter (I know it sucks, but that’s on my todo list for v0.3).

Apply this patch to your upload.html (provided it is the same as the
file on the webpage you referenced):

— upload.html.orig 2007-10-11 14:21:26.000000000 +0200
+++ upload.html 2007-10-11 14:21:36.000000000 +0200
@@ -40,7 +40,7 @@
if(!req) return;
 req.open("GET", "/progress", 1);
req.setRequestHeader(“X-Progress-Id”, uuid);

req.setRequestHeader(“X-Progress-ID”, uuid);
req.onreadystatechange = function () {
if (!req) {
window.clearTimeout(interval);

And then it will work fine.

If you are testing locally (ie the server is on the same computer or LAN
as the client), you won’t see the progress bar evolving, because the
bandwidth is too high.
If you try on a distant server with limited bandwidth it works fine, the
progress bar is slowly increasing.
If you want to see the progress evolving on localhost, I suggest you
reduce interval_msec in upload.html to something like 150 or 200,
otherwise the first /progress probe is sent after the upload has
finished

To test that the upload progress works fine, I’m using curl:
To initiate the upload:
curl -v -Ffilename=@file-to-upload --limit-rate 1k
“http://localhost/upload.html?X-Progress-ID=1234567890123456789019”

And to see the probe in action:
curl -v -H’X-Progress-ID: 1234567890123456789019’
“http://localhost/progress”

If you want to send me debug material, please send them privately and
not on the list, that might annoy people not interested in our
discussion…

Anyway, let me know if that’s working for you, and do not hesitate to
report any issues or improvements,

Brice_F · February 6, 2008, 1:37pm

Hi Brice,

thanks a lot for looking into it. I can’t access the servers at this
stage, but will be able to give you a the log file tomorrow if that is
ok. How do I get your email address?

Joerg

Please activate debug (compile nginx with --with-debug, and put the
error log in debug mode) and send me privately the gzipped logs for one
request.
I’ll try to understand what happens from the log.

Brice_F · February 6, 2008, 2:59pm

Ok cool - no worries.
I’m using the forum, so I can’t see your email address …

Just reply to this e-mail
I don’t have lots of time this week to have a look to your problem, but
I’ll try to find what’s wrong early next week (or this week-end maybe).

Brice_F · February 6, 2008, 3:30pm

On Wed, 2008-02-06 at 14:59 +0100, Joerg D. wrote:

Ok cool - no worries.
I’m using the forum, so I can’t see your email address …

Ah ok, understood. You can contact me privately at:
[email protected]

Brice_F · February 6, 2008, 2:45pm

Hi Joerg,

On Wed, 2008-02-06 at 13:37 +0100, Joerg D. wrote:

thanks a lot for looking into it. I can’t access the servers at this
stage, but will be able to give you a the log file tomorrow if that is
ok. How do I get your email address?

Just reply to this e-mail
I don’t have lots of time this week to have a look to your problem, but
I’ll try to find what’s wrong early next week (or this week-end maybe).

Thanks,

Brice_F · May 6, 2008, 11:54am

Hi,

I’m proud to announce the availability of the version 0.3 of the
nginx_upload_progress_module.

As usual the module is available as an attachment of the wiki page:
http://wiki.codemongers.com/NginxHttpUploadProgressModule

I will try to setup a public git repository of the module soon.

Changes since v0.2:

fixed crash if the upload was denied by nginx because of any error
condition (thanks to Michal Drapiewski for his detailed report)
report original upload error condition to client in the upload
progress
probe, instead of only 413 (Entity too Large).
case-insensitive checking of the X-Progress-ID header to overcome
Internet
Explorer XMLHttpRequest header lowercasing issue.

More Information:

This module implements a progress upload ala Lighttpd mod_uploadprogress
for nginx. It monitors RFC1867 uploads sent to upstream servers and can
serve special JSON response containing information about the upload
progress on request.

This module doesn’t implement a RFC1867 parser and client code. As such
I consider this module as a temporary hack until Igor S. implements
a full client body filter chain into Nginx.

This module is delivered as is (no guarantee of any kind) under the BSD
license with the following caveats and warnings:

this software has never been tested under load. It only passed a few
lab tests.
it is not intended to be deployed on production systems.
when compiled with --with-debug, this module will produce high number
of log messages.

More information on the wiki page:
http://wiki.codemongers.com/NginxHttpUploadProgressModule

The code is available as an attachment of the aforementionned page.

Installation instructions:

nginx_uploadprogress_module has been compiled/tested with Nginx 0.6.1,
0.6.12, 0.6.13 and 0.6.30. The legacy 0.5.x versions are not supported,
please upgrade your nginx installation to the current stable (0.6.30 and
more).

download the Nginx sources from http://nginx.net/ and unpack it.
build Nginx: change to the directory which contains the Nginx
sources, and run the configuration script making sure to add the path
to the nginx_uploadprogress_module sources using the --add-module
option:
$ ./configure --add-module=/path/to/nginx_uploadprogress_module/

Now you can build and install the software:
$ make

and as root:
$ make install

For usage, configuration and examples, please see the wiki page or the
README file.

Anyway enjoy