Hi, I'm proud to announce the availability of the first version of nginx_uploadprogress_module (v0.1). This module implements a progress upload ala Lighttpd mod_uploadprogress for nginx. It monitors RFC1867 uploads sent to upstream servers and can serve special JSON response containing information about the upload progress on request. This module doesn't implement a RFC1867 parser and client code. As such I consider this module as a temporary hack until Igor Sysoev implements a full client body filter chain into Nginx. This module is delivered as is (no guarantee of any kind) under the BSD license with the following caveats and warnings: * this software has never been tested under load. It only passed a few lab tests. * it is not intended to be deployed on *production systems*. * when compiled with --with-debug, this module will produce high number of log messages. More information on the wiki page I just created: http://wiki.codemongers.com/NginxHttpUploadProgressModule The code is available as an attachment of the aforementionned page. Installation instructions: nginx_uploadprogress_module has been compiled/tested with Nginx 0.6.1, 0.6.12 and 0.6.13. 1) download the Nginx sources from http://nginx.net/ and unpack it. 2) build Nginx: change to the directory which contains the Nginx sources, and run the configuration script making sure to add the path to the nginx_uploadprogress_module sources using the --add-module option: $ ./configure --add-module=/path/to/nginx_uploadprogress_module/ Now you can build and install the software: $ make and as root: $ make install For usage, configuration and examples, please see the wiki page or the README file. There are still some missing functionnalities, like retaining information about a finished upload for a few seconds to be able to serve to report upload request that the upload has finished... Anyway enjoy :-)
on 03.10.2007 18:01
on 03.10.2007 21:07
Brice- This is pretty sweet, thanks for sharing! I'll give this some testing and let you know any feedback I have. Thanks -ezra On Oct 3, 2007, at 8:49 AM, Brice Figureau wrote: > progress on request. > * this software has never been tested under load. It only passed a > The code is available as an attachment of the aforementionned page. > to the nginx_uploadprogress_module sources using the --add-module > README file. > > There are still some missing functionnalities, like retaining > information about a finished upload for a few seconds to be able to > serve to report upload request that the upload has finished... > > Anyway enjoy :-) > -- > Brice Figureau <brice+nginx@daysofwonder.com> > > -- Ezra Zygmuntowicz -- Founder & Ruby Hacker -- ez@engineyard.com -- Engine Yard, Serious Rails Hosting -- (866) 518-YARD (9273)
on 09.10.2007 14:45
Brice Figureau wrote: > Hi, > > I'm proud to announce the availability of the first version of > nginx_uploadprogress_module (v0.1). Very cool, I've been waiting for a module like this one. Today I tried to compile it but I get errors like these: /nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:264:38: macro "ngx_log_debug2" passed 7 arguments, but takes just 6 /nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:548:73: macro "ngx_log_debug2" requires 6 arguments, but only 5 given /nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:850:34: macro "ngx_log_debug1" passed 6 arguments, but takes just 5 /nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:878:34: macro "ngx_log_debug1" passed 6 arguments, but takes just 5 Any idea what's going wrong? I've tried to build on both OpenBSD and OSX, same errors. The configure scripts detects variadic macros: checking for gcc variadic macros ... found checking for C99 variadic macros ... found and nginx compiles fine if I disable the uploadprogress module. cheers, Otto
on 09.10.2007 16:14
On Tue, 2007-10-09 at 14:34 +0200, Otto Bretz wrote: > /nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:548:73: > checking for C99 variadic macros ... found > and nginx compiles fine if I disable the uploadprogress module. I'll check tonight (sorry I'm at work atm) what could be the cause, but these are debug messages so you can remove the lines 264, 548, 850 and 878 of ngx_http_uploadprogress_module.c without any problem. When I released this version last week it compiled fine on OSX (10.4, stock Apple GCC) and a Debian Linux box (gcc 4.2.1). I'm almost ready to ship version 0.2 with a few enhancements, so I'll check twice this specific issue. Thanks for notifying me,
on 09.10.2007 16:14
On Tue, 2007-10-09 at 14:34 +0200, Otto Bretz wrote: > Brice Figureau wrote: > > Hi, > > > > I'm proud to announce the availability of the first version of > > nginx_uploadprogress_module (v0.1). > [snip] > Any idea what's going wrong? I've tried to build on both OpenBSD and > OSX, same errors. The configure scripts detects variadic macros: > checking for gcc variadic macros ... found > checking for C99 variadic macros ... found > and nginx compiles fine if I disable the uploadprogress module. I fixed this issue, if you don't want to wait the upcoming version 0.2 (available later this week), here is a patch to apply. You can apply it by using: patch -p0 < ngx_uploadprogress-fixes-ngx_debug_log.patch in the nginx_uploadprogress_module directory... Hope this helps,
on 09.10.2007 20:21
This bit me before. Passing the wrong # of arguments to a debug function will cause the compile to fail unless "--with-debug" was passed to ./configure. Authors of modules should be sure to compile both with and without the debug module enabled. Evan
on 10.10.2007 11:32
Brice Figureau wrote: > I fixed this issue, if you don't want to wait the upcoming version 0.2 > (available later this week), here is a patch to apply. Thanks for the patch, I tried it but now I get this: /nginx_uploadprogress_module/ngx_http_uploadprogress_module.c: In function `ngx_http_reportuploads_handler': /home/wedge/temp/software/nginx_uploadprogress_module/ngx_http_uploadprogress_module.c:190: warning: `lz' might be used uninitialized in this function *** Error code 1 But I can wait for the 0.2 release, no hurry. :) thanks, Otto
on 10.10.2007 15:26
On Wed, 2007-10-10 at 11:21 +0200, Otto Bretz wrote: > But I can wait for the 0.2 release, no hurry. :) And it is now available :-) This time it should compile fine with nginx 0.6.x. More information on the module wiki page: http://wiki.codemongers.com/NginxHttpUploadProgressModule The source code is still available on the wiki page. Changes since v0.1: * fixed compilation without --with-debug. Thanks to Otto Bretz for the original report, and thanks to Evan Miller for pointing the real culprit. * added the new paramater 'timeout' to directive track_upload. The timeout parameter controls the time upload information are kept after the upload request has finished. This allows the upload progress probe to be able to get the "upload done" information or when an error as occurred. Usual value is 30 seconds. * added handling of HTTP error 413 (request entity too large). Now, the right status is sent to upload progress probe if the original upload connection errored with this error code. For usage, configuration and examples, please see the wiki page or the README file. Thanks,
on 11.10.2007 14:07
Brice Figureau wrote: > And it is now available :-) > This time it should compile fine with nginx 0.6.x. And it did! Though when I try it with the attached config I get this message: 2007/10/11 12:05:40 [error] 1347#0: *138 open() "/etc/nginx/html/progress" failed (2: No such file or directory), client: my.client.tld, server: my.server.tld, URL: "/progress", host: "my.server.tld", referrer: "http://my.server.tld/otto/upload.html" To try it out I used a copy of the lighttpd upload page[1]. Is that ok? I compiled the module with-debug as well and have attached the log of an upload attempt. cheers, Otto [1]http://upload.lighttpd.net/index.html
on 11.10.2007 14:54
On Thu, 2007-10-11 at 13:55 +0200, Otto Bretz wrote: > > To try it out I used a copy of the lighttpd upload page[1]. Is that ok? > I compiled the module with-debug as well and have attached the log of an > upload attempt. The progress report handler doesn't yet support case-insentive progress id parameter (I know it sucks, but that's on my todo list for v0.3). Apply this patch to your upload.html (provided it is the same as the file on the webpage you referenced): --- upload.html.orig 2007-10-11 14:21:26.000000000 +0200 +++ upload.html 2007-10-11 14:21:36.000000000 +0200 @@ -40,7 +40,7 @@ if(!req) return; req.open("GET", "/progress", 1); - req.setRequestHeader("X-Progress-Id", uuid); + req.setRequestHeader("X-Progress-ID", uuid); req.onreadystatechange = function () { if (!req) { window.clearTimeout(interval); And then it will work fine. If you are testing locally (ie the server is on the same computer or LAN as the client), you won't see the progress bar evolving, because the bandwidth is too high. If you try on a distant server with limited bandwidth it works fine, the progress bar is slowly increasing. If you want to see the progress evolving on localhost, I suggest you reduce interval_msec in upload.html to something like 150 or 200, otherwise the first /progress probe is sent after the upload has finished :-) To test that the upload progress works fine, I'm using curl: To initiate the upload: curl -v -Ffilename=@file-to-upload --limit-rate 1k "http://localhost/upload.html?X-Progress-ID=1234567890123456789019" And to see the probe in action: curl -v -H'X-Progress-ID: 1234567890123456789019' "http://localhost/progress" If you want to send me debug material, please send them privately and not on the list, that might annoy people not interested in our discussion... Anyway, let me know if that's working for you, and do not hesitate to report any issues or improvements,
on 05.02.2008 13:14
Does this work for Internet Explorer? When I do this in my FF: http://www.mydomain.com/progress?X-Progress-ID=4a9d9c3264ccfabd2bce1aaf919cfbdd I get back - as expexted - the following plain text: new Object({ 'state' : 'starting' }) But, if I do this in Internet Explorer: I get back an error alert box saying: Internet Explorer cannot download .... 9d9c3264ccfabd2bce1aaf919cfbdd from www.mydomain.com. Internet Explorer was not able to open this Internet site. The requested site is either unavailable or cannot be found. Please try again later. When I look into my nginx log files, I can see that the request returned a 200 status. Any ideas. Any manual overrides I need to perform on IE so that it works? Thanks Joerg Brice Figureau wrote: > On Thu, 2007-10-11 at 13:55 +0200, Otto Bretz wrote: >> >> To try it out I used a copy of the lighttpd upload page[1]. Is that ok? >> I compiled the module with-debug as well and have attached the log of an >> upload attempt. > > The progress report handler doesn't yet support case-insentive progress > id parameter (I know it sucks, but that's on my todo list for v0.3). > > Apply this patch to your upload.html (provided it is the same as the > file on the webpage you referenced): > > --- upload.html.orig 2007-10-11 14:21:26.000000000 +0200 > +++ upload.html 2007-10-11 14:21:36.000000000 +0200 > @@ -40,7 +40,7 @@ > if(!req) return; > > req.open("GET", "/progress", 1); > - req.setRequestHeader("X-Progress-Id", uuid); > + req.setRequestHeader("X-Progress-ID", uuid); > req.onreadystatechange = function () { > if (!req) { > window.clearTimeout(interval); > > > And then it will work fine. > > If you are testing locally (ie the server is on the same computer or LAN > as the client), you won't see the progress bar evolving, because the > bandwidth is too high. > If you try on a distant server with limited bandwidth it works fine, the > progress bar is slowly increasing. > If you want to see the progress evolving on localhost, I suggest you > reduce interval_msec in upload.html to something like 150 or 200, > otherwise the first /progress probe is sent after the upload has > finished :-) > > To test that the upload progress works fine, I'm using curl: > To initiate the upload: > curl -v -Ffilename=@file-to-upload --limit-rate 1k > "http://localhost/upload.html?X-Progress-ID=1234567890123456789019" > > And to see the probe in action: > curl -v -H'X-Progress-ID: 1234567890123456789019' > "http://localhost/progress" > > If you want to send me debug material, please send them privately and > not on the list, that might annoy people not interested in our > discussion... > > Anyway, let me know if that's working for you, and do not hesitate to > report any issues or improvements,
on 05.02.2008 16:40
Hi, On Tue, 2008-02-05 at 13:14 +0100, Joerg Diekmann wrote: > > > Any ideas. Any manual overrides I need to perform on IE so that it > works? I think that's because the upload progress module is returning the response with a javascript mime-type. Internet Explorer which always tries to be smarter than everyone just says it can't do anything with it, and can't display it. I don't think it's an issue if you just programmatically (I mean from javascript) try to access to the progress url. I never tested the whole setup with IE, but if it doesn't work let me know, and I'll try to reproduce it and fix it (if necessary). Thanks,
on 05.02.2008 16:54
Hi Brice, I have set it up according to this article: http://blog.new-bamboo.co.uk/2007/11/23/upload-progress-with-nginx In it I call that URL using an AJAX method. What I get back from that are 403 errors(?) from nginx when using Internet Explorer; and then Internet Explorer freaks out and responds with a 500, but works fine when I use FF or Safari. Any ideas on that? Joerg > > I don't think it's an issue if you just programmatically (I mean from > javascript) > try to access to the progress url. >
on 05.02.2008 18:20
On Tue, 2008-02-05 at 16:54 +0100, Joerg Diekmann wrote: > Hi Brice, > > I have set it up according to this article: > > http://blog.new-bamboo.co.uk/2007/11/23/upload-progress-with-nginx Oh, I wasn't aware of this blog article. > In it I call that URL using an AJAX method. What I get back from that > are 403 errors(?) from nginx when using Internet Explorer; and then > Internet Explorer freaks out and responds with a 500, but works fine > when I use FF or Safari. Please activate debug (compile nginx with --with-debug, and put the error log in debug mode) and send me privately the gzipped logs for one request. I'll try to understand what happens from the log. Also, if you are using exactly what is outlined in the article, you might want to use a somewhat simpler example (like the one I provide in the readme) to isolate javascript issues. Thanks,
on 06.02.2008 13:37
Hi Brice, thanks a lot for looking into it. I can't access the servers at this stage, but will be able to give you a the log file tomorrow if that is ok. How do I get your email address? Joerg > Please activate debug (compile nginx with --with-debug, and put the > error log in debug mode) and send me privately the gzipped logs for one > request. > I'll try to understand what happens from the log.
on 06.02.2008 14:45
Hi Joerg, On Wed, 2008-02-06 at 13:37 +0100, Joerg Diekmann wrote: > thanks a lot for looking into it. I can't access the servers at this > stage, but will be able to give you a the log file tomorrow if that is > ok. How do I get your email address? Just reply to this e-mail :-) I don't have lots of time this week to have a look to your problem, but I'll try to find what's wrong early next week (or this week-end maybe). Thanks,
on 06.02.2008 14:59
Ok cool - no worries. I'm using the forum, so I can't see your email address ... > Just reply to this e-mail :-) > I don't have lots of time this week to have a look to your problem, but > I'll try to find what's wrong early next week (or this week-end maybe).
on 06.02.2008 15:30
On Wed, 2008-02-06 at 14:59 +0100, Joerg Diekmann wrote: > Ok cool - no worries. > I'm using the forum, so I can't see your email address ... Ah ok, understood. You can contact me privately at: brice+nginx@daysofwonder.com
on 06.05.2008 11:54
Hi, I'm proud to announce the availability of the version 0.3 of the nginx_upload_progress_module. As usual the module is available as an attachment of the wiki page: http://wiki.codemongers.com/NginxHttpUploadProgressModule I will try to setup a public git repository of the module soon. Changes since v0.2: =================== * fixed crash if the upload was denied by nginx because of any error condition (thanks to Michal Drapiewski for his detailed report) * report original upload error condition to client in the upload progress probe, instead of only 413 (Entity too Large). * case-insensitive checking of the X-Progress-ID header to overcome Internet Explorer XMLHttpRequest header lowercasing issue. More Information: ================= This module implements a progress upload ala Lighttpd mod_uploadprogress for nginx. It monitors RFC1867 uploads sent to upstream servers and can serve special JSON response containing information about the upload progress on request. This module doesn't implement a RFC1867 parser and client code. As such I consider this module as a temporary hack until Igor Sysoev implements a full client body filter chain into Nginx. This module is delivered as is (no guarantee of any kind) under the BSD license with the following caveats and warnings: * this software has never been tested under load. It only passed a few lab tests. * it is not intended to be deployed on *production systems*. * when compiled with --with-debug, this module will produce high number of log messages. More information on the wiki page: http://wiki.codemongers.com/NginxHttpUploadProgressModule The code is available as an attachment of the aforementionned page. Installation instructions: ========================== nginx_uploadprogress_module has been compiled/tested with Nginx 0.6.1, 0.6.12, 0.6.13 and 0.6.30. The legacy 0.5.x versions are not supported, please upgrade your nginx installation to the current stable (0.6.30 and more). 1) download the Nginx sources from http://nginx.net/ and unpack it. 2) build Nginx: change to the directory which contains the Nginx sources, and run the configuration script making sure to add the path to the nginx_uploadprogress_module sources using the --add-module option: $ ./configure --add-module=/path/to/nginx_uploadprogress_module/ Now you can build and install the software: $ make and as root: $ make install For usage, configuration and examples, please see the wiki page or the README file. Anyway enjoy :-)