# Nginx-lua and nginx upload module **URL:** \ **Category:** NGINX\ **Created:** [November 26, 2011, 8:37am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262 "2011-11-26T08:37:21Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\ **Post date:** [November 26, 2011, 8:37am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/1 "2011-11-26T08:37:21Z")
Hi all, I’m hoping to write an nginx-lua script which handles file uploads from the nginx upload module. I can do this fine using an external backend and the upload\_pass directive, but the lua module has no functionality for parsing the multipart form data that this generates. I’ve also tried accessing the $upload\_file\_name, $upload\_tmp\_path etc. variables generated by the upload module directly, but I don’t seem to be able to access these outside of the upload\_set\_form\_field directive. Is there any way to get the contents of those variables into a lua handler in a form that can be easily parsed? Posted at Nginx Forum: > **[nginx-lua and nginx upload module](https://forum.nginx.org/read.php?2,219085,219085#msg-219085)** > > Hi all, I'm hoping to write an nginx-lua script which handles file uploads from the nginx upload module. I can do this fine using an external backend and the upload\_pass directive, but the lua module has no functionality for parsing the multipart... ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 9:08am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/2 "2011-11-26T09:08:52Z")
On Sat, Nov 26, 2011 at 3:36 PM, strawman [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Hi all, > > I’m hoping to write an nginx-lua script which handles file uploads from > the nginx upload module. I can do this fine using an external backend > and the upload\_pass directive, but the lua module has no functionality > for parsing the multipart form data that this generates. I’ve also > tried accessing the $upload\_file\_name, $upload\_tmp\_path etc. variables > generated by the upload module directly, but I don’t seem to be able to > access these outside of the upload\_set\_form\_field directive. Well, there will be a streaming request body reading API for ngx\_lua soon. For instance: ``` local data, err = ngx.req.recv_body_data() ``` Atop this, we can easily build a multipart streaming parser in pure Lua and no longer rely on the ngx\_upload module to read and process the request body. (The UI of ngx\_upload is not so friendly for Lua scripting and it’ll be technically difficult to make them work together without lots of line noises.) Someone is already willing to sponsor this work but I’m too busy to implement this lately 😛 Regards, -agentzh ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 10:17am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/3 "2011-11-26T10:17:33Z")
On 26 November 2011 10:36, strawman [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Is there any way to get the contents of those variables into a lua > handler in a form that can be easily parsed? Hi There, I hit the multipart form limitation of the ngx\_lua module recently and wrote a lua module to do the job. Basically I combined a number of functions from various cgilua modules and added a few of my own to create a single module. I have attached a copy and the instructions for calling it are added as comments at the top of the module. It will handle both url encoded and multipart encoded forms and return the post data as a lua key/value pairs ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 10:47am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/4 "2011-11-26T10:47:21Z")
On Sat, Nov 26, 2011 at 5:16 PM, Nginx U. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > It will handle both url encoded and multipart encoded forms and return > the post data as a lua key/value pairs This approach certainly works but it requires the request body to be hold completely in memory, which may be unacceptable for big file uploading. Regards, -agentzh ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 11:17am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/5 "2011-11-26T11:17:50Z")
On 26 November 2011 12:16, Nginx U. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I have attached a copy and the instructions for calling it are added > as comments at the top of the module. > > It will handle both url encoded and multipart encoded forms and return > the post data as a lua key/value pairs Use this version instead ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 11:07am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/6 "2011-11-26T11:07:57Z")
On 26 November 2011 12:46, agentzh [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > This approach certainly works but it requires the request body to be > hold completely in memory, which may be unacceptable for big file > uploading. Correct. This is why there is a default 2mb “maxinput” flag. Hopefully a directive to handle multipart encoded forms will be added to ngx\_lua by the developer soon 🙂 ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 2:11pm UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/7 "2011-11-26T14:11:36Z")
Thanks for the swift replies, unfortunately I’m still having trouble getting it to play nicely with the upload module. Here’s the config I’m using: [##### truncated vhost config:server { location /upload { upload\_pass - Pastebin.com](http://pastebin.com/GsrYHwj0) It seems as though the request body isn’t being read correctly; ngx.req.get\_body\_data() works as expected if I run it from inside /upload and make a multipart post to it, but it only returns the first line of the body data when run from inside @handler. echo\_request\_body works fine either way. Here’s what I get when posting to it with the above config: $ curl -F file=@TDXBN.gif [http://localhost/upload](http://localhost/upload) - post defs: content\_type: multipart/form-data; boundary=----------------------------ff78acc43147 maxinput: 2097152 content\_length: 358 maxfilesize: 1048576 args: - post args: - request body: ------------------------------ff78acc43147 On a slightly different tack, I noticed one of the release announcements for ngx\_echo showed this directive being used: echo\_subrequest PUT /some\_upstream/$upload\_file\_md5 -f $upload\_tmp\_path Is there something in particular you need to do to get access to $upload\_tmp\_path etc. outside of the upload\_set\_form\_field directive? I’ve tried hacking this into working with echo\_subrequest, but if i try to use any of the $upload\_\* variables anywhere outside of that directive then it doesn’t work at all. Posted at Nginx Forum: > **[Re: nginx-lua and nginx upload module](https://forum.nginx.org/read.php?2,219085,219107#msg-219107)** > > Hi all, I'm hoping to write an nginx-lua script which handles file uploads from the nginx upload module. I can do this fine using an external backend and the upload\_pass directive, but the lua module has no functionality for parsing the multipart... ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 2:25pm UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/8 "2011-11-26T14:25:16Z")
On 26 November 2011 16:16, agentzh [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Sigh. I really hope that ngx\_upload could be implemented as a rewrite > or access phase handler rather than a content handler, then we can > trivially combine with content\_by\_lua in the same location, for > example, and no longer need to do any kind of internal redirections > that clear everything including nginx variables and module contexts. For your info, I believe agentzh may be refering, obliquely, to the restriction in the last line here: [GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/HttpLuaModule#content_by_lua) (which also applies to content\_by\_lua\_file). ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 2:16pm UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/9 "2011-11-26T14:16:36Z")
On Sat, Nov 26, 2011 at 9:10 PM, strawman [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Sigh. I really hope that ngx\_upload could be implemented as a rewrite or access phase handler rather than a content handler, then we can trivially combine with content\_by\_lua in the same location, for example, and no longer need to do any kind of internal redirections that clear everything including nginx variables and module contexts. Maybe Valery can work on an alternative UI for his ngx\_upload module? 😉 Regards, -agentzh ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 10:18pm UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/10 "2011-11-26T22:18:31Z")
On 26 November 2011 16:10, strawman [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > echo\_request\_body works fine either way. If it was possible to access echo\_request\_body from ngx\_lua, then could possibly should solve your issue as the form parser should be able to use this with some modifications. ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 9:14pm UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/11 "2011-11-26T21:14:43Z")
The limitation on the phase that is used to process uploads at the moment comes from nginx itself and not from upload module. If you describe what sort of API you need from upload module, I will be able to implement it. – Best regards, Valery K. ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 26, 2011, 11:09pm UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/12 "2011-11-26T23:09:12Z")
Hi, Valery! On Sun, Nov 27, 2011 at 4:14 AM, Valery K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > The limitation on the phase that is used to process uploads at the moment comes > from nginx itself and not from upload module. Happily we no longer have such limitations for nginx 0.8.54+ 🙂 > If you describe what sort of API you need from upload module, I will be able to > implement it. Great! Just a quick example from my head: ``` location /upload { upload_in_access_phase; proxy_pass http://backend; } ``` That is, no longer introducing an internal location here for internal redirections 😉 Thanks! -agentzh ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 27, 2011, 3:02pm UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/13 "2011-11-27T15:02:05Z")
On 26 November 2011 10:36, strawman [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Is there any way to get the contents of those variables into a lua > handler in a form that can be easily parsed? Hi, I have been intrigued by your question and been looking at various possible solutions but then it struck me as odd as to why you would want to do this in the first place. A bit of trying to step out of a potential XY Problem situation. What is it exactly that you are trying to achieve? ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 28, 2011, 10:52am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/14 "2011-11-28T10:52:13Z")
On Mon, Nov 28, 2011 at 5:45 PM, Valery K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > Happily we no longer have such limitations for nginx 0.8.54+ 🙂 > > Explain? Now we can happily read and process the request body in both rewrite and access phase handlers in nginx 0.8.54+. And ngx\_lua is already doing that for both the lua\_need\_request\_body config directive and the “ngx.req.read\_body()” Lua API. > Doesn’t look like API. We are talking about being able to use accelerated > uploads feature from lua, aren’t we? No, I think we’re talking about making ngx\_upload easier to work with other nginx modules, especially with those registering a content handler like ngx\_lua, ngx\_echo, ngx\_proxy, and ngx\_fastcgi. The whole point here is to make ngx\_upload read and process the request body in an earlier phase like “rewrite” and “access” phases, such that we can preserve the content handler for other modules and can eliminate internal redirects altogether to reduce runtime cost. Sorry for the confusions in my previous emails 🙂 Thanks! -agentzh ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 28, 2011, 11:22am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/15 "2011-11-28T11:22:44Z")
----- agentzh [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > doing that for both the lua\_need\_request\_body config directive and the > “ngx.req.read\_body()” Lua API. I think we’ve been able to do that even before. But that’s not point. The problem is not in what particular phase upload module will run, but that it must run whenever some module tries to read the request body. This kind of customisation is not supported by nginx yet. > > > > can eliminate internal redirects altogether to reduce runtime cost. > > Sorry for the confusions in my previous emails 🙂 Again, the point is not in what particular phase it will run… And by the way, the costs of internal redirects are negligible. – Regards, Valery K. ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 28, 2011, 10:46am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/16 "2011-11-28T10:46:36Z")
----- agentzh [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Hi, Valery! > > On Sun, Nov 27, 2011 at 4:14 AM, Valery K. > [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > The limitation on the phase that is used to process uploads at the moment > > comes from nginx itself and not from upload module. > > Happily we no longer have such limitations for nginx 0.8.54+ 🙂 Explain? > That is, no longer introducing an internal location here for internal > redirections 😉 Doesn’t look like API. We are talking about being able to use accelerated uploads feature from lua, aren’t we? – Regards, Valery K. ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 29, 2011, 7:01am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/17 "2011-11-29T07:01:46Z")
On Mon, Nov 28, 2011 at 6:22 PM, Valery K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I think we’ve been able to do that even before. Sadly for nginx 0.8.41 ~ 0.8.53, “IO interruptions” are explicitly prohibited in rewrite phase handlers and for exactly the same reason your ngx\_eval module does not work at all for these versions of Nginx 🙂 > But that’s not point. The problem is not in what particular phase upload module > will run, but that it must run whenever some module tries to read the request > body. This kind of customisation is not supported by nginx yet. True. We definitely need an input filter mechanism for Nginx. I’ve cc’d Andrew A.. Maybe he can help make this happen in the Nginx core 😉 > Again, the point is not in what particular phase it will run… And by the way, > the costs of internal redirects are negligible. Not really if the user has a lot of regex style locations defined in his nginx.conf. For now, the nginx core does not (yet) utilize the PCRE JIT feature for its pattern matching, so it does consume quite a few CPU cycles for complicated location patterns 😉 And…if we _can_ eliminate the cost of internal redirects altogether, why shouldn’t we make that happen? 😉 Also, I believe it can simplify the user config file greatly and does not cause variable scope problems (like $uri and $args’ scope). I know there is a upload\_pass\_args directive to forward the original URI query args, but if the $args scope issue does not exist in the first place, we do not need this config directive at all. BTW, I notice that if ngx\_upload’s special variables like $upload\_file\_md5 are used in a wrong context, it will simply crash the nginx worker (segmentation faults) due to null module context pointer access (at least for the latest 2.2.0 release). Maybe it’s more reasonable to check the ctx pointer and print out a helpful error message in functions like ngx\_http\_upload\_md5\_variable? Best, -agentzh ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 29, 2011, 11:30am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/18 "2011-11-29T11:30:42Z")
----- agentzh [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > But that’s not point. The problem is not in what particular phase upload > > module will run, but that it must run whenever some module tries to read the > > request body. This kind of customisation is not supported by nginx yet. > > Not really if the user has a lot of regex style locations defined in > > his nginx.conf. For now, the nginx core does not (yet) utilize the > > PCRE JIT feature for its pattern matching, so it does consume quite a > > few CPU cycles for complicated location patterns 😉 And…if we _can_ > > eliminate the cost of internal redirects altogether, why shouldn’t we > > make that happen? 😉 Did you mean “quite a lot”? > access (at least for the latest 2.2.0 release). > I’ll add a check for that. > Maybe it’s more > reasonable to check the ctx pointer and print out a helpful error > message in functions like ngx\_http\_upload\_md5\_variable? – Regards, Valery K. ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 29, 2011, 11:50am UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/19 "2011-11-29T11:50:38Z")
On Tue, Nov 29, 2011 at 6:29 PM, Valery K. \>\> > > Not really if the user has a lot of regex style locations defined in > > his nginx.conf. For now, the nginx core does not (yet) utilize the > > PCRE JIT feature for its pattern matching, so it does consume quite a > > few CPU cycles for complicated location patterns 😉 And…if we _can_ > > eliminate the cost of internal redirects altogether, why shouldn’t we > > make that happen? 😉 > > Did you mean “quite a lot”? I didn’t say it costs quite a lot in the real world settings. I just said we should avoid wasting CPU cycles wherever possible (just like what Marcus C. originally told me two years ago) 🙂 Regards, -agentzh ---
**Author:** ![strawman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/74df32/32.png) [@strawman](https://www.ruby-forum.com/u/strawman)\ **Post date:** [November 29, 2011, 3:59pm UTC](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262/20 "2011-11-29T15:59:31Z")
Hi, > > But that’s not point. The problem is not in what particular phase upload module > > will run, but that it must run whenever some module tries to read the request > > body. This kind of customisation is not supported by nginx yet. > > True. We definitely need an input filter mechanism for Nginx. I’ve > cc’d Andrew A.. Maybe he can help make this happen in the Nginx > core 😉 I’m not a programmer (at least not anymore), so I personally can’t implement this 🙂 Input filters are planned in the next major release of nginx, where the core architecture would allow that. > his nginx.conf. For now, the nginx core does not (yet) utilize the > PCRE JIT feature for its pattern matching, so it does consume quite a Btw, we have an almost ready implementation for PCRE JIT and it’ll soon be released in the dev train. [Next page](https://www.ruby-forum.com/t/nginx-lua-and-nginx-upload-module/214262.md?page=2)