# Mongrel + RubyOnRails + FileUploads = Problems? **URL:** \ **Category:** Rails\ **Created:** [May 3, 2006, 7:08pm UTC](https://www.ruby-forum.com/t/mongrel-rubyonrails-fileuploads-problems/59474 "2006-05-03T19:08:09Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![dacbo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b9e5f3/32.png) [@dacbo](https://www.ruby-forum.com/u/dacbo)\ **Post date:** [May 3, 2006, 7:08pm UTC](https://www.ruby-forum.com/t/mongrel-rubyonrails-fileuploads-problems/59474/1 "2006-05-03T19:08:09Z")
I really like the idea of using Mongrel. But reading the FAQ, I noticed something that might be an issue. It stated: “Ruby on Rails is not thread safe so there is a synchronized block around the calls to Dispatcher.dispatch. This means that everything is threaded right before and right after Rails runs. While Rails is running there is only one controller in operation at a time.” So does this mean that if someone us uploading a large file (say around 80mb), that Rails will stop serving any other requests until the file is finished uploading? Thanks, Chris ---
**Author:** ![dacbo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b9e5f3/32.png) [@dacbo](https://www.ruby-forum.com/u/dacbo)\ **Post date:** [May 3, 2006, 8:49pm UTC](https://www.ruby-forum.com/t/mongrel-rubyonrails-fileuploads-problems/59474/2 "2006-05-03T20:49:43Z")
Yes. That’s why you need more than 1 FastCGI or Mongrel instance, behind a load balanced proxy. – – Tom M. ---
**Author:** ![dacbo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b9e5f3/32.png) [@dacbo](https://www.ruby-forum.com/u/dacbo)\ **Post date:** [May 3, 2006, 11:18pm UTC](https://www.ruby-forum.com/t/mongrel-rubyonrails-fileuploads-problems/59474/3 "2006-05-03T23:18:48Z")
Hi Chris, On 5/3/06 12:42 PM, “Chris B.” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I really like the idea of using Mongrel. But reading the FAQ, I noticed > something that might be an issue. It stated: > > “Ruby on Rails is not thread safe so there is a synchronized block > around the calls to Dispatcher.dispatch. This means that everything is > threaded right before and right after Rails runs. While Rails is running > there is only one controller in operation at a time.” Keep in mind that this is a rails issue, not a Mongrel issue. All other aspect of mongrel are as thread safe as Ruby can be. > So does this mean that if someone us uploading a large file (say around > 80mb), that Rails will stop serving any other requests until the file is > finished uploading? Yes and no. If you get the pre-release (which I’ve got to release soon or people will kill me), then you’ll get file uploads which are streamed to the disk if they are too big. You can install the pre-release like so: gem install mongrel --source=[http://mongrel.rubyforge.org/releases/](http://mongrel.rubyforge.org/releases/) But _NOT ON WINDOWS_. I tend to exclude windows folks from testing pre-releases since it’s a more complicated platform to build on. This will let many people upload files at the same time, but there’s kind of an annoying catch. Ruby uses the cgi.rb to process the multipart sections of the uploaded file. So, while Mongrel and multithread crank the upload to a temp file, once this temp file is handed to Rails it has to reprocess it all over again. This second processing is to find the multipart boundaries and uses a lot of horrible regex and backtracking. So, with large uploads you can see pretty big CPU spikes and fill up your rails processes fairly quick. So, in general, if you’re doing smallish uploads–like around 1 or 2 MB–then the pre-release is good stuff. Once the upload gets above about 50MB things start to get a little slow. At around 100MB it just sucks to be you. Zed A. Shaw > **[Zed A. shaw](https://zedshaw.com/)** > > Personal essays and writings by Zed A. Shaw. [http://mongrel.rubyforge.org/](http://mongrel.rubyforge.org/) ---
**Author:** ![dacbo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b9e5f3/32.png) [@dacbo](https://www.ruby-forum.com/u/dacbo)\ **Post date:** [May 5, 2006, 2:32am UTC](https://www.ruby-forum.com/t/mongrel-rubyonrails-fileuploads-problems/59474/4 "2006-05-05T02:32:43Z")
On 5/3/06 10:26 PM, “Bob H.” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > On May 3, 2006, at 5:13 PM, Zed S. wrote: \>\> This second processing is to find the multipart boundaries and uses \>\> a lot of \>\> horrible regex and backtracking. So, with large uploads you can \>\> see pretty \>\> big CPU spikes and fill up your rails processes fairly quick. \> \> Does the same issue exists for downloading? \> No since Mongrel or a fronting web server handles those so you get much better concurrency. Now, if your rails app is generating the content for each request then you’re screwed. Remember, it’s not just queue length but how long each request stays in the queue. If you have 10 backend Mongrels, then you have a queue length of 10 (basically). That doesn’t mean that you can only handle 10/second. Queues are really weird and kind of don’t make sense until you simulate them. You have to use some statistical distribution of time for each request to get an idea of how such a queue performs. And nothing beats straight up measurement with a tool like httperf. It’s the reality bringer. > This problem is tending to the uglier side of things, so maybe a bit > of an ugly hack won’t look so bad? What you mention below isn’t a hack, it’s actually the primary advantage of using Mongrel over fastcgi. If a Rails action is slow, you can spend a bit more effort and write a Mongrel Handler that will do it faster. It’s a little tricker–kind of bare metal–but not impossible. > What happens if you use a Mongrel handler for the file upload, > storing the file somewhere on disk, then interfere with the normal > proceedings? Maybe hacking at the request object sent to Rails by > adding a query parameter or HTTP header that says where the file was > put? Maybe a redirect with the file name instead of the file? Bingo. I actually have documentation in the queue for such a thing. What you could do is have the uploads be done with mongrel, have even an ajax progress thing done with mongrel, and when it’s all finished bounce it over to rails to complete the process. Best of all, if you did it and managed to completely avoid an “railsisms” shooting for a nice REST uplaod/progress/done process, you could even scale that up by writing a little apache or lighttpd module. Of course you’d have to be REALLY desperate to do that, but the possiblity is there. Key with this is to avoid Ruby’s sessions. What you can do is grab the cookie and parse out the rails session ID. Use that as the name of a directory where the user’s uploaded file is stored. Then, when rails is run to process the file you just have to match the current session id to that directory. Very lightweight. > Would that give you the multi-threaded upload? Yep, but remember, just making something multi-threaded doesn’t instantly solve all your problems. All computers eventually have a finite level of concurrency, and there’s always a point where just one more request can literally break the server’s back. Where this breaking point is can be 10 concurrent or 10 million. What you should do is test your current setup and see if that meets your needs. There’s no point in getting paranoid and spending the next 3 months rewriting your whole app in Mongrel only. If what you have now meets your performance requirements (you do have measurable performance requirements right?) then don’t bother. Once you see that particular Rails actions don’t meet your needs, make a plan to try something else, but don’t just grab for a Mongrel handler. Figure out how far off your Rails action is from your needs (you do have measurable performance requirements right?) and then develop a set of possible solutions that might meet those needs. Do a few small prototypes to see if your proposed solutions will meet the requirements (you do have measurable performance requirements right?). Then pick the solution that does the job. Also, after you’ve written the solution, retest everything and continually verify. Don’t believe that your own shit don’t smell. I’ve seen people spend months making something they thought was really fast only to find out it gave them no statistically significant improvement. I myself have saved tons of potentially lost development time by testing a potential solution before investing, and by testing as I go I avoid going down bad paths for no benefit. Basing your decisions on evidence is always much better than just blindly reaching for what everyone else is doing. Zed A. Shaw > **[Zed A. shaw](https://zedshaw.com/)** > > Personal essays and writings by Zed A. Shaw. [http://mongrel.rubyforge.org/](http://mongrel.rubyforge.org/) ---
**Author:** ![dacbo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b9e5f3/32.png) [@dacbo](https://www.ruby-forum.com/u/dacbo)\ **Post date:** [May 4, 2006, 4:27am UTC](https://www.ruby-forum.com/t/mongrel-rubyonrails-fileuploads-problems/59474/5 "2006-05-04T04:27:09Z")
On May 3, 2006, at 5:13 PM, Zed S. wrote: > > So does this mean that if someone us uploading a large file (say > > around > > 80mb), that Rails will stop serving any other requests until the > > file is > > finished uploading? > > Yes and no. [snip] > This second processing is to find the multipart boundaries and uses > a lot of > horrible regex and backtracking. So, with large uploads you can > see pretty > big CPU spikes and fill up your rails processes fairly quick. Does the same issue exists for downloading? This problem is tending to the uglier side of things, so maybe a bit of an ugly hack won’t look so bad? What happens if you use a Mongrel handler for the file upload, storing the file somewhere on disk, then interfere with the normal proceedings? Maybe hacking at the request object sent to Rails by adding a query parameter or HTTP header that says where the file was put? Maybe a redirect with the file name instead of the file? Would that give you the multi-threaded upload? Bob > [http://mongrel.rubyforge.org/](http://mongrel.rubyforge.org/) * * * Bob H. – blogs at \<[http://www.recursive.ca/](http://www.recursive.ca/) hutch/\> Recursive Design Inc. – [http://www.recursive.ca/](http://www.recursive.ca/) Raconteur – [http://www.raconteur.info/](http://www.raconteur.info/) xampl for Ruby – [http://rubyforge.org/projects/xampl/](http://rubyforge.org/projects/xampl/) ---
**Author:** ![dacbo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b9e5f3/32.png) [@dacbo](https://www.ruby-forum.com/u/dacbo)\ **Post date:** [May 5, 2006, 4:03am UTC](https://www.ruby-forum.com/t/mongrel-rubyonrails-fileuploads-problems/59474/6 "2006-05-05T04:03:41Z")
On May 4, 2006, at 8:27 PM, Zed S. wrote: > > > a lot of > > > content for > > > each request then you’re screwed. Doesn’t look good for me then 🙂 I’ll probably do something like below. > to get an > spend a bit > more effort and write a Mongrel Handler that will do it faster. > It’s a > little tricker–kind of bare metal–but not impossible. This is good. I’m worried about a couple of occasional things that are a lot nasty. > you could do is have the uploads be done with mongrel, have even an > you’d have > to be REALLY desperate to do that, but the possiblity is there. > > Key with this is to avoid Ruby’s sessions. What you can do is grab > the > cookie and parse out the rails session ID. Use that as the name of a > directory where the user’s uploaded file is stored. Then, when > rails is run > to process the file you just have to match the current session id > to that > directory. Very lightweight. Okay, this is in line with what I was hoping. No way am I going to be doing anything like an apache or lighttpd module. > literally break the server’s back. Where this breaking point is > can be 10 > concurrent or 10 million. I’m only worried about two concurrent users, but where one is doing something like uploading a 45 minute video. When that happens I think the number of concurrent users is going to get a bit higher 🙂 The remainder of your advice is good-all-the-time optimisation advice. Cheers, Bob > measurable performance requirements right?). Then pick the > have saved > Zed A. Shaw > [http://www.zedshaw.com/](http://www.zedshaw.com/) > [http://mongrel.rubyforge.org/](http://mongrel.rubyforge.org/) > > * * * > > Rails mailing list > removed\_email\_address@domain.invalid > [http://lists.rubyonrails.org/mailman/listinfo/rails](http://lists.rubyonrails.org/mailman/listinfo/rails) * * * Bob H. – blogs at \<[http://www.recursive.ca/](http://www.recursive.ca/) hutch/\> Recursive Design Inc. – [http://www.recursive.ca/](http://www.recursive.ca/) Raconteur – [http://www.raconteur.info/](http://www.raconteur.info/) xampl for Ruby – [http://rubyforge.org/projects/xampl/](http://rubyforge.org/projects/xampl/) ---
**Author:** ![dacbo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b9e5f3/32.png) [@dacbo](https://www.ruby-forum.com/u/dacbo)\ **Post date:** [May 5, 2006, 8:26am UTC](https://www.ruby-forum.com/t/mongrel-rubyonrails-fileuploads-problems/59474/7 "2006-05-05T08:26:55Z")
Zed S. wrote: > > What happens if you use a Mongrel handler for the file upload, > > storing the file somewhere on disk, then interfere with the normal > > proceedings? Maybe hacking at the request object sent to Rails by > > adding a query parameter or HTTP header that says where the file was > > put? Maybe a redirect with the file name instead of the file? > > Bingo. I actually have documentation in the queue for such a thing. > What > you could do is have the uploads be done with mongrel, have even an ajax > progress thing done with mongrel, and when it’s all finished bounce it > over > to rails to complete the process. > > Best of all, if you did it and managed to completely avoid an > “railsisms” > shooting for a nice REST uplaod/progress/done process, you could even > scale > that up by writing a little apache or lighttpd module. Of course you’d > have > to be REALLY desperate to do that, but the possiblity is there. Call me desperate… I’ve been struggling for quite a while on getting a proxy working between IIS (Windows Web Server) and Mongrel (actually - most time was spend on SCGI :-S). The thing I have been fighting was file uploads as there is a file encoding issues between the IIS and Mongrel/SCGI. This seemed to be unsolvable on the .NET part of things - don’t ask - so I actually parked the whole project and decided to spend some time learning Rails rather than fighting IIS and .NET on this issue. I you have any information on the Mongrel & Rails part this would be sweet. Especially any ideaâ??s on making this as â??transparentâ?? as possible from a Rails point of view would be very helpful. My Ruby/Rails skills are too limited at this point to tackle this. /Boris ---
**Author:** ![dacbo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b9e5f3/32.png) [@dacbo](https://www.ruby-forum.com/u/dacbo)\ **Post date:** [May 5, 2006, 10:45pm UTC](https://www.ruby-forum.com/t/mongrel-rubyonrails-fileuploads-problems/59474/8 "2006-05-05T22:45:40Z")
Brian H. wrote: > Boris: > I will have a solution to thisdocumented very shortly… I’m just > finishing > up the documentation for my rails reverse\_proxy\_fix plugin. > > You’ll be able to use IIS to receive the requests but you’ll use > ISAPIRewrite (the paid version only) to forward requests to Mongrel, and > my > plugin to handle the URL rewriting (with some limitations… the URL > rewriting only works if the URLs were written with Rails helpers.) > > Docs are coming soon which include a simple (IIS to Mongrel) and an > advanced > (IIS to Lighttpd to Mongrel cluster) > > I’ve been promising this for some time but things keep changing so fast > it’s > hard to keep up. However I’m hoping to have something on my site at the > end > of the day. Cool stuff Brian. I was planning to create a sort of mod\_proxy for IIS, so include the possibility to use multiple Mongrel instances and do not have a need for a commercial isapi plugin. ---
**Author:** ![dacbo](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/b9e5f3/32.png) [@dacbo](https://www.ruby-forum.com/u/dacbo)\ **Post date:** [May 5, 2006, 2:45pm UTC](https://www.ruby-forum.com/t/mongrel-rubyonrails-fileuploads-problems/59474/9 "2006-05-05T14:45:28Z")
Boris: I will have a solution to thisdocumented very shortly… I’m just finishing up the documentation for my rails reverse\_proxy\_fix plugin. You’ll be able to use IIS to receive the requests but you’ll use ISAPIRewrite (the paid version only) to forward requests to Mongrel, and my plugin to handle the URL rewriting (with some limitations… the URL rewriting only works if the URLs were written with Rails helpers.) Docs are coming soon which include a simple (IIS to Mongrel) and an advanced (IIS to Lighttpd to Mongrel cluster) I’ve been promising this for some time but things keep changing so fast it’s hard to keep up. However I’m hoping to have something on my site at the end of the day.