i want to upload a very very very large file(>=4G),i have googled and
tested two days,but still not found a better solution.
at first ,i just find some rails plugins ,such as Acts As
Attachment,Attachment Fu,File Column,but they seemingly just deal with
small file,just like image.
then i find some other ways,using merb, Mongrel,Mongrel Handlers ,
Sinatra Rack Metal ,Mod_Rails aka Passenger and so on .
i test it using webrick and mongrel,when i upload a file which size is
700M,the webrick die,and the mongrel cost 56587ms to complete.
because of limiting of the test environment,i never test others. i
didn’t known which is better?
can anyone help me ,give me suggesion?
tks!
Hi, I don’t know if there are differing opinions on this, but seems to
me that apps that use such large files will write a file manager
(desktop based app or at least activex) that monitor or possibly
divide such files into smaller pieces. Just seems even with a
webserver that would handle such a direct upload you are going to be
subject to so many extraneous factors which could affect success of
such.
On 7/22/10, Qin Q. [email protected] wrote:
didn’t known which is better?
[email protected].
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en.
–
Sent from my mobile device
You are looking too deeply into your app stack. This kind of file
uploads needs to be handled by the webserver itself. All modern
webservers have a way to stream files from a user onto the disc (this
way you don’t have memory issues as only the current chunk is in memory
at any give time) and then notify your app once the upload is completed.
For nginx, you want to use the nginx_upload_module:
http://www.grid.net.ru/nginx/upload.en.html
I’m not sure right off hand how you’d configure Apache for this.
Hope that helps.
Jason
DK wrote:
Hi, I don’t know if there are differing opinions on this, but seems to
me that apps that use such large files will write a file manager
(desktop based app or at least activex) that monitor or possibly
divide such files into smaller pieces.
Not necessarily Look at Dropbox (though they do use a Flash upload
tool).
Just seems even with a
webserver that would handle such a direct upload you are going to be
subject to so many extraneous factors which could affect success of
such.
Theoretically, uploading a big file should be just like uploading a
small one. The problem is that there’s more time for the connection to
be interrupted.
Marnen Laibow-Koser wrote:
DK wrote:
Hi, I don’t know if there are differing opinions on this, but seems to
me that apps that use such large files will write a file manager
(desktop based app or at least activex) that monitor or possibly
divide such files into smaller pieces.Not necessarily Look at Dropbox (though they do use a Flash upload
tool).
Dropbox is what he is describing. The start you off by having you
install a filesystem extension.
–
Alan G. - [email protected] - http://twitter.com/bigeasy
Hi,
maybe I’m wrong, but I think that you cannot upload files greater than
1Gb with native browser fonctions.
Jerome F. wrote:
Hi,
maybe I’m wrong, but I think that you cannot upload files greater than
1Gb with native browser fonctions.
Why not? I’ve never heard of this limitation. Is it in the HTTP
protocol itself, or what?
–Â
Marnen Laibow-Koser
http://www.marnen.org
[email protected]
Sent from my iPhone
Marnen Laibow-Koser wrote:
install a filesystem extension.
But you don’t have to. I exchange large files (hundreds of MB, not 4
GB) on Dropbox all the time, but I’ve never installed their desktop
software. It’s only for convenience AFAIK.
I think it is the other way around. Their getting started link says
“Download”. They want you to get the download software. The HTTP upload
is for convenience. I’m offering this as an opinion.
The OP wants to upload more than 4GB and I wouldn’t trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload. If the OP is building a web application,
they are going to have to find an alternative. If they are building an
application with a web front end, maybe they can use SFTP as a file
transfer client for large files, and the web UI to manage the files once
they have been uploaded.
Basically, Qin Q., you are right in noting that HTTP upload is only
good for reasonable sized uploads, whatever size reasonable is at the
time that you read this. (Images, documents, etc.)
–
Alan G. - [email protected] - http://twitter.com/bigeasy
Alan G. wrote:
Marnen Laibow-Koser wrote:
DK wrote:
Hi, I don’t know if there are differing opinions on this, but seems to
me that apps that use such large files will write a file manager
(desktop based app or at least activex) that monitor or possibly
divide such files into smaller pieces.Not necessarily Look at Dropbox (though they do use a Flash upload
tool).Dropbox is what he is describing. The start you off by having you
install a filesystem extension.
But you don’t have to. I exchange large files (hundreds of MB, not 4
GB) on Dropbox all the time, but I’ve never installed their desktop
software. It’s only for convenience AFAIK.
–
Alan G. - [email protected] - http://twitter.com/bigeasy
Best,
–Â
Marnen Laibow-Koser
http://www.marnen.org
[email protected]
Sent from my iPhone
Marnen Laibow-Koser wrote:
Alan G. wrote:
Marnen Laibow-Koser wrote:
install a filesystem extension.
But you don’t have to. I exchange large files (hundreds of MB, not 4
GB) on Dropbox all the time, but I’ve never installed their desktop
software. It’s only for convenience AFAIK.I think it is the other way around. Their getting started link says
“Download”. They want you to get the download software. The HTTP upload
is for convenience. I’m offering this as an opinion.However you want to look at it, the point is that the HTTP upload option
(or whatever the Flash side of the tool does) exists.The OP wants to upload more than 4GB and I wouldn’t trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.I agree with you on that.
If the OP is building a web application,
they are going to have to find an alternative. If they are building an
application with a web front end, maybe they can use SFTP as a file
transfer client for large files, and the web UI to manage the files once
they have been uploaded.Basically, Qin Q., you are right in noting that HTTP upload is only
good for reasonable sized uploads, whatever size reasonable is at the
time that you read this. (Images, documents, etc.)Yup.
Best,
Marnen Laibow-Koser
http://www.marnen.org
[email protected]
Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can’t browse others
at the same time .
I met with that when I upload a large file the computer system halted
under ubuntu 10.04 LTS, it brings me a lot of trouble.
besides ,when I use mongrel ,the web server will create temp files
,which several times of original file size,in other word ,it needs more
disk or memory to complete it.
maybe I should have a try with nginx firstly …
Alan G. wrote:
Marnen Laibow-Koser wrote:
install a filesystem extension.
But you don’t have to. I exchange large files (hundreds of MB, not 4
GB) on Dropbox all the time, but I’ve never installed their desktop
software. It’s only for convenience AFAIK.I think it is the other way around. Their getting started link says
“Download”. They want you to get the download software. The HTTP upload
is for convenience. I’m offering this as an opinion.
However you want to look at it, the point is that the HTTP upload option
(or whatever the Flash side of the tool does) exists.
The OP wants to upload more than 4GB and I wouldn’t trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.
I agree with you on that.
If the OP is building a web application,
they are going to have to find an alternative. If they are building an
application with a web front end, maybe they can use SFTP as a file
transfer client for large files, and the web UI to manage the files once
they have been uploaded.Basically, Qin Q., you are right in noting that HTTP upload is only
good for reasonable sized uploads, whatever size reasonable is at the
time that you read this. (Images, documents, etc.)
Yup.
Best,
Marnen Laibow-Koser
http://www.marnen.org
[email protected]
FYI
Flash/Flex (AS3 only) now have a File IO Class to handle uploads and
downloads, I use it to upload images to member areas in my Rails App
with no problem, but the file sizes are no bigger than about 250K -
never tried anything larger, although you do get progress information
sent back to the event handler in flash so you can provide a progress
information, and also handle any errors during the upload/download.
Qin Q. wrote:
Marnen Laibow-Koser wrote:
Alan G. wrote:
Marnen Laibow-Koser wrote:
However you want to look at it, the point is that the HTTP upload option
(or whatever the Flash side of the tool does) exists.
The OP wants to upload more than 4GB and I wouldn’t trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.
I agree with you on that.
Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can’t browse others
at the same time .
And then they think it has hung. And then they go away.
I met with that when I upload a large file the computer system halted
under ubuntu 10.04 LTS, it brings me a lot of trouble.
besides ,when I use mongrel ,the web server will create temp files
,which several times of original file size,in other word ,it needs more
disk or memory to complete it.
maybe I should have a try with nginx firstly …
This isn’t going to get any better for you. Are you expecting that by
switching to nginx you’re going to be able upload 4GB with nothing more
than a screen flicker when the success page is returned?
Why not follow up on by looking at the Flash upload control at Dropbox
that Marnen Laibow-Koser describes? See how fast it can upload 250MB,
what it does if you interrupt the upload, etc.
–
Alan G. - [email protected] - http://twitter.com/bigeasy
paul h wrote:
On Jul 23, 4:11 am, Alan G. [email protected] wrote:
Qin Q. wrote:
Marnen Laibow-Koser wrote:
Alan G. wrote:
Marnen Laibow-Koser wrote:
However you want to look at it, the point is that the HTTP upload option
(or whatever the Flash side of the tool does) exists.
The OP wants to upload more than 4GB and I wouldn’t trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.
Why not follow up on by looking at the Flash upload control at Dropbox
that Marnen Laibow-Koser describes? See how fast it can upload 250MB,
what it does if you interrupt the upload, etc.
Flash/Flex (AS3 only) now have a File IO Class to handle uploads and
downloads, I use it to upload images to member areas in my Rails App
with no problem, but the file sizes are no bigger than about 250K -
never tried anything larger, although you do get progress information
sent back to the event handler in flash so you can provide a progress
information, and also handle any errors during the upload/download.
Can it restart a failed download do you know?
–
Alan G. - [email protected] - http://twitter.com/bigeasy
Alan G. wrote:
Qin Q. wrote:
Marnen Laibow-Koser wrote:
Alan G. wrote:
Marnen Laibow-Koser wrote:
However you want to look at it, the point is that the HTTP upload option
(or whatever the Flash side of the tool does) exists.The OP wants to upload more than 4GB and I wouldn’t trust HTTP to
transfer 4GB from a client to a server on a regular basis. There is no
way to resume a failed upload.I agree with you on that.
Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can’t browse others
at the same time .And then they think it has hung. And then they go away.
I met with that when I upload a large file the computer system halted
under ubuntu 10.04 LTS, it brings me a lot of trouble.
besides ,when I use mongrel ,the web server will create temp files
,which several times of original file size,in other word ,it needs more
disk or memory to complete it.
maybe I should have a try with nginx firstly …This isn’t going to get any better for you. Are you expecting that by
switching to nginx you’re going to be able upload 4GB with nothing more
than a screen flicker when the success page is returned?Why not follow up on by looking at the Flash upload control at Dropbox
that Marnen Laibow-Koser describes? See how fast it can upload 250MB,
what it does if you interrupt the upload, etc.–
Alan G. - [email protected] - http://twitter.com/bigeasy
I found that the dropbox is just a Cross-platform client,and need to
connect a dropbox account. because of the network environment , we
didn’t be allowed to do it. I download the software in other web
site,but I could’t connect successfully. And I found that the total
capacity is 2G. So,it is still not a good idea for me.
dropbox is on behalf of network disks. there are so many productions
like this,but the capacity all is small.
So , I doublt that it is possible for browser to upload the very very
large file ? and I whether can to use a web browser to upload files via
FTP ?
Qin Q. wrote:
I agree with you on that.
maybe I should have a try with nginx firstly …
This isn’t going to get any better for you. Are you expecting that by
switching to nginx you’re going to be able upload 4GB with nothing more
than a screen flicker when the success page is returned?Why not follow up on by looking at the Flash upload control at Dropbox
that Marnen Laibow-Koser describes? See how fast it can upload 250MB,
what it does if you interrupt the upload, etc.
I found that the dropbox is just a Cross-platform client,and need to
connect a dropbox account. because of the network environment , we
didn’t be allowed to do it. I download the software in other web
site,but I could’t connect successfully. And I found that the total
capacity is 2G. So,it is still not a good idea for me.
dropbox is on behalf of network disks. there are so many productions
like this,but the capacity all is small.
So , I doublt that it is possible for browser to upload the very very
large file ? and I whether can to use a web browser to upload files via
FTP ?
Okay. Seems like a (human) language barrier. I appreciate your taking
the time to investigate Dropbox as has people here have suggested. Er,
thank you for looking at Dropbox like I asked.
We said look at Dropbox because Dropbox uses a Flash control for upload.
You could use a Flash control for upload. Maybe. I’m not sure because I
do not know Flash. Another poster said Flash has a File I/O library and
a network library. So, maybe there is a Flash based upload control out
there, or maybe you could write one.
Maybe this Flash control can upload 4 GB while showing a progress bar
and time estimate. Maybe this Flash control can restart an upload that
is interrupted.
You would have to write something on the server side that understood how
to handle the restart. I believe that restart failed upload is
important. Your user will be so unhappy if he accidentally closes a
window at 3.9GB and has to start over.
Finally, upload of 4GB just using the HTML file form control is
technically possible, and you could iframe it so that page is
responsive, but the failed upload thing is going to be a bad user
experience.
–
Alan G. - [email protected] - http://twitter.com/bigeasy
Alan G. wrote:
Qin Q. wrote:
I agree with you on that.
Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can’t browse others
at the same time .
And then they think it has hung. And then they go away.I found that the dropbox is just a Cross-platform client,and need to
connect a dropbox account. because of the network environment , we
didn’t be allowed to do it. I download the software in other web
site,but I could’t connect successfully. And I found that the total
capacity is 2G. So,it is still not a good idea for me.
dropbox is on behalf of network disks. there are so many productions
like this,but the capacity all is small.
So , I doublt that it is possible for browser to upload the very very
large file ? and I whether can to use a web browser to upload files via
FTP ?So, just to be as clear as possible.
I suggested you look at Dropbox to see how they used Flash. Not because
they would be a possible 3rd party library or add-on. Just see how they
do a big upload. See if they offer restart of a failed upload. Have a
look at how they solved the same problem.–
Alan G. - [email protected] - http://twitter.com/bigeasy
Thank you very much for your kind help.I see your excellent proposal.
I will learn some knowledge about the realization principles of dropbox
concerning how to upload the large file and how to use Flash to achieve
the goal firstly.
thank you !
Qin Q. wrote:
I agree with you on that.
Sincerely thank all answer the question firstly.
Upload large file will block the app AFAIK,and user can’t browse others
at the same time .
And then they think it has hung. And then they go away.
I found that the dropbox is just a Cross-platform client,and need to
connect a dropbox account. because of the network environment , we
didn’t be allowed to do it. I download the software in other web
site,but I could’t connect successfully. And I found that the total
capacity is 2G. So,it is still not a good idea for me.
dropbox is on behalf of network disks. there are so many productions
like this,but the capacity all is small.
So , I doublt that it is possible for browser to upload the very very
large file ? and I whether can to use a web browser to upload files via
FTP ?
So, just to be as clear as possible.
I suggested you look at Dropbox to see how they used Flash. Not because
they would be a possible 3rd party library or add-on. Just see how they
do a big upload. See if they offer restart of a failed upload. Have a
look at how they solved the same problem.
–
Alan G. - [email protected] - http://twitter.com/bigeasy
If what you mean by restart a failed download is to continue appending
data to the local file, continuing from where the fail happened, then
AFAIK the answer would be no.
But you can handle the errors and provide feedback to the user. It may
also be possible to automatically restart the download after a
failure, but I would have to check that out. In order to begin an
upload or download, the FlashPlayer security requires user activation
by either selecting the file to upload from the local file system or
the location and file name to save a download to on the local file
system - however, once the reference to the files (local or remote)
has been obtained, it may be possible to begin it again after a
failure, and just notify the user of a major error if it fails x
number of times consecutively.
I forgot before that I also use Flash to download .mov files (private
site I set up for my brother to distribute movies of his new baby girl
to the rest of the family) the largest so far is 93MB and it downloads
fine - dunno about upload though, because I wrote an FTP client for
him to do the uploads.
I’ll look into the above after the weekend (busy diving), and post the
results to clarify, I’m pretty new to this group, and had a little
help from it so far, so would gladly give something back when I can 
The Rails side is simple, and the Flash side isn’t tough - examples
follow:
Rails
Controller
def upload
ModelClass.new.save_image( params )
end
Model
def save_image params
data = params[:Filedata]
name = params[:Filename]
path = File.join( “public” , “users” , params[:id] ,
“images” , name)
# params[:Filedata] and params[:Filename] are sent automatically
from Flash
# params[:id] is appended to the url:
www.mysite.com/controller_name/method_name_in_model/id
File.open(path,'wb') do |file|
file.puts data.read
end
end
Flash/Flex
The place to look for anyone interested would be at the FileReference
class.
Create a FileReference instance and call the browse() method with a
FileFilter if necessary to filter to specific file types.
Once the user has selected a file to upload (or a file location and
name if downloading) the select Event is triggered, and you can then
send the file to the Rails Controller.
Here’s some simple AS3 code:
private var _fileReference:FileReference = new FileReference();
private var _imageFilter:FileFilter = new FileFilter(“Images (.jpg,
.jpeg, .gif, .png)", ".jpg;.jpeg;.gif;.png”);
private var _uploadURL:String = “http://www.mysite.com/controller/
model_method/id_if_required”;
private function upload():void
{
_fileReference.addEventListener(Event.SELECT, select); // Add
additional event handlers to handle progress, security errors, IO
errors and of course file complete etc
_fileReference.browse([_imageFilter]); // You can add as many
different filters as required within the array parameter
}
private function select( event:Event ):void
{
_fileReference = FileReference(event.target);
_fileReference.upload(new URLRequest(_uploadURL));
}
HTH
Paul