Failed to allocate memory while downloading large files


#1

hii ,
I uploaded 1 GB of file to filesystem,but now when i trying to
download this 1 GB of file ,i am getting failed to allocate memory.
am running mongrel as server in development mode.

NoMemoryError (failed to allocate memory):
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:in
<<' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:inexpand_cache_key’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/response.rb:118:in
etag=' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/response.rb:148:inhandle_conditional_get!’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/response.rb:138:in
prepare!' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/base.rb:532:insend_response’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/base.rb:526:in
process_without_filters' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/filters.rb:606:inprocess_without_session_ma
gement_support’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/session_management.rb:134:in
process' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/base.rb:392:inprocess’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:183:in
handle_request' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:110:indispatch_unlocked’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:123:in
dispatch' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:122:insynchronize’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:122:in
dispatch' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:132:indispatch_cgi’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:39:in
dispatch' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/bin/../lib/mongrel/rails.rb:76:inprocess’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/bin/…/lib/mongrel/rails.rb:74:in
synchronize' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/bin/../lib/mongrel/rails.rb:74:inprocess’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:159:in
process_client' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:158:ineach’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:158:in
process_client' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:285:inrun’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:285:in
initialize' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:285:innew’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:285:in
run' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:268:ininitialize’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:268:in
new' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel.rb:268:inrun’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel/configurator.rb:282:in
run' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel/configurator.rb:281:ineach’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel/configurator.rb:281:in
run' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/bin/mongrel_rails:128:inrun’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/lib/mongrel/command.rb:212:in
run' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/mongrel-1.1.2-x86-mswin32/bin/mongrel_rails:281 C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:142:inload_without_new_cons
nt_marking’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:142:in
load' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:521:innew_constants_in’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:142:in
load' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/rails-2.2.2/lib/commands/servers/mongrel.rb:64 C:/InstantRails-2.0-win/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:ingem_original_require’
C:/InstantRails-2.0-win/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in
require' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:153:inrequire’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:521:in
new_constants_in' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/dependencies.rb:153:inrequire’
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/rails-2.2.2/lib/commands/server.rb:49
C:/InstantRails-2.0-win/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in
gem_original_require' C:/InstantRails-2.0-win/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:inrequire’
script/server:3

ndered
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/templates/rescues/_trace
(250.0ms)


#2

On Mon, Sep 13, 2010 at 7:33 AM, Amit T. removed_email_address@domain.invalid
wrote:

hii ,
I uploaded 1 GB of file to filesystem,but now when i trying to
download this 1 GB of file ,i am getting failed to allocate memory.
am running mongrel as server in development mode.

NoMemoryError (failed to allocate memory):
C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:in
<<' C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:inexpand_cache_key’

This has nothing to do with Mongrel.

From the backtrace it seems you’re trying to cache all the file
contents in memory.

What is the code you’re using to serve the file? Mongrel is quite
capable to serve any file from public folder without any Rails code
interfering it.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry


#3

Luis L. wrote:

On Mon, Sep 13, 2010 at 7:33 AM, Amit T. removed_email_address@domain.invalid
wrote:

hii ,
? ? ?I uploaded 1 GB of file to filesystem,but now when i trying to
download this 1 GB of file ,i am getting failed to allocate memory.
am running mongrel as server in development mode.

NoMemoryError (failed to allocate memory):
?C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:in
<<' ?C:/InstantRails-2.0-win/ruby/lib/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/cache.rb:60:inexpand_cache_key’

This has nothing to do with Mongrel.

From the backtrace it seems you’re trying to cache all the file
contents in memory.

What is the code you’re using to serve the file? Mongrel is quite
capable to serve any file from public folder without any Rails code
interfering it.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹

Luis ,this is how am downloading my files

send_data(@data,:filename => @containerformat.name+extension,
:disposition => ‘attachment’)
but i am getting error failed to allocate memory for large files ,for
small files it’s working fine

and one thing more i observed while uploading large files , whenever i
upload data more than 2gb from IE OR from mozila ,both restarts but for
google chorme its working fine???


#4

On Tue, Sep 14, 2010 at 2:49 AM, Amit T. removed_email_address@domain.invalid
wrote:

Luis ,this is how am downloading my files

send_data(@data,:filename => @containerformat.name+extension,
:disposition => ‘attachment’)
but i am getting error failed to allocate memory for large files ,for
small files it’s working fine

Because send_data will try to load the whole file in memory.

http://apidock.com/rails/ActionController/Streaming/send_data

Since the file is not dynamically generated, serve it directly, don’t
put it behind Rails.

and one thing more i observed while uploading large files , whenever i
upload data more than 2gb from IE OR from mozila ,both restarts but for
google chorme its working fine???

Neither IE or Mozilla do chunked uploads. Chrome does and mongrels
properly streams the chunked upload to a temporary file.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry


#5

On Tue, Sep 14, 2010 at 9:21 AM, Amit T. removed_email_address@domain.invalid
wrote:

thanks luis
but could you please explain how i serve it directly

Place the file inside public, provide the URL to the file as a link,
your browser will start download once the file is served.

send_data is not recommended for big file serving.

and for my
application i can’nt use IE because as you said it don’t do chunked
upload.

Can’t help you on that, please research about it.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry


#6

Luis L. wrote:

On Tue, Sep 14, 2010 at 9:21 AM, Amit T. removed_email_address@domain.invalid
wrote:

thanks luis
but could you please explain how i serve ?it directly

Place the file inside public, provide the URL to the file as a link,
your browser will start download once the file is served.

send_data is not recommended for big file serving.

and for my
application i can’nt use ?IE because as you said it don’t ?do chunked
upload.

Can’t help you on that, please research about it.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹

sorry luis but i didn’t get you
i uploaded my file in filesystem than how can i place file in public
folder
could you please give some example??


#7

Luis L. wrote:

On Tue, Sep 14, 2010 at 2:49 AM, Amit T. removed_email_address@domain.invalid
wrote:

Luis ,this is how am downloading my files

send_data(@data,:filename => @containerformat.name+extension,
? ? ? ?:disposition => ‘attachment’)
but i am getting error failed to allocate memory for large files ,for
small files it’s working fine

Because send_data will try to load the whole file in memory.

http://apidock.com/rails/ActionController/Streaming/send_data

Since the file is not dynamically generated, serve it directly, don’t
put it behind Rails.

and one thing more i observed while uploading large files , whenever i
upload data more than 2gb from IE OR from mozila ,both restarts but for
google chorme its working fine???

Neither IE or Mozilla do chunked uploads. Chrome does and mongrels
properly streams the chunked upload to a temporary file.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹

thanks luis
but could you please explain how i serve it directly and for my
application i can’nt use IE because as you said it don’t do chunked
upload.

thanks again


#8

On Tue, Sep 14, 2010 at 9:39 AM, Amit T. removed_email_address@domain.invalid
wrote:

sorry luis but i didn’t get you
i uploaded my file in filesystem than how can i place file in public
folder
could you please give some example??

Are you using a plugin to manage your uploads? How are you saving the
file?

Take the temporary file and use File.move or FileUtils.mv, wonder what
are you doing right now for that.

Please use google to search, there are plenty of file uploading
tutorials about Ruby on Rails on the internet covering “do from
scratch” or using gem/plugins.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry


#9

Amit T. wrote:

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹

luis i put file in public folder but now i have four copies of same file
one is mongel temp file ,2nd cgi file,3rd in filesyste and 4th in
public
for large don’t you think its not efficient approch
??


#10

Luis L. wrote:

On Tue, Sep 14, 2010 at 9:39 AM, Amit T. removed_email_address@domain.invalid
wrote:

sorry luis but i didn’t get you
i uploaded my file in filesystem than how ?can ?i place file in public
folder
could you please give some example??

Are you using a plugin to manage your uploads? How are you saving the
file?

Take the temporary file and use File.move or FileUtils.mv, wonder what
are you doing right now for that.

Please use google to search, there are plenty of file uploading
tutorials about Ruby on Rails on the internet covering “do from
scratch” or using gem/plugins.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹

this is how am uploading
def upload(params)

    path = get_path(params)
    File.open(path, "wb") do |f|
    while buff= params[:location].read(4096)

    f.write(buff)
    end
    end

end

def get_path(params)
name = params[:location].original_filename
directory = “//192.168.247.77/Smruti/streams
path = File.join(directory, name)
return path
end

def copyFile(uploaded_file)
@uploaded_file = uploaded_file
directory = “//192.168.147.17/Smruti/streams
#url_dir = “data”
url_dir = “//192.168.147.17/Smruti/streams
name = File.basename(@uploaded_file)
path = File.join(directory, name)
if (!FileTest.exist?(path))
FileUtils.copy @uploaded_file, path
else
return “duplicate”
end

        url = File.join(url_dir, name)
        return url

  end

end
and after copying it filesystem


#11

Luis L. wrote:

On Tue, Sep 14, 2010 at 9:50 AM, Amit T. removed_email_address@domain.invalid
wrote:

? ? ? ?end
end

That is copying the file every 4k for you, you can simply File.move
form the temporay filename to the final name in your remote location.

It is expected you dispose the temporary file when you’re done with it.

Please see how proper file uploading and processing is done with Ruby on
Rails.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹

Thanks luis
you said right ,i have to check how uploading takes place…
could you suggest some documnet ??


#12

On Tue, Sep 14, 2010 at 9:50 AM, Amit T. removed_email_address@domain.invalid
wrote:

   end

end

That is copying the file every 4k for you, you can simply File.move
form the temporay filename to the final name in your remote location.

It is expected you dispose the temporary file when you’re done with it.

Please see how proper file uploading and processing is done with Ruby on
Rails.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry


#13

Luis L. wrote:

On Tue, Sep 14, 2010 at 10:18 AM, Amit T. removed_email_address@domain.invalid
wrote:

Thanks luis
you said right ,i have to check how uploading takes place…
could you suggest some documnet ??

Google is your friend, “Rails File upload”

http://www.rubyinside.com/rails-file-uploading-101-406.html

And others.

If the files are located in a remote server that is not in public
directory, then you will not have alternative but use something like
apache to serve map the files to the remote disk and serve them.

But please, avoid send_data at any cost, and even the file
reading/writing you’re doing, because with the size you’re working,
you’re just slowing your application with IO operations.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹

luis ,but do n’t have option other than send_data and one thing i can do
is while downloading large files i can do it in chunks but i only
download 4096 byte ,this is my code

File.open(@containerformat.location,“rb”)do
|f| @data = f.read(4096) end

  ext = File.extname(@containerformat.streamName)



    send_data(@data,:filename => name+extension,
    :disposition => 'attachment')

end
what happening here…


#14

On Tue, Sep 14, 2010 at 11:14 AM, Amit T. removed_email_address@domain.invalid
wrote:

luis ,but do n’t have option other than send_data and one thing i can do
is while downloading large files i can do it in chunks but i only
download 4096 byte ,this is my code

please read about streaming data without loading all the file contents.

You will need to either use rack directly or create a mongrel handler
to serve these files outside rails.

Rails can only deliver one full request meaning it will load the full
file before completing.


Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry


#15

On Tue, Sep 14, 2010 at 10:18 AM, Amit T. removed_email_address@domain.invalid
wrote:

Thanks luis
you said right ,i have to check how uploading takes place…
could you suggest some documnet ??

Google is your friend, “Rails File upload”

http://www.rubyinside.com/rails-file-uploading-101-406.html

And others.

If the files are located in a remote server that is not in public
directory, then you will not have alternative but use something like
apache to serve map the files to the remote disk and serve them.

But please, avoid send_data at any cost, and even the file
reading/writing you’re doing, because with the size you’re working,
you’re just slowing your application with IO operations.

Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry


#16

Sorry i think am taking lot of your time


#17

Luis L. wrote:

On Tue, Sep 14, 2010 at 11:14 AM, Amit T. removed_email_address@domain.invalid
wrote:

luis ,but do n’t have option other than send_data and one thing i can do
is while downloading large files i can do it in chunks but i only
download 4096 byte ,this is my code

please read about streaming data without loading all the file contents.

You will need to either use rack directly or create a mongrel handler
to serve these files outside rails.

Rails can only deliver one full request meaning it will load the full
file before completing.


Luis L.
AREA 17

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupé²¹

  File.open(@containerformat.location,"rb") do |f|
  pos=f.tell
  while @buffer = f.read(4096)

  ext = File.extname(@containerformat.streamName)

  if ext == ''
    extension = File.extname(@containerformat.location)



    #send_to_web_app(pos, buffer)

    send_data(@buffer,:filename => @containerformat.name+extension,
    :disposition => 'attachment')

  else

  send_data(@buffer,:filename => @containerformat.streamName,
    :disposition => 'attachment')
  pos=f.tell
  end

end

now am doing it like above and trying to download 284mb of file
but only 211 bytes of file being downloaded…
why??


#18

Luis L. wrote:

please read about streaming data without loading all the file contents.

You will need to either use rack directly or create a mongrel handler
to serve these files outside rails.

Rails can only deliver one full request meaning it will load the full
file before completing.

Notice that now you can use send_file which is now (Rails 3) implemented
through a rack middleware and includes X-sendfile headers.


#19

Daniel Cadenas wrote:

Luis L. wrote:

please read about streaming data without loading all the file contents.

You will need to either use rack directly or create a mongrel handler
to serve these files outside rails.

Rails can only deliver one full request meaning it will load the full
file before completing.

Notice that now you can use send_file which is now (Rails 3) implemented
through a rack middleware and includes X-sendfile headers.

yaa Daniel i already imlemented send_file method and now i downloaded 7
gb of file without fail …
The only problem am facing is how do i delete
mongrel temporary after upload large files…