# Mirroring large files over HTTP **URL:** \ **Category:** Ruby\ **Created:** [September 30, 2008, 7:35pm UTC](https://www.ruby-forum.com/t/mirroring-large-files-over-http/149877 "2008-09-30T19:35:20Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![lheese](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/a183cd/32.png) [@lheese](https://www.ruby-forum.com/u/lheese)\ **Post date:** [September 30, 2008, 7:35pm UTC](https://www.ruby-forum.com/t/mirroring-large-files-over-http/149877/1 "2008-09-30T19:35:20Z")
I’m working on a script where I want to download large files off a remote web server and store on a local filesystem. At the moment I’m using code like this: require ‘open-uri’ open(filename, ‘w’) do |file| file.write(open(remote\_url).read) end I assume this will read the complete content of the remote file into memory before writing it to the local file. If that assumption is correct, what is the best/easiest way to do a buffered piecemeal fetch/store? I’ve looked at the net/http library but haven’t found anything in there that looks relevant to this. – Lars H. “If anyone disagrees with anything I say, I am quite prepared not only to retract it, but also to deny under oath that I ever said it.” -Tom Lehrer ---
**Author:** ![lheese](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/a183cd/32.png) [@lheese](https://www.ruby-forum.com/u/lheese)\ **Post date:** [October 2, 2008, 10:05am UTC](https://www.ruby-forum.com/t/mirroring-large-files-over-http/149877/2 "2008-10-02T10:05:35Z")
- Lars H. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > end > > I assume this will read the complete content of the remote file into > memory before writing it to the local file. If that assumption is > correct, what is the best/easiest way to do a buffered piecemeal > fetch/store? I’ve looked at the net/http library but haven’t found > anything in there that looks relevant to this. Turns out the OpenURI module is indeed fetching the remote resource in segments and storing to a temporary file. However, my code above will read the complete contents of that file into memory before writing it back out to another file. By inspecting the OpenURI source code I’ve learned that this is how it’s done (sans proxy handling, error handling etc.): uri = URI.parse(url) http = Net::HTTP.new(uri.host, uri.port) http.request\_get(uri.path) do |response| open(filename, ‘w’) do |file| response.read\_body do |segment| file.write(segment) end end end I’m a little surprised not to find any convenience method in the standard libraries doing all this for me, though. – Lars H. “If anyone disagrees with anything I say, I am quite prepared not only to retract it, but also to deny under oath that I ever said it.” -Tom Lehrer ---
**Author:** ![lheese](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/a183cd/32.png) [@lheese](https://www.ruby-forum.com/u/lheese)\ **Post date:** [October 2, 2008, 10:38am UTC](https://www.ruby-forum.com/t/mirroring-large-files-over-http/149877/3 "2008-10-02T10:38:29Z")
On Oct 2, 2008, at 01:03 AM, Lars H. wrote: > Turns out the OpenURI module is indeed fetching the remote resource > in segments and storing to a temporary file. However, my code above > will read the complete contents of that file into memory before > writing it back out to another file. > > I’m a little surprised not to find any convenience method in the > standard > libraries doing all this for me, though. Why? It’s all of one line: output.write input.read(16384) until input.eof? ---
**Author:** ![lheese](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/a183cd/32.png) [@lheese](https://www.ruby-forum.com/u/lheese)\ **Post date:** [October 6, 2008, 11:35pm UTC](https://www.ruby-forum.com/t/mirroring-large-files-over-http/149877/4 "2008-10-06T23:35:31Z")
- Lars H. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > standard > > > libraries doing all this for me, though. > > > > Why? It’s all of one line: > > > > output.write input.read(16384) until input.eof? > > Nice enough, but one will need a bit more than that singke line to do the > whole operation from start to finish. > > I was thinking more of something like SomeClass.mirror(url, filename). Today I came across the the curb¹ gem (Ruby bindings for libcurl) while reading a blog posting² about net/http performance, and this gem provides a convenient class method that does exactly what I want: require ‘curb’ Curl::Easy.download(url, filename) It also provides lots of other nice stuff, so I will definitely look into using this one for my future HTTP client needs. [1] [http://curb.rubyforge.org/](http://curb.rubyforge.org/) [2] [http://apocryph.org/analysis\_ruby\_18x\_http\_client\_performance](http://apocryph.org/analysis_ruby_18x_http_client_performance) – Lars H. “If anyone disagrees with anything I say, I am quite prepared not only to retract it, but also to deny under oath that I ever said it.” -Tom Lehrer ---
**Author:** ![lheese](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/a183cd/32.png) [@lheese](https://www.ruby-forum.com/u/lheese)\ **Post date:** [October 2, 2008, 11:35am UTC](https://www.ruby-forum.com/t/mirroring-large-files-over-http/149877/5 "2008-10-02T11:35:23Z")
- Eric H. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Why? It’s all of one line: > > output.write input.read(16384) until input.eof? Nice enough, but one will need a bit more than that singke line to do the whole operation from start to finish. I was thinking more of something like SomeClass.mirror(url, filename). – Lars H. “If anyone disagrees with anything I say, I am quite prepared not only to retract it, but also to deny under oath that I ever said it.” -Tom Lehrer