# Efficient file downloading

**URL:** <https://www.ruby-forum.com/t/efficient-file-downloading/129855>\
**Category:** Ruby\
**Created:** [February 22, 2008, 2:54am UTC](https://www.ruby-forum.com/t/efficient-file-downloading/129855 "2008-02-22T02:54:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\
**Post date:** [February 22, 2008, 2:54am UTC](https://www.ruby-forum.com/t/efficient-file-downloading/129855/1 "2008-02-22T02:54:40Z")

</div>

Hello,

I’m using open-uri to download files using a buffer. It seems very  
inefficient in terms of resource usage (CPU is ~10-20% in usage).

If possible, I’d like some suggestions for downloading a file which  
names the outputted file the same as the URL, and does not actually  
write if the file comes out to a 404 (or some other exception hits).

Current code:  
BUFFER\_SIZE=4096  
def download(url)  
from = open(url)  
if (buffer = from.read(BUFFER\_SIZE))  
puts “Downloading #{url}”  
File.open(url.split(’/’).last, ‘wb’) do |file|  
begin  
file.write(buffer)  
end while (buffer = from.read(BUFFER\_SIZE))  
end  
end  
end

---

<div class="post-metadata">

**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\
**Post date:** [February 22, 2008, 2:55am UTC](https://www.ruby-forum.com/t/efficient-file-downloading/129855/2 "2008-02-22T02:55:41Z")

</div>

To clarify, I mean the file-name should be the same as it is on the web,  
not the same as the URL.

---

<div class="post-metadata">

**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\
**Post date:** [February 22, 2008, 3:25am UTC](https://www.ruby-forum.com/t/efficient-file-downloading/129855/3 "2008-02-22T03:25:59Z")

</div>

On 22 Feb 2008, at 01:54, Kyle H. wrote:

> BUFFER\_SIZE=4096  
> Try making that a lot lot bigger.

---

<div class="post-metadata">

**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\
**Post date:** [February 22, 2008, 11:07pm UTC](https://www.ruby-forum.com/t/efficient-file-downloading/129855/4 "2008-02-22T23:07:14Z")

</div>

On Feb 21, 2008, at 8:54 PM, Kyle H. wrote:

> BUFFER\_SIZE=4096  
> end  
> $ sudo gem install snoopy  
> $ snoopy [Wikipedia, the free encyclopedia](http://en.wikipedia.org/wiki/Main_Page)  
> =\> file Main\_Page

Ta dah! there’s a lot of magic behind it right now, and torrentz  
don’t work (fixed on my machine, need to release it). It does  
segmented downloading, ideal for large files. For smaller ones, it  
still works fine.

The problem with open-uri is this: it downloads the whole thing to  
your tmp directory first, so using the BUFFER\_SIZE thing won’t  
actually help.

snoopy won’t not write the file if there’s an error.

-------------------------------------------------------|  
~ Ari  
Some people want love  
Others want money  
Me… Well…  
I just want this code to compile

---

<div class="post-metadata">

**Author:** ![kyleh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@kyleh](https://www.ruby-forum.com/u/kyleh)\
**Post date:** [February 22, 2008, 3:34am UTC](https://www.ruby-forum.com/t/efficient-file-downloading/129855/5 "2008-02-22T03:34:24Z")

</div>

James T. wrote:

> On 22 Feb 2008, at 01:54, Kyle H. wrote:
> 
> > BUFFER\_SIZE=4096  
> > Try making that a lot lot bigger.

Doh! Thanks James. Brings it down to much more reasonable usage. I  
totally overlooked that very small buffer size that was set - thanks.