How to download file from web site

Ruby
1

Okay, it’s easy to download a file from an FTP server with Net::FTP.
Given a URL like this:
“http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz” is
there a way using pure Ruby to download this file?

2

On Sat, 3 Mar 2007, Timothy H. wrote:

Okay, it’s easy to download a file from an FTP server with Net::FTP. Given a
URL like this:
“http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz” is there
a way using pure Ruby to download this file?

harp:~ > cat a.rb
require ‘open-uri’
uri =
‘http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz’

open(uri) do |fin|
open(File.basename(uri), ‘w’) do |fout|
while(buf = fin.read(8192))
fout.write buf
end
end
end

harp:~ > ruby a.rb

harp:~ > new RMagick-1.15.3.tar.gz
-rw-rw-r-- 1 ahoward ahoward 1048764 Mar 3 08:00
RMagick-1.15.3.tar.gz

-a

3

[email protected] wrote:

uri = ‘http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz’

harp:~ > new RMagick-1.15.3.tar.gz
-rw-rw-r-- 1 ahoward ahoward 1048764 Mar 3 08:00
RMagick-1.15.3.tar.gz

-a
Thanks again, Ara!

4

On 03.03.2007 15:47, Timothy H. wrote:

Okay, it’s easy to download a file from an FTP server with Net::FTP.
Given a URL like this:
“http://rubyforge.org/frs/download.php/17670/RMagick-1.15.3.tar.gz” is
there a way using pure Ruby to download this file?

You might be surprised to learn that there is also Net::HTTP…

http://www.ruby-doc.org/stdlib/libdoc/net/http/rdoc/index.html

And then there’s open-uri

http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/index.html

robert

5

[email protected] wrote:

dirname, basename = File.split path

thanks for rmagick.

-a
You’re welcome!

The reason I’m asking these questions is because I’m working on a Ruby
program to download and install ImageMagick, all its delegate libraries,
and RMagick, on OS X, starting from scratch and not using MacPorts.

BTW, I saw your comment about the rmagick gem. You’re right, it’s
screwy. I’m hoping to mothball it in RMagick 2.0.0, if I can figure out
a way to replace my autoconf-generated configure script with pure Ruby.

6

On Mar 3, 2007, at 9:39 AM, Timothy H. wrote:

The reason I’m asking these questions is because I’m working on a
Ruby program to download and install ImageMagick, all its delegate
libraries, and RMagick, on OS X, starting from scratch and not
using MacPorts.

I just spent about a day fixing my RMagick install (on Mac OS X),
that had become horribly broken, so releasing something like this
would make you a hero in my eyes.

James Edward G. II

7

On Sun, 4 Mar 2007, Timothy H. wrote:

-a
Thanks again, Ara!

no way - thank you! i’m preparing a class right now, here’s one of many
examples i’ve got put together:

gem install RMagick @ http://rmagick.rubyforge.org/

require “rubygems”
require “RMagick”

converts all images named on the command-line into thumbnails

ARGV.each do |path|
path = File.expand_path path
dirname, basename = File.split path
base, ext = basename.split %r/[.]/, 2

 img = Magick::ImageList.new path

 thumb = img.thumbnail 96, 96
 thumb_path = File.join dirname, "#{ base }.thumb.#{ ext }"
 thumb.write thumb_path

 puts thumb_path

end

thanks for rmagick.

-a

8

James Edward G. II wrote:

James Edward G. II

Ouch! Sorry you had trouble. Email me when you have a problem, James.
I’ll be glad to do what I can to help.

BTW, just last week I updated my “Installing RMagick on OS X HOWTO”
here: http://rmagick.rubyforge.org/install-osx.html. I guess it’s too
late to help you but maybe it’ll help somebody else.

9

On Mar 3, 2007, at 11:53 AM, Timothy H. wrote:

would make you a hero in my eyes.

James Edward G. II

Ouch! Sorry you had trouble. Email me when you have a problem,
James. I’ll be glad to do what I can to help.

No worries. It wasn’t your fault. I broke it.

James Edward G. II

10

On Mar 3, 2007, at 9:53 AM, Timothy H. wrote:

would make you a hero in my eyes.

James Edward G. II

Ouch! Sorry you had trouble. Email me when you have a problem,
James. I’ll be glad to do what I can to help.

BTW, just last week I updated my “Installing RMagick on OS X HOWTO”
here: http://rmagick.rubyforge.org/install-osx.html. I guess it’s
too late to help you but maybe it’ll help somebody else.

Here is a shell script I wrote a while ago to build ruby/imagemagick/
rmagick on OSX without using macports. Feel free to use it if you want.

#!/bin/sh

Install Ruby & IMageMagick & RMagick on Macintosh OS X 10.4.7 (Intel)

Stock Apple Developer Tools (2.3)

Readline

READLINE_VERSION=“5.1”
wget ftp://ftp.gnu.org/gnu/readline/readline-${READLINE_VERSION}.tar.gz
tar xzvf readline-${READLINE_VERSION}.tar.gz
pushd readline-${READLINE_VERSION}
./configure --prefix=/usr/local
make
sudo make install
popd

Ruby

RUBY_VERSION=“1.8.5”

wget ftp://ftp.ruby-lang.org/pub/ruby/ruby-${RUBY_VERSION}.tar.gz
tar xfz ruby-${RUBY_VERSION}.tar.gz

pushd ruby-${RUBY_VERSION}
./configure
make
sudo make install
popd

Ruby Gems

GEM_VERSION=“0.9.2”

wget http://rubyforge.org/frs/download.php/11289/rubygems-$
{GEM_VERSION}.tgz
tar xfz rubygems-${GEM_VERSION}.tgz

pushd rubygems-${GEM_VERSION}
sudo ruby setup.rb
popd

ImageMagick

LPNG_VERSION=“1.2.12”

wget ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-$
{LPNG_VERSION}.tar.gz
tar xfz libpng-${LPNG_VERSION}.tar.gz

pushd libpng-${LPNG_VERSION}
./configure
make
sudo make install
popd

JPEG_VERSION=“6b”

wget http://www.ijg.org/files/jpegsrc.v${JPEG_VERSION}.tar.gz
tar xfz jpegsrc.v${JPEG_VERSION}.tar.gz

pushd jpeg-${JPEG_VERSION}
./configure
make
sudo make install
sudo make install-lib
popd

TIFF_VERSION=“3.8.2”

wget ftp://ftp.remotesensing.org/pub/libtiff/tiff-${TIFF_VERSION}.tar.gz
tar xfz tiff-${TIFF_VERSION}.tar.gz

pushd tiff-${TIFF_VERSION}
./configure
make
sudo make install
popd

IM_VERSION_MAJOR=“6.2.9”
IM_VERSION_MINOR=“4”

IM_VERSION=“${IM_VERSION_MAJOR}-${IM_VERSION_MINOR}”

wget ftp://ftp.imagemagick.net/pub/ImageMagick/ImageMagick-$
{IM_VERSION}.tar.gz
tar xfz ImageMagick-${IM_VERSION}.tar.gz

pushd ImageMagick-${IM_VERSION_MAJOR}
./configure
make
sudo make install
popd

sudo gem install rmagick

Cheers-

– Ezra Z.
– Lead Rails Evangelist
– [email protected]
– Engine Y., Serious Rails Hosting
– (866) 518-YARD (9273)

11

On Mon, 5 Mar 2007, Joel VanderWerf wrote:

end

Didn’t the rio library[1] have a way to abstract this operation?

This is from the docs:

Copy a web-page to a file rio(‘http://rubydoc.org/’) > rio(‘afile’)

I suppose it’s using the same buffering technique underneath that
abstraction.

[1] http://rio.rubyforge.org/ (never used it myself)

reminds me of an RCR i’ve had in mind:

io.relay another_io, bufsize

seems like a nice one huh?

-a

12

On Mon, 5 Mar 2007, Bil K. wrote:

end
end

He asks naively…

Why is this so complicated, e.g., the 8192 buffered write, etc.?

well, this will do it too

def wget src, dst
open(dst,‘w’){|b| open(src){|a| b.write a.read}}
end

but if you

wget ‘http://host/huge_giant_output.cgi’, ‘big.out’

and then start 1000 of them in threads you might eat a lot of memory

the 8192 thing is just being robust in the face of potential giant docs.
not
an issue for html probably but the OP was talking about downloading
tar.gz
files, so those could be big. we’ve got plenty of them that are > 10GB
for
download on our servers…

cheers.

-a

13

[email protected] wrote:

end
He asks naively…

Why is this so complicated, e.g., the 8192 buffered write, etc.?

Later,

14

Ezra Z. wrote:

Here is a shell script I wrote a while ago to build

ruby/imagemagick/rmagick on OSX without using macports. Feel free to
use it if you want.

Thanks, Ezra! That’s a very nice gift. I’ll incorporate all the best
parts into my script.

There are a couple of these kinds of bash scripts for installing RMagick
floating around. None of them implement my exact set of superstitions
about installing ImageMagick[1], so I thought it would be useful to do
my own. I’m using Ruby instead of bash, though, since I get very little
opportunity to write Ruby code.

[1] “Speed up your RMagick apps in 1 easy step”
http://rubyforge.org/forum/forum.php?thread_id=10975&forum_id=1618

15

On 04.03.2007 20:50, [email protected] wrote:

open(File.basename(uri), ‘w’) do |fout|
It does seem too low-level, doesn’t it? Especially in the same snippet

[1] http://rio.rubyforge.org/ (never used it myself)

reminds me of an RCR i’ve had in mind:

io.relay another_io, bufsize

seems like a nice one huh?

Yes, although I’m not sure whether “relay” is immediately clear to
everyone - especially non native speakers. I briefly thought of
“copy_to” but that is not correct in every case, i.e. not always it is
actually a copy. What about “forward_to”? Hm…

Kind regards

robert

16

Bil K. wrote:

end
end

He asks naively…

Why is this so complicated, e.g., the 8192 buffered write, etc.?

It does seem too low-level, doesn’t it? Especially in the same snippet
with open(uri), which abstracts so much.

Didn’t the rio library[1] have a way to abstract this operation?

This is from the docs:

Copy a web-page to a file
rio(‘http://rubydoc.org/’) > rio(‘afile’)

I suppose it’s using the same buffering technique underneath that
abstraction.

[1] http://rio.rubyforge.org/ (never used it myself)

17

On Mar 4, 2007, at 10:10 AM, Bil K. wrote:

end

He asks naively…

Why is this so complicated, e.g., the 8192 buffered write, etc.?

So you don’t try to slurp a very large file into Ruby’s memory I’m
guessing.

James Edward G. II