# WGETting from within rails

**URL:** <https://www.ruby-forum.com/t/wgetting-from-within-rails/104005>\
**Category:** Rails\
**Created:** [May 12, 2007, 8:38pm UTC](https://www.ruby-forum.com/t/wgetting-from-within-rails/104005 "2007-05-12T20:38:16Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![plewizard](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/ce7236/32.png) [@plewizard](https://www.ruby-forum.com/u/plewizard)\
**Post date:** [May 12, 2007, 8:38pm UTC](https://www.ruby-forum.com/t/wgetting-from-within-rails/104005/1 "2007-05-12T20:38:16Z")

</div>

i’m writing an app that wants to fetch a URL from a remote server and  
return the html text file to the app. i’ve been told there are some  
xml parsers for perl that have request functionality built right in.  
is there such a thing for Ruby or if nothing else the possibility of  
using WGET from within a Ruby program.

thanks in advance for any help.

---

<div class="post-metadata">

**Author:** ![plewizard](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/ce7236/32.png) [@plewizard](https://www.ruby-forum.com/u/plewizard)\
**Post date:** [May 12, 2007, 8:43pm UTC](https://www.ruby-forum.com/t/wgetting-from-within-rails/104005/2 "2007-05-12T20:43:16Z")

</div>

On May 12, 11:37 am, plewizard [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> i’m writing an app that wants to fetch a URL from a remote server and  
> return the html text file to the app. i’ve been told there are some  
> xml parsers for perl that have request functionality built right in.  
> is there such a thing for Ruby or if nothing else the possibility of  
> using WGET from within a Ruby program.
> 
> thanks in advance for any help.

you probably want #read in open-uri (or Net::HTTP)  
[http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/index.html](http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/index.html)

or you could just backtick:  
`curl http::/yourURLhere`

---

<div class="post-metadata">

**Author:** ![plewizard](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/ce7236/32.png) [@plewizard](https://www.ruby-forum.com/u/plewizard)\
**Post date:** [May 12, 2007, 9:32pm UTC](https://www.ruby-forum.com/t/wgetting-from-within-rails/104005/3 "2007-05-12T21:32:20Z")

</div>

plewizard,

> i’m writing an app that wants to fetch a URL from a remote server  
> and  
> return the html text file to the app. i’ve been told there are some

If you just want to fetch the page text :

require ‘open-uri’;  
html = open(“[http://yahoo.com](http://yahoo.com)”).readlines

puts html

If you need to extract part(s) of the page, use Hpricot

```
require 'rubygems';
require 'open-uri'
require 'hpricot'

doc = Hpricot(open("http://yahoo.com"))

title = doc.search('title').text
body = doc.search('body' ).text
links = doc .search("a").collect{|a| a['href'] }

puts title
puts body
puts links.to_yaml

```

Hpricot also lets you modify the page contents :  
f.ex,  
doc.search(‘a’).each{|link| link.swap(link.inner\_html)}

turns all the  
something  
into  
something

## Alain R.

[http://blog.ravet.com](http://blog.ravet.com)