\
**Category:** Ruby\
**Created:** [May 15, 2006, 10:22am UTC](https://www.ruby-forum.com/t/simple-crawling-html-extraction/60526 "2006-05-15T10:22:24Z")\
**Posts on this page:** 3\
**Page:** 1
**Author:**  [@splattael](https://www.ruby-forum.com/u/splattael)\
**Post date:** [May 15, 2006, 10:22am UTC](https://www.ruby-forum.com/t/simple-crawling-html-extraction/60526/1 "2006-05-15T10:22:24Z")
Hello all,
I would like to accomplish the following task:
1. I have a bunch of links. Visit every link one by one
2. For every visited link, extract a string with a regexp from the page
3. At the end, merge every extracted string into one list
The question is: do i have to do everything by hand or are there some
higher level APIs for such stuff? What should i look at into (if i would
not like to reinvent the wheel)?
Cheers,
Peter
---
**Author:**  [@splattael](https://www.ruby-forum.com/u/splattael)\
**Post date:** [May 15, 2006, 8:19pm UTC](https://www.ruby-forum.com/t/simple-crawling-html-extraction/60526/2 "2006-05-15T20:19:39Z")
On Mon, May 15, 2006 at 05:21:27PM +0900, Peter S. wrote:
> 1. At the end, merge every extracted string into one list
>
> The question is: do i have to do everything by hand or are there some
> higher level APIs for such stuff? What should i look at into (if i would
> not like to reinvent the wheel)?
I suggest checking out WWW::Mechanize:
[http://rubyforge.org/projects/mechanize/](http://rubyforge.org/projects/mechanize/)
[http://mechanize.rubyforge.org/](http://mechanize.rubyforge.org/)
Or just ‘gem install mechanize’
Make sure to check out the examples in the RDoc.
–Aaron
---
**Author:**  [@splattael](https://www.ruby-forum.com/u/splattael)\
**Post date:** [May 15, 2006, 9:37pm UTC](https://www.ruby-forum.com/t/simple-crawling-html-extraction/60526/3 "2006-05-15T21:37:06Z")
Peter S. wrote:
> 1. At the end, merge every extracted string into one list
>
> The question is: do i have to do everything by hand or are there some
> higher level APIs for such stuff? What should i look at into (if i
> would not like to reinvent the wheel)?
>
> Cheers,
> Peter
I think Rubyful soup [1] will be of interest to you if you don’t want to
do the searching of the pages by hand.
Also, you’ll probably want to look into the URI [2] and OpenURI [2]
standard libraries for fetching the webpages.
HTH
-Justin
[1] [Rubyful Soup: "The brush has got entangled in it!"](http://www.crummy.com/software/RubyfulSoup/)
[2] [http://ruby-doc.org/stdlib/libdoc/uri/rdoc/index.html](http://ruby-doc.org/stdlib/libdoc/uri/rdoc/index.html)
[3] [http://ruby-doc.org/stdlib/libdoc/open-uri/rdoc/index.html](http://ruby-doc.org/stdlib/libdoc/open-uri/rdoc/index.html)