# Seeking for a ruby spider robot example

**URL:** <https://www.ruby-forum.com/t/seeking-for-a-ruby-spider-robot-example/66169>\
**Category:** Ruby\
**Created:** [August 23, 2006, 7:10pm UTC](https://www.ruby-forum.com/t/seeking-for-a-ruby-spider-robot-example/66169 "2006-08-23T19:10:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Haofei](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/a4c791/32.png) [@Haofei](https://www.ruby-forum.com/u/Haofei)\
**Post date:** [August 23, 2006, 7:10pm UTC](https://www.ruby-forum.com/t/seeking-for-a-ruby-spider-robot-example/66169/1 "2006-08-23T19:10:06Z")

</div>

Hi, i want to write a little spider to do some web search  
but have no idea how to start it. is there any example or something like  
that?  
thanks 🙂

yuesefa

---

<div class="post-metadata">

**Author:** ![Haofei](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/a4c791/32.png) [@Haofei](https://www.ruby-forum.com/u/Haofei)\
**Post date:** [August 23, 2006, 7:38pm UTC](https://www.ruby-forum.com/t/seeking-for-a-ruby-spider-robot-example/66169/2 "2006-08-23T19:38:11Z")

</div>

Haofei wrote:

> Hi, i want to write a little spider to do some web search  
> but have no idea how to start it. is there any example or something  
> like that?  
> thanks 🙂
> 
> yuesefa

You can see an extremely simple/limited one that I made a while back:

[http://students.seattleu.edu/collinsj/programs\_netcrawler.html](http://students.seattleu.edu/collinsj/programs_netcrawler.html)

It may give you a place to start, but there are very good libraries for  
getting and parsing websites, like Rubyful Soup, Mechanize, open-uri,  
and so on.

Also, try searching the archives for more.

-Justin

---

<div class="post-metadata">

**Author:** ![Haofei](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/a4c791/32.png) [@Haofei](https://www.ruby-forum.com/u/Haofei)\
**Post date:** [August 24, 2006, 1:28am UTC](https://www.ruby-forum.com/t/seeking-for-a-ruby-spider-robot-example/66169/3 "2006-08-24T01:28:47Z")

</div>

On Thu, Aug 24, 2006 at 02:09:28AM +0900, Haofei wrote:

> Hi, i want to write a little spider to do some web search  
> but have no idea how to start it. is there any example or something like  
> that?  
> thanks 🙂
> 
> yuesefa

You can write one with WWW::Mechanize. I have an example on my blog:

[http://tenderlovemaking.com/2006/05/26/mechanize-one-liners/](http://tenderlovemaking.com/2006/05/26/mechanize-one-liners/)

There is also an example spider that comes along with Mechanize, just  
look in the ‘eg’ directory.

Here is the spider for those that don’t want to click (its not perfect,  
but its small!):

(mech = WWW::Mechanize.new).get(ARGV[0])  
(a = lambda { |p|  
mech.page.links.each { |l|  
mech.click(l) && p.call(p) if ! mech.visited? l  
}  
}).call(a)

–Aaron

---

<div class="post-metadata">

**Author:** ![Haofei](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/a4c791/32.png) [@Haofei](https://www.ruby-forum.com/u/Haofei)\
**Post date:** [August 25, 2006, 12:44pm UTC](https://www.ruby-forum.com/t/seeking-for-a-ruby-spider-robot-example/66169/4 "2006-08-25T12:44:02Z")

</div>

I found hpricot easy to use :  
[http://code.whytheluckystiff.net/hpricot/](http://code.whytheluckystiff.net/hpricot/)

There’s some good code examples on the site too.

Chris