# Crawler for Ferret

**URL:** <https://www.ruby-forum.com/t/crawler-for-ferret/163403>\
**Category:** Ferret\
**Created:** [March 19, 2009, 11:12pm UTC](https://www.ruby-forum.com/t/crawler-for-ferret/163403 "2009-03-19T23:12:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![HuangS\_ZijianSVicto](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/a3d4f5/32.png) [@HuangS\_ZijianSVicto](https://www.ruby-forum.com/u/HuangS_ZijianSVicto)\
**Post date:** [March 19, 2009, 11:12pm UTC](https://www.ruby-forum.com/t/crawler-for-ferret/163403/1 "2009-03-19T23:12:48Z")

</div>

Hi, guys:  
Can you please recommend a good crawler for Ferret? Nutch is pretty  
powerful in the Java side, do we have some thing is similar in Ruby? It  
will be great if the crawler also handlers incremental index update  
easily.

Thanks

Victor

---

<div class="post-metadata">

**Author:** ![HuangS\_ZijianSVicto](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/a3d4f5/32.png) [@HuangS\_ZijianSVicto](https://www.ruby-forum.com/u/HuangS_ZijianSVicto)\
**Post date:** [March 19, 2009, 11:29pm UTC](https://www.ruby-forum.com/t/crawler-for-ferret/163403/2 "2009-03-19T23:29:13Z")

</div>

On 19.03.2009, at 22:32, Huang, Zijian(Victor) wrote:

> Hi, guys:  
> Can you please recommend a good crawler for Ferret? Nutch is  
> pretty powerful in the Java side, do we have some thing is similar  
> in Ruby? It will be great if the crawler also handlers incremental  
> index update easily.

RDig can do http crawling, but cannot really be compared with Nutch  
feature- and performance wise as it was designed for intranet use, say  
indexing the web pages of a few hosts.

Cheers,  
Jens

–  
Jens Krämer  
webit! Gesellschaft für neue Medien mbH  
Schnorrstraße 76 | 01069 Dresden  
Telefon +49351467660 | Telefax +493514676666  
removed\_email\_address@domain.invalid | [www.webit.de](http://www.webit.de)

Amtsgericht Dresden | HRB 15422  
GF Sven Haubold

---

<div class="post-metadata">

**Author:** ![HuangS\_ZijianSVicto](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/a3d4f5/32.png) [@HuangS\_ZijianSVicto](https://www.ruby-forum.com/u/HuangS_ZijianSVicto)\
**Post date:** [March 20, 2009, 12:27am UTC](https://www.ruby-forum.com/t/crawler-for-ferret/163403/3 "2009-03-20T00:27:09Z")

</div>

I wrote one called Suckr.

[http://goddard.net.nz/projects/suckr/](http://goddard.net.nz/projects/suckr/)

It does the crawling, including incremental update and provides a  
command line  
search interface. I’ve had some periodic stability issues with this on  
the old  
Debian box I’ve been using it on myself - please test thoroughly.

It has some documentation in the README file. Please let me know if you  
have  
any questions.

Cheers,

Tim

---

<div class="post-metadata">

**Author:** ![HuangS\_ZijianSVicto](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/a3d4f5/32.png) [@HuangS\_ZijianSVicto](https://www.ruby-forum.com/u/HuangS_ZijianSVicto)\
**Post date:** [March 20, 2009, 1:54pm UTC](https://www.ruby-forum.com/t/crawler-for-ferret/163403/4 "2009-03-20T13:54:01Z")

</div>

On Thu, 19 Mar 2009, Huang, Zijian(Victor) wrote:

> Hi, guys:  
> Can you please recommend a good crawler for Ferret? Nutch is pretty  
> powerful in the Java side, do we have some thing is similar in Ruby? It  
> will be great if the crawler also handlers incremental index update  
> easily.

And then this shows up in my news feeds:

[http://www.rubyinside.com/building-a-search-engine-in-200ish-lines-of-ruby-1655.html](http://www.rubyinside.com/building-a-search-engine-in-200ish-lines-of-ruby-1655.html)

I’ve not followed the links off it, though, so YMMV.

> Thanks
> 
> Victor

```
    Hugh

```