Forum: Ruby Looking for web crawler written in Ruby

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
569dc399cc92c579c2c4b11dbd08cdf5?d=identicon&s=25 Kev (Guest)
on 2006-01-31 08:14
(Received via mailing list)
Hi, I am looking for web crawler(spider) written in Ruby. I googled but
couldn't find much information. Can anyone give me some links?

Thanks in advance.

Kevin
F3b7109c91841c7106784d229418f5dd?d=identicon&s=25 Justin Collins (Guest)
on 2006-01-31 20:46
(Received via mailing list)
Kev wrote:
> Hi, I am looking for web crawler(spider) written in Ruby. I googled but
> couldn't find much information. Can anyone give me some links?
>
> Thanks in advance.
>
> Kevin
>

If you are _really_ desperate, the first Ruby program I ever wrote was
kind of a web crawler:
http://students.seattleu.edu/collinsj/programs_net...

But it's really terrible. It isn't hard to write your own, if you need
to, using Net::HTTP
(http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/index.html) and URI
(http://ruby-doc.org/stdlib/libdoc/uri/rdoc/index.html).

-Justin
4299e35bacef054df40583da2d51edea?d=identicon&s=25 James Gray (bbazzarrakk)
on 2006-01-31 22:45
(Received via mailing list)
On Jan 31, 2006, at 1:11 AM, Kev wrote:

> Hi, I am looking for web crawler(spider) written in Ruby. I googled
> but
> couldn't find much information. Can anyone give me some links?

The code I translated from Higher-Order Perl at the tail end of the
following blog article may be of use to you.

http://blog.grayproductions.net/articles/2006/01/3...
chapters-4-and-5

James Edward Gray II
569dc399cc92c579c2c4b11dbd08cdf5?d=identicon&s=25 Kev (Guest)
on 2006-02-01 13:49
(Received via mailing list)
Many thanks, I appreciate it very much.
9dfe8c734b0f9b37a4e218425c0a2138?d=identicon&s=25 Gene Tani (Guest)
on 2006-02-01 23:46
(Received via mailing list)
Kev wrote:
> Hi, I am looking for web crawler(spider) written in Ruby. I googled but
> couldn't find much information. Can anyone give me some links?
>

http://www.jamesbritt.com/articles/RubyAndVbaForWe...
http://www.linux-magazine.com/issue/51/Ruby_Web_Spiders.pdf
http://www.acc.umu.se/~r2d2/programming/ruby/webfetcher/

http://snippets.textdrive.com/posts/show/74
(look at cached page)

http://rubyforge.org/projects/wee/
look at Mechanize
Ff63c03fd68754adbadd2c6314646bef?d=identicon&s=25 Bill Guindon (agorilla)
on 2006-02-02 01:30
(Received via mailing list)
On 1/31/06, Kev <tuweiwen@gmail.com> wrote:
> Hi, I am looking for web crawler(spider) written in Ruby. I googled but
> couldn't find much information. Can anyone give me some links?

Some more offerings here:
http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/rub...

now you're practically crawling with spiders :)
Bc6d88907ce09158581fbb9b469a35a3?d=identicon&s=25 James Britt (Guest)
on 2006-02-02 01:45
(Received via mailing list)
Gene Tani wrote:
> Kev wrote:
>
>>Hi, I am looking for web crawler(spider) written in Ruby. I googled but
>>couldn't find much information. Can anyone give me some links?
>>
>
>
> http://www.jamesbritt.com/articles/RubyAndVbaForWe...

There's not much spidering going on there, just some page scraping

>
> http://rubyforge.org/projects/wee/
> look at Mechanize
>

My Mechanize example may be helpful, though.

http://neurogami.com/cafe-fetcher/



--
James Britt

http://www.ruby-doc.org       - Ruby Help & Documentation
http://www.artima.com/rubycs/ - The Journal By & For Rubyists
http://www.rubystuff.com      - The Ruby Store for Ruby Stuff
http://www.jamesbritt.com     - Playing with Better Toys
http://www.30secondrule.com   - Building Better Tools
This topic is locked and can not be replied to.