# Looking for web crawler written in Ruby **URL:** \ **Category:** Ruby\ **Created:** [January 31, 2006, 8:14am UTC](https://www.ruby-forum.com/t/looking-for-web-crawler-written-in-ruby/53285 "2006-01-31T08:14:41Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Kev](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e9bcb4/32.png) [@Kev](https://www.ruby-forum.com/u/Kev)\ **Post date:** [January 31, 2006, 8:14am UTC](https://www.ruby-forum.com/t/looking-for-web-crawler-written-in-ruby/53285/1 "2006-01-31T08:14:41Z")
Hi, I am looking for web crawler(spider) written in Ruby. I googled but couldn’t find much information. Can anyone give me some links? Thanks in advance. Kevin ---
**Author:** ![Kev](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e9bcb4/32.png) [@Kev](https://www.ruby-forum.com/u/Kev)\ **Post date:** [January 31, 2006, 8:46pm UTC](https://www.ruby-forum.com/t/looking-for-web-crawler-written-in-ruby/53285/2 "2006-01-31T20:46:17Z")
Kev wrote: > Hi, I am looking for web crawler(spider) written in Ruby. I googled but > couldn’t find much information. Can anyone give me some links? > > Thanks in advance. > > Kevin If you are _really_ desperate, the first Ruby program I ever wrote was kind of a web crawler: [http://students.seattleu.edu/collinsj/programs\_netcrawler.html](http://students.seattleu.edu/collinsj/programs_netcrawler.html) But it’s really terrible. It isn’t hard to write your own, if you need to, using Net::HTTP ([http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/index.html](http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/index.html)) and URI ([http://ruby-doc.org/stdlib/libdoc/uri/rdoc/index.html](http://ruby-doc.org/stdlib/libdoc/uri/rdoc/index.html)). -Justin ---
**Author:** ![Kev](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e9bcb4/32.png) [@Kev](https://www.ruby-forum.com/u/Kev)\ **Post date:** [January 31, 2006, 10:45pm UTC](https://www.ruby-forum.com/t/looking-for-web-crawler-written-in-ruby/53285/3 "2006-01-31T22:45:11Z")
On Jan 31, 2006, at 1:11 AM, Kev wrote: > Hi, I am looking for web crawler(spider) written in Ruby. I googled > but > couldn’t find much information. Can anyone give me some links? The code I translated from Higher-Order Perl at the tail end of the following blog article may be of use to you. [http://blog.grayproductions.net/articles/2006/01/31/iterators-](http://blog.grayproductions.net/articles/2006/01/31/iterators-) chapters-4-and-5 James Edward G. II ---
**Author:** ![Kev](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e9bcb4/32.png) [@Kev](https://www.ruby-forum.com/u/Kev)\ **Post date:** [February 1, 2006, 1:49pm UTC](https://www.ruby-forum.com/t/looking-for-web-crawler-written-in-ruby/53285/4 "2006-02-01T13:49:54Z")
Many thanks, I appreciate it very much. ---
**Author:** ![Kev](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e9bcb4/32.png) [@Kev](https://www.ruby-forum.com/u/Kev)\ **Post date:** [February 1, 2006, 11:46pm UTC](https://www.ruby-forum.com/t/looking-for-web-crawler-written-in-ruby/53285/5 "2006-02-01T23:46:32Z")
Kev wrote: > Hi, I am looking for web crawler(spider) written in Ruby. I googled but > couldn’t find much information. Can anyone give me some links? [http://www.jamesbritt.com/articles/RubyAndVbaForWebDictionarySearches.html](http://www.jamesbritt.com/articles/RubyAndVbaForWebDictionarySearches.html) [http://www.linux-magazine.com/issue/51/Ruby\_Web\_Spiders.pdf](http://www.linux-magazine.com/issue/51/Ruby_Web_Spiders.pdf) [http://www.acc.umu.se/~r2d2/programming/ruby/webfetcher/](http://www.acc.umu.se/~r2d2/programming/ruby/webfetcher/) [http://snippets.textdrive.com/posts/show/74](http://snippets.textdrive.com/posts/show/74) (look at cached page) [http://rubyforge.org/projects/wee/](http://rubyforge.org/projects/wee/) look at Mechanize ---
**Author:** ![Kev](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e9bcb4/32.png) [@Kev](https://www.ruby-forum.com/u/Kev)\ **Post date:** [February 2, 2006, 1:30am UTC](https://www.ruby-forum.com/t/looking-for-web-crawler-written-in-ruby/53285/6 "2006-02-02T01:30:35Z")
On 1/31/06, Kev [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Hi, I am looking for web crawler(spider) written in Ruby. I googled but > couldn’t find much information. Can anyone give me some links? Some more offerings here: [http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/146169?146040-146977](http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/146169?146040-146977) now you’re practically crawling with spiders 🙂 ---
**Author:** ![Kev](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/e9bcb4/32.png) [@Kev](https://www.ruby-forum.com/u/Kev)\ **Post date:** [February 2, 2006, 1:45am UTC](https://www.ruby-forum.com/t/looking-for-web-crawler-written-in-ruby/53285/7 "2006-02-02T01:45:50Z")
Gene T. wrote: > Kev wrote: > > > Hi, I am looking for web crawler(spider) written in Ruby. I googled but > > couldn’t find much information. Can anyone give me some links? > > [http://www.jamesbritt.com/articles/RubyAndVbaForWebDictionarySearches.html](http://www.jamesbritt.com/articles/RubyAndVbaForWebDictionarySearches.html) There’s not much spidering going on there, just some page scraping > [http://rubyforge.org/projects/wee/](http://rubyforge.org/projects/wee/) > look at Mechanize My Mechanize example may be helpful, though. > **[How I Did It: Building RubyStuff.com](https://neurogami.com/cafe-fetcher/)** > > A microformat that eases the transition from page to print with CSS and HTML – James B. [http://www.ruby-doc.org](http://www.ruby-doc.org) - Ruby Help & Documentation [Ruby Code & Style](http://www.artima.com/rubycs/) - The Journal By & For Rubyists [http://www.rubystuff.com](http://www.rubystuff.com) - The Ruby Store for Ruby Stuff [http://www.jamesbritt.com](http://www.jamesbritt.com) - Playing with Better Toys [http://www.30secondrule.com](http://www.30secondrule.com) - Building Better Tools