# Wikipedia Parser **URL:** \ **Category:** Rails\ **Created:** [April 12, 2007, 8:25pm UTC](https://www.ruby-forum.com/t/wikipedia-parser/91845 "2007-04-12T20:25:54Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![david](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/aeb1de/32.png) [@david](https://www.ruby-forum.com/u/david)\ **Post date:** [April 12, 2007, 8:25pm UTC](https://www.ruby-forum.com/t/wikipedia-parser/91845/1 "2007-04-12T20:25:54Z")
I need to parse and redisplay in html wikipedia articles (formatted with the wikipedia style). Has anyone encountered such a library in ruby ? Any libraries that are good at that? Thanks ---
**Author:** ![david](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/aeb1de/32.png) [@david](https://www.ruby-forum.com/u/david)\ **Post date:** [April 12, 2007, 8:41pm UTC](https://www.ruby-forum.com/t/wikipedia-parser/91845/2 "2007-04-12T20:41:57Z")
David wrote: > I need to parse and redisplay in html wikipedia articles (formatted > with the wikipedia style). Has anyone encountered such a library in > ruby ? Any libraries that are good at that? > > Thanks > > > ## Check out [http://shanesbrain.net/articles/2006/10/02/screen-scraping-wikipedia](http://shanesbrain.net/articles/2006/10/02/screen-scraping-wikipedia) Makes it dead easy to roll your own. Chris [http://www.autopendium.co.uk](http://www.autopendium.co.uk) Stuff about old cars ---
**Author:** ![david](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/aeb1de/32.png) [@david](https://www.ruby-forum.com/u/david)\ **Post date:** [April 12, 2007, 8:47pm UTC](https://www.ruby-forum.com/t/wikipedia-parser/91845/3 "2007-04-12T20:47:58Z")
Usually you shouldn’t use bots on wikipedia, but should download the free database instead and use that. Read about their policy here: > **[Wikipedia:Bots](https://en.wikipedia.org/wiki/Wikipedia:Bots)** > > A bot (a common nickname for software robot) is an automated tool that carries out repetitive and mundane tasks to maintain the 58,534,848 pages of the English Wikipedia. Bots are able to make edits very rapidly, but can disrupt Wikipedia if they are incorrectly designed or operated. For these reasons, a bot policy has been developed. > There are currently 2,646 bot tasks approved for use on the English Wikipedia; however, not all approved tasks involve actively carrying out edits. Bots will lea... If you have your own mediawiki install and want to use a bot, you can check out pywikipedia bot: [pywikibot download | SourceForge.net](http://sourceforge.net/projects/pywikipediabot/) It’s not in ruby, but it works great. ---
**Author:** ![david](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/aeb1de/32.png) [@david](https://www.ruby-forum.com/u/david)\ **Post date:** [April 12, 2007, 10:53pm UTC](https://www.ruby-forum.com/t/wikipedia-parser/91845/4 "2007-04-12T22:53:13Z")
Actually, I’m not entirely sure that you shouldn’t use bots at all on the Wikipedia. According to the link you provided: “_Robots_ or _bots_ are automatic processes[http://en.wikipedia.org/wiki/Process\_(computing)](http://en.wikipedia.org/wiki/Process_%28computing%29)that interact with Wikipedia as though they were human editors” That last bit sounds like they’re talking about a very specific kind of bot and not just a scraper. RSL ---
**Author:** ![david](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/aeb1de/32.png) [@david](https://www.ruby-forum.com/u/david)\ **Post date:** [April 12, 2007, 11:13pm UTC](https://www.ruby-forum.com/t/wikipedia-parser/91845/5 "2007-04-12T23:13:16Z")
I wrote that article a while ago. It’ll be interesting to use WWW::Mechanize, or better yet, scRUBYt, which use Hpricot in the backend anyway. Shane [http://shanesbrain.net](http://shanesbrain.net) ---
**Author:** ![david](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/aeb1de/32.png) [@david](https://www.ruby-forum.com/u/david)\ **Post date:** [April 12, 2007, 11:13pm UTC](https://www.ruby-forum.com/t/wikipedia-parser/91845/6 "2007-04-12T23:13:15Z")
“_Robots_ or _bots_ are automatic processes[http://en.wikipedia.org/wiki/Process\_(computing)](http://en.wikipedia.org/wiki/Process_%28computing%29)that interact with Wikipedia as though they were human editors.” There’s nothing against screen-scraping there. That policy is about bots which edit content. Otherwise, Google would be breaking WP policy. This is taking the discussion a little off topic though. -Nathan ---
**Author:** ![david](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/aeb1de/32.png) [@david](https://www.ruby-forum.com/u/david)\ **Post date:** [April 12, 2007, 11:21pm UTC](https://www.ruby-forum.com/t/wikipedia-parser/91845/7 "2007-04-12T23:21:16Z")
If you just need to cache some pages for displaying later, screen scraping Wikipedia is a good choice compared to downloading the db. If you’re going to be parsing and redisplaying the content in real time that is against Wikipedia’s policy. See [Wikipedia, the free encyclopedia](http://en.wikipedia.org/wiki/) Wikipedia:Database\_download#Why\_not\_just\_retrieve\_data\_from\_wikipedia.or g\_at\_runtime.3F