# Regrex\_crawler -- a crawler which uses regular expression to catch data from website

**URL:** <https://www.ruby-forum.com/t/regrex-crawler-a-crawler-which-uses-regular-expression-to-catch-data-from-website/175419>\
**Category:** Rails\
**Created:** [September 13, 2009, 2:59pm UTC](https://www.ruby-forum.com/t/regrex-crawler-a-crawler-which-uses-regular-expression-to-catch-data-from-website/175419 "2009-09-13T14:59:09Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![flyerhzm](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/d26b3c/32.png) [@flyerhzm](https://www.ruby-forum.com/u/flyerhzm)\
**Post date:** [September 13, 2009, 2:59pm UTC](https://www.ruby-forum.com/t/regrex-crawler-a-crawler-which-uses-regular-expression-to-catch-data-from-website/175419/1 "2009-09-13T14:59:09Z")

</div>

RegexpCrawler is a crawler which uses regular expression to catch data  
from website. It is easy to use and less code if you are familiar with  
regular expression.  
The project site is: [http://github.com/flyerhzm/regexp\_crawler/tree](http://github.com/flyerhzm/regexp_crawler/tree)

I give an example: a script to synchronize your github projects except  
fork projects, , please check example/github\_projects.rb

require ‘rubygems’  
require ‘regexp\_crawler’

crawler = RegexpCrawler::Crawler.new(  
:start\_page =\> “[flyerhzm (Richard Huang) · GitHub](http://github.com/flyerhzm)”,  
:continue\_regexp =\> %r{

**}m,  
:capture\_regexp =\> %r{\<a href=“http://github.com/flyerhzm/[^/”]_?(?😕  
tree)?"\>(._?)._\<span id=“repository\_description”._?\>(._?)._  
(
._?_
 _)_**
**_}m,  
:named\_captures =\> [‘title’, ‘description’, ‘body’],  
:save\_method =\> Proc.new do |result, page|  
puts ‘=============================’  
puts page·  
puts result[:title]  
puts result[:description]  
puts result[:body][0…100] + “…”  
end,·  
:need\_parse =\> Proc.new do |page, response\_body|  
page =~ %r{[http://github.com/flyerhzm/\w+](http://github.com/flyerhzm/%5Cw+)} && !response\_body.index  
(/Fork of._?/)  
end)·  
crawler.start**

 

**The results are as follows:**

 

=============================

> **[GitHub - flyerhzm/bullet: help to kill N+1 queries and unused eager loading](https://github.com/flyerhzm/bullet/tree/main)**
>
> main

bullet  
A rails plugin/gem to kill N+1 queries and unused eager loading

# Bullet

The Bullet plugin/gem is designed to help you increase your... ============================= http://github.com/flyerhzm/regexp\_crawler/tree/master regexp\_crawler A crawler which use regular expression to catch data.

# RegexpCrawler

RegexpCrawler is a crawler which use regex expressi... ============================= http://github.com/flyerhzm/sitemap/tree/master sitemap This plugin will generate a sitemap.xml from sitemap.rb whose format is very similar to routes.rb

# Sitemap

This plugin will generate a sitemap.xml or sitemap.xml.gz ... ============================= http://github.com/flyerhzm/visual\_partial/tree/master visual\_partial This plugin provides a way that you can see all the partial pages rendered. So it can prevent you from using partial page too much, which hurts the performance.

# VisualPartial

This plugin provides a way that you can see all the ... ============================= http://github.com/flyerhzm/chinese\_regions/tree/master chinese\_regions provides all chinese regions, cities and districts

# ChineseRegions

Provides all chinese regions, cities and districts\<... ============================= http://github.com/flyerhzm/chinese\_permalink/tree/master chinese\_permalink This plugin adds a capability for ar model to create a seo permalink with your chinese text. It will translate your chinese text to english url based on google translate.

# ChinesePermalink

This plugin adds a capability for ar model to cre... ============================= http://github.com/flyerhzm/codelinestatistics/tree/master codelinestatistics The code line statistics takes files and directories from GUI, counts the total files, total sizes of files, total lines, lines of codes, lines of comments and lines of blanks in the files, displays the results and can also export results to html file.

```
codelinestatistics README file:

Wha…
```