# Extract information from a large text

**URL:** <https://www.ruby-forum.com/t/extract-information-from-a-large-text/130930>\
**Category:** Ruby\
**Created:** [March 4, 2008, 4:05pm UTC](https://www.ruby-forum.com/t/extract-information-from-a-large-text/130930 "2008-03-04T16:05:42Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![phoenix](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/0ea827/32.png) [@phoenix](https://www.ruby-forum.com/u/phoenix)\
**Post date:** [March 4, 2008, 4:05pm UTC](https://www.ruby-forum.com/t/extract-information-from-a-large-text/130930/1 "2008-03-04T16:05:42Z")

</div>

I 'm new to ruby.I want to extract some useful information from a web  
page to generate a RSS feeds.My first instinct is to provide a regular  
expression like /sometext(.+?)sometext/, the problem is I can only get  
the first match to this regex, how can I iterate over the multiple  
matches?  
And further more, this kind of naive solution, is it too slow to  
search over a very large text?Because the performance requirement is  
high.Is there a better way to do this?

---

<div class="post-metadata">

**Author:** ![phoenix](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/0ea827/32.png) [@phoenix](https://www.ruby-forum.com/u/phoenix)\
**Post date:** [March 4, 2008, 5:00pm UTC](https://www.ruby-forum.com/t/extract-information-from-a-large-text/130930/2 "2008-03-04T17:00:12Z")

</div>

On Wed, 5 Mar 2008 00:04:52 +0900, phoenix [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> I 'm new to ruby.I want to extract some useful information from a web  
> page to generate a RSS feeds.My first instinct is to provide a regular  
> expression like /sometext(.+?)sometext/, the problem is I can only get  
> the first match to this regex, how can I iterate over the multiple  
> matches?  
> And further more, this kind of naive solution, is it too slow to  
> search over a very large text?Because the performance requirement is  
> high.Is there a better way to do this?  
> –

I recommend that you use the hpricot gem with lets you use xpath  
expressions on html.

Install by typing this into a command line:

```
gem install hpricot

```

Here’s a little example that extracts the link texts from a google  
search:

require ‘rubygems’  
require ‘open-uri’  
require ‘hpricot’

g = Hpricot(open(“[hpricot xpath - Google Search](http://www.google.com/search?q=hpricot%20xpath)”))  
(g/“a[@class=‘l’]”).each { |hit|  
puts “#{(hit/“text()”)}”  
}  
nil

```
    Kristian

```

---

<div class="post-metadata">

**Author:** ![phoenix](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/0ea827/32.png) [@phoenix](https://www.ruby-forum.com/u/phoenix)\
**Post date:** [March 4, 2008, 7:00pm UTC](https://www.ruby-forum.com/t/extract-information-from-a-large-text/130930/3 "2008-03-04T19:00:46Z")

</div>

On 04.03.2008 16:04, phoenix wrote:

> I 'm new to ruby.I want to extract some useful information from a web  
> page to generate a RSS feeds.My first instinct is to provide a regular  
> expression like /sometext(.+?)sometext/, the problem is I can only get  
> the first match to this regex, how can I iterate over the multiple  
> matches?

String#scan.

> And further more, this kind of naive solution, is it too slow to  
> search over a very large text?Because the performance requirement is  
> high.Is there a better way to do this?

Try it out. Also Hpricot like Kristian suggested.

Cheers

robert