# Scraping \<table\> from a website

**URL:** <https://www.ruby-forum.com/t/scraping-table-from-a-website/120686>\
**Category:** Ruby\
**Created:** [November 19, 2007, 7:45pm UTC](https://www.ruby-forum.com/t/scraping-table-from-a-website/120686 "2007-11-19T19:45:58Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![cskilbeck](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/c89c15/32.png) [@cskilbeck](https://www.ruby-forum.com/u/cskilbeck)\
**Post date:** [November 19, 2007, 7:45pm UTC](https://www.ruby-forum.com/t/scraping-table-from-a-website/120686/1 "2007-11-19T19:45:58Z")

</div>

Hi,

I need to extract everything between

 and 

 on a website  
(there’s only one table on the page. So far I have:

require ‘open-uri’  
page = open(‘[http://xxx.html](http://xxx.html)’).read  
page.gsub!(/\n/,“”)  
page.gsub!(/\r/,“”)  
inner = page.scan(%r{._\<table._\>(._)._}m)  
print inner

but inner is empty - any ideas?

If I substitute line 2 with

page = '123

456

789

I get inner = 456, which is correct.

---

<div class="post-metadata">

**Author:** ![cskilbeck](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/c89c15/32.png) [@cskilbeck](https://www.ruby-forum.com/u/cskilbeck)\
**Post date:** [November 19, 2007, 7:56pm UTC](https://www.ruby-forum.com/t/scraping-table-from-a-website/120686/2 "2007-11-19T19:56:29Z")

</div>

On Nov 19, 2007 1:45 PM, cskilbeck [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Hi,
> 
> I need to extract everything between
> 
> and 
> 
> on a website  
> (there’s only one table on the page. So far I have:
> 
> require ‘open-uri’  
> page = open(‘[http://xxx.html](http://xxx.html)’).read  
> page.gsub!(/\n/,“”)  
> page.gsub!(/\r/,“”)  
> inner = page.scan(%r{._\<table._\>(._)._}m)

Untested, but try:

inner = page.scan(%r{._\<table[^\>]_\>(._)._}m)

> print inner
> 
> but inner is empty - any ideas?
> 
> If I substitute line 2 with
> 
> page = '123
> 
> 456
> 
> 789
> 
> I get inner = 456, which is correct.

If you try page = ‘123

| 456 |

789’, it  
will fail again.

You only want to capture up to the next closing angle bracket. What’s  
happening is that the second .\* is matching the contents of the entire  
table, up to the closing angle bracket of the last tag (probably

) right before the , and inner gets only the leftover whitespace inbetween. So only capture characters that are NOT a closing angle bracket.

-Alex

---

<div class="post-metadata">

**Author:** ![cskilbeck](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/c89c15/32.png) [@cskilbeck](https://www.ruby-forum.com/u/cskilbeck)\
**Post date:** [November 19, 2007, 8:01pm UTC](https://www.ruby-forum.com/t/scraping-table-from-a-website/120686/3 "2007-11-19T20:01:11Z")

</div>

On Nov 19, 2007, at 3:45 PM, cskilbeck wrote:

> print inner
> 
> but inner is empty - any ideas?
> 
> If I substitute line 2 with
> 
> page = '123
> 
> 456
> 
> 789
> 
> I get inner = 456, which is correct.

use the right tools for the right job 🙂

require ‘hpricot’  
require ‘open-uri’

doc = Hpricot(open(‘[http://xxx.html](http://xxx.html)’))  
table = doc.at(‘table’)  
puts table.inner\_html

(not tested)  
regards,

---

<div class="post-metadata">

**Author:** ![cskilbeck](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/c89c15/32.png) [@cskilbeck](https://www.ruby-forum.com/u/cskilbeck)\
**Post date:** [November 19, 2007, 8:17pm UTC](https://www.ruby-forum.com/t/scraping-table-from-a-website/120686/4 "2007-11-19T20:17:02Z")

</div>

On Nov 19, 12:41 pm, cskilbeck [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> print inner
> 
> but inner is empty - any ideas?
> 
> If I substitute line 2 with
> 
> page = '123
> 
> 456
> 
> 789
> 
> I get inner = 456, which is correct.

inner = page[%r{\<table._?\>(._?)}mi, 1]

---

<div class="post-metadata">

**Author:** ![cskilbeck](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/c89c15/32.png) [@cskilbeck](https://www.ruby-forum.com/u/cskilbeck)\
**Post date:** [November 19, 2007, 10:01pm UTC](https://www.ruby-forum.com/t/scraping-table-from-a-website/120686/5 "2007-11-19T22:01:30Z")

</div>

On Nov 19, 7:14 pm, William J. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > page = open(‘[http://xxx.html](http://xxx.html)’).read
> 
> > I get inner = 456, which is correct.
> 
> inner = page[%r{\<table._?\>(._?)}mi, 1]

Thanks all for your help. non greedy matching is the key.

---

<div class="post-metadata">

**Author:** ![cskilbeck](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/c89c15/32.png) [@cskilbeck](https://www.ruby-forum.com/u/cskilbeck)\
**Post date:** [November 20, 2007, 8:15am UTC](https://www.ruby-forum.com/t/scraping-table-from-a-website/120686/6 "2007-11-20T08:15:19Z")

</div>

On Tue, 20 Nov 2007 04:00:35 +0900, Rolando A. wrote:

> require ‘hpricot’  
> require ‘open-uri’
> 
> doc = Hpricot(open(‘[http://xxx.html](http://xxx.html)’)) table = doc.at(‘table’)  
> puts table.inner\_html

Amazing – I thought that the above would be a massive project, not what  
appears to be pseudo-code! Not everything in Ruby is magically easy,  
but  
the above is pretty good 🙂

-Thufir