(.\*?)\
/mx
page.scan(row) { |e|
rowtext = e.to\_s
rowtext.scan(milecells) {
miletable \<\< [$1,$2,$3,$4,$5]
}
rowtext.scan(summarycells) {
summarytable \<\< [$1,$2]
}
rowtext.scan(activitycells) {
summarytable \<\< [$1,$3]
}
}
table\_clean(miletable)
table\_clean(summarytable)
miletable.each {|e| print e.join(“:”),“\n”}
summarytable.each {|e| print e.join(“:”),“\n”}
```
- dan
```
---
**Author:**  [@dan](https://www.ruby-forum.com/u/dan)\
**Post date:** [December 2, 2005, 7:14pm UTC](https://www.ruby-forum.com/t/screen-scraping-via-regex-vs-htmltools-vs-rexml/49333/2 "2005-12-02T19:14:37Z")
Dan K. wrote:
> I’ve finally reimplemented the screen scraper I mentioned on
> [http://groups.google.com/group/comp.lang.ruby/browse\_thread/thread/76e8bbd4a9e48277/396cb7ea35eab14f#396cb7ea35eab14f](http://groups.google.com/group/comp.lang.ruby/browse_thread/thread/76e8bbd4a9e48277/396cb7ea35eab14f#396cb7ea35eab14f)
> using regexes and no external libraries. It is, as Daz suggested, many
> times faster than REXML. My question is whether it would be smarter
> (faster?, easier to code?) to use htmltools or HTMLTree::Parser
> instead.
The code in your post seems to use Mechanize.
If you are using agent.get to fetch the HTML then you’ve already parsed
the html using htmltools & REXML. You can register callback objects
that are invoked when the parsing process encounters matching nodes.
Mechanize does this automatically for certain nodes (form stuff, I
think), but you can use watch\_for\_set= {} to define a set of nodes to
watch for.
This is what I use to construct the product pages for [rubystuff.com](http://rubystuff.com) from
the multiple CafePress pages that contain the images, prices, and
product description. I tell Mechanize to watch for img, tr, and td
elements, and it constructs sets of custom objects of just the parts of
the source HTML matching certain criteria. Then I extract the data,
create RSS feeds, and turn those into a set of aggregated HTML pages.
What I like about this is that the parse process gives me business
objects, with (hopefully) self-explanatory behavior. For example, I can
ask one of these objects for ‘product\_id’ or ‘description’; the object
encapsulates the assorted XPath/regex code needed to get that from the
source HTML node, making the main part of the app easier to maintain.
James B.
–
[http://www.ruby-doc.org](http://www.ruby-doc.org) - Ruby Help & Documentation
[Ruby Code & Style](http://www.artima.com/rubycs/) - Ruby Code & Style: Writers wanted
[http://www.rubystuff.com](http://www.rubystuff.com) - The Ruby Store for Ruby Stuff
[http://www.jamesbritt.com](http://www.jamesbritt.com) - Playing with Better Toys
[http://www.30secondrule.com](http://www.30secondrule.com) - Building Better Tools
---
**Author:**  [@dan](https://www.ruby-forum.com/u/dan)\
**Post date:** [December 2, 2005, 7:59pm UTC](https://www.ruby-forum.com/t/screen-scraping-via-regex-vs-htmltools-vs-rexml/49333/3 "2005-12-02T19:59:45Z")
Thanks for the response, James. My next question was actually about
debugging Mechanize
[http://groups.google.com/group/comp.lang.ruby/msg/04fc7473b08c16fc](http://groups.google.com/group/comp.lang.ruby/msg/04fc7473b08c16fc).
Would you mind emailing me your scraping code, as I’ve been suffering
from a lack of examples to copy?
Also, are you sure Mechanize parses the whole page with get? It
doesn’t wait for a find?
```
- dan
```
---
**Author:**  [@dan](https://www.ruby-forum.com/u/dan)\
**Post date:** [December 2, 2005, 8:07pm UTC](https://www.ruby-forum.com/t/screen-scraping-via-regex-vs-htmltools-vs-rexml/49333/4 "2005-12-02T20:07:52Z")
“James B.” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote in message
> This is what I use to construct the product pages for [rubystuff.com](http://rubystuff.com) from
Any chance you could make that code available? Sounds like a useful
example.
Is Mechanize also a good option for writing acceptance tests, compared
to
Watir?
Thanks.
---
**Author:**  [@dan](https://www.ruby-forum.com/u/dan)\
**Post date:** [December 2, 2005, 9:17pm UTC](https://www.ruby-forum.com/t/screen-scraping-via-regex-vs-htmltools-vs-rexml/49333/5 "2005-12-02T21:17:24Z")
itsme213 wrote:
>
WATIR exposes the HTML DOM as seen by IE, which is not the raw HTML
source returned from the server (but perhaps someone more up on the
latest WATIR knows otherwise). Mechanize will get you the source HTML,
albeit sanitized for REXML parsing.
I find WATIR most useful for walking though a series of pages where
automated typing and clicking is essential. Pretty much every Web app
I’ve written in the last 9 months uses WATIR (plus my own custom DSL on
top of it) for functional testing. Major time saver.
I use Mechanize for data snarfing and occasional feed building.
James B.
–
[http://www.ruby-doc.org](http://www.ruby-doc.org) - Ruby Help & Documentation
[Ruby Code & Style](http://www.artima.com/rubycs/) - Ruby Code & Style: Writers wanted
[http://www.rubystuff.com](http://www.rubystuff.com) - The Ruby Store for Ruby Stuff
[http://www.jamesbritt.com](http://www.jamesbritt.com) - Playing with Better Toys
[http://www.30secondrule.com](http://www.30secondrule.com) - Building Better Tools
---
**Author:**  [@dan](https://www.ruby-forum.com/u/dan)\
**Post date:** [December 2, 2005, 9:09pm UTC](https://www.ruby-forum.com/t/screen-scraping-via-regex-vs-htmltools-vs-rexml/49333/6 "2005-12-02T21:09:09Z")
Dan K. wrote:
> Thanks for the response, James. My next question was actually about
> debugging Mechanize
> [http://groups.google.com/group/comp.lang.ruby/msg/04fc7473b08c16fc](http://groups.google.com/group/comp.lang.ruby/msg/04fc7473b08c16fc).
> Would you mind emailing me your scraping code, as I’ve been suffering
> from a lack of examples to copy?
>
> Also, are you sure Mechanize parses the whole page with get? It
> doesn’t wait for a find?
Don’t think so, but I might be wrong. My code calls agent get, then
goes right into looping over the collected nodes.
I’ll see about putting my code together as an example.
As for debugging Mechanize, I’ve found it helpful to go to the lib
source and stick in some STDERR.puts calls to inspect request and
response data to be sure things are getting passed around as expected.
After that, unit tests are helpful.
## James
[http://www.ruby-doc.org](http://www.ruby-doc.org) - Ruby Help & Documentation
[Ruby Code & Style](http://www.artima.com/rubycs/) - Ruby Code & Style: Writers wanted
[http://www.rubystuff.com](http://www.rubystuff.com) - The Ruby Store for Ruby Stuff
[http://www.jamesbritt.com](http://www.jamesbritt.com) - Playing with Better Toys
[http://www.30secondrule.com](http://www.30secondrule.com) - Building Better Tools