Forum: Ruby Screen scraping an html text contents into a file

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Acdabad15b58ba105df230e3ea02523d?d=identicon&s=25 basi_lio (Guest)
on 2005-12-06 18:18
(Received via mailing list)
Hello,

I'm looking for a screen scraper that will extract text contents off
html pages and save the text into files. I have looked at Mechanize and
Rubyful_Soup, but they are a bit over my head to modify to save just
the text contents to a file. (I'm a researcher trying to use Ruby for
real world text analysis tasks, and trying to learn Ruby at the same
time.)  The levels of usage I'd love to have (choosey beggar):

> program prompts me for url address to scrape and file name to save texts into,
> or edit program to enter url address and file name

Of course a program that, given a url, would walk down the links, open
the pages, and save the text contents to a file would be ... that would
be a commercial product. Is there one?

Thanks!
basi
Ce60c4f78a63b0695e4dafc4bd7964f7?d=identicon&s=25 vanek (Guest)
on 2005-12-06 19:23
(Received via mailing list)
you might get away with just using curl:

	curl www.apple.com > mytextfile

or wget, which is capable of acting recursively on an entire site.
	http://www.delorie.com/gnu/docs/wget/wget_14.html
6b4566518f6675477dab9b8ba813cf3c?d=identicon&s=25 ruby.brian (Guest)
on 2005-12-06 22:02
(Received via mailing list)
On 06/12/05, basi <basi_lio@hotmail.com> wrote:
> > or edit program to enter url address and file name
>
> Of course a program that, given a url, would walk down the links, open
> the pages, and save the text contents to a file would be ... that would
> be a commercial product. Is there one?
>
> Thanks!
> basi
>
>
>

try this:

$ w3m -dump www.ruby-lang.org

cheers,

Brian

--
http://ruby.brian-schroeder.de/

Stringed instrument chords: http://chordlist.brian-schroeder.de/
9dfe8c734b0f9b37a4e218425c0a2138?d=identicon&s=25 gene.tani (Guest)
on 2005-12-06 22:06
(Received via mailing list)
basi wrote:
> > or edit program to enter url address and file name
>
> Of course a program that, given a url, would walk down the links, open
> the pages, and save the text contents to a file would be ... that would
> be a commercial product. Is there one?
>
> Thanks!
> basi

i think open-uri and Rubyful_soup are pretty straightforward.  I like
this shows open-uri vs. Net::HTTP:
http://www.zenspider.com/dl/rubyconf2005/open-uri.pdf

There's commercial website downloaders that will follow every link in
every page, hit the server hundreds of times in a few seconds and get
your IP blacklisted pretty quickly (so run them from Starbuck's
wireless).  Look in Oreilly Spidering Hacks, for the right way to do
it. the (perl) examples are straightforward.
036a1b88dafaab8ffd73a8b0a74b5b38?d=identicon&s=25 ef (Guest)
on 2005-12-06 22:23
(Received via mailing list)
> basi wrote:
> > Of course a program that, given a url, would walk down the links, open
> > the pages, and save the text contents to a file would be ... that would
> > be a commercial product. Is there one?

No need for a commercial product.  wget does all that.
Acdabad15b58ba105df230e3ea02523d?d=identicon&s=25 basi_lio (Guest)
on 2005-12-07 00:21
(Received via mailing list)
Hi,
Thanks for the info on wget and curl. Both are powerful page
downloaders. The downloaded pages are still "tagged". I need to find a
way to "run" the pages and capture only the text display.
Thanks again.
basi
38a8230ed3d5c685558b4f0aad3fc74b?d=identicon&s=25 joevandyk (Guest)
on 2005-12-07 00:45
(Received via mailing list)
On 12/6/05, basi <basi_lio@hotmail.com> wrote:
> Hi,
> Thanks for the info on wget and curl. Both are powerful page
> downloaders. The downloaded pages are still "tagged". I need to find a
> way to "run" the pages and capture only the text display.

$ lynx -dump www.rubystuff.com

Ruby Stuff. The Ruby Store for Ruby programmers.

Got the Right Stuff?

   The Ruby Stuff store is one-stop shopping for assorted Ruby hacker
   goodness.

   T-shirts, hats, coffee mugs, clocks, mouse pads, and more.

Shirts

   Ruby Stuff offers a nifty variety of stunning T-shirts and jerseys
for
   men and women. You'll feel naked without one.

   [1]More ...

Coffee Mugs

   Hackers + caffeine = max coding pleasure. Drink your beverage of
   choice from one of these mugs.

   [2]More ...

RubyStuff now has Stamps!

   These first-class U. S. postage stamps won't make the mail go any
   faster, but they are sure to raise an eyebrow.

   [3]More ...

Stuff

   There's yet more Ruby stuff: Knock-out clocks; mighty mouse pads,
   handsome hats.

   [4]Mouse pads, [5]bags, [6]undies, [7]hats, [8]buttons, [9]more

   [10]About RubyStuff.com ...

References

   1. http://www.rubystuff.com/shirts.html
   2. http://www.rubystuff.com/mugs.html
   3. http://www.rubystuff.com/stamps.html
   4. http://www.rubystuff.com/mousepads.html
   5. http://www.rubystuff.com/bags.html
   6. http://www.rubystuff.com/undies.html
   7. http://www.rubystuff.com/hats.html
   8. http://www.rubystuff.com/buttons_and_magnets.html
   9. http://www.rubystuff.com/assorted.html
  10. http://www.rubystuff.com/about.html
Acdabad15b58ba105df230e3ea02523d?d=identicon&s=25 basi_lio (Guest)
on 2005-12-07 01:06
(Received via mailing list)
Can't find a w3m binaries for windows xp. I'll continue to look.
Thanks,
basi
Acdabad15b58ba105df230e3ea02523d?d=identicon&s=25 basi_lio (Guest)
on 2005-12-07 01:18
(Received via mailing list)
Hello,
Can't find windows xp binaries of w3m, snarf, also tried cUrl, wget,
but lynx does look like it renders the page close to what I'm looking
for.
Thanks to all who responded!
basi
27d5dabc3e46673e9859405459d0cef3?d=identicon&s=25 sjc2000_uk (Guest)
on 2005-12-07 05:33
(Received via mailing list)
I will throw something like this together in Ruby over
the next days when I get some time and post it on
RubyForge. I have already done this sort of stuff in
Java and the concepts just really need a port.All we
are looking at Basi's initial level of requirements is
to send an HTTP get, and pipe the response to a file.

Link following is a little more tricky since you need
to parse the HTML, issue a get, pipe the file, rinse
and repeat, but again not exactly rocket science.

rgds

Steve

--- basi <basi_lio@hotmail.com> wrote:

>
__________________________________________
Yahoo! DSL ? Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com
2cf6d8e639314abd751f83a72e9a2ac5?d=identicon&s=25 martindemello (Guest)
on 2005-12-07 09:54
(Received via mailing list)
Steve Callaway <sjc2000_uk@yahoo.com> wrote:
> I will throw something like this together in Ruby over
> the next days when I get some time and post it on
> RubyForge. I have already done this sort of stuff in
> Java and the concepts just really need a port.All we
> are looking at Basi's initial level of requirements is
> to send an HTTP get, and pipe the response to a file.

Nope, according to the OP's requirements, you also need to render the
html and spit out the rendered version as text, which makes lynx --dump
the right tool for the job. It'd be quite a big task to duplicate this
in ruby, I think.

martin
27d5dabc3e46673e9859405459d0cef3?d=identicon&s=25 sjc2000_uk (Guest)
on 2005-12-07 10:06
(Received via mailing list)
--- Martin DeMello <martindemello@yahoo.com> wrote:

> > to send an HTTP get, and pipe the response to a
> martin
>
>

By rendering the html, my interpretation of this was
that it is merely a question of stripping tags etc,
which can quickly be accomplished with gsub. Or am I
missing something?

rgds

Steve



__________________________________________
Yahoo! DSL ? Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com
6b4566518f6675477dab9b8ba813cf3c?d=identicon&s=25 ruby.brian (Guest)
on 2005-12-07 11:56
(Received via mailing list)
On 07/12/05, Steve Callaway <sjc2000_uk@yahoo.com> wrote:
> > > Java and the concepts just really need a port.All
> > the right tool for the job. It'd be quite a big task
> missing something?
>
> rgds
>
> Steve
>

E.g. Tables and frames. So better use links2 or w3m for the task.

cheers,

Brian



--
http://ruby.brian-schroeder.de/

Stringed instrument chords: http://chordlist.brian-schroeder.de/
27d5dabc3e46673e9859405459d0cef3?d=identicon&s=25 sjc2000_uk (Guest)
on 2005-12-07 13:38
(Received via mailing list)
Ah, yeah, forgot all about those nasty little things.
Not insuperable but would certainly add an overhead to
handle them effectively.

Steve

--- Brian Schröder <ruby.brian@gmail.com> wrote:

> > > over
> > > > to send an HTTP get, and pipe the response to
> > > to duplicate this
> > which can quickly be accomplished with gsub. Or am
>
> http://chordlist.brian-schroeder.de/
>
>




__________________________________________
Yahoo! DSL ? Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com
2cf6d8e639314abd751f83a72e9a2ac5?d=identicon&s=25 martindemello (Guest)
on 2005-12-07 14:40
(Received via mailing list)
Steve Callaway <sjc2000_uk@yahoo.com> wrote:
> >
>
> By rendering the html, my interpretation of this was
> that it is merely a question of stripping tags etc,
> which can quickly be accomplished with gsub. Or am I
> missing something?

Even without things like tables, the significance of various whitespace
elements (space, tab, newline) in html is very different from its
significance in the rendered page. e.g. the following can't be done by
just stripping tags:

<ul><li>one
two</li><li>three<li>four</ul>

martin
4b174722d1b1a4bbd9672e1ab50c30a9?d=identicon&s=25 leavengood (Guest)
on 2005-12-07 17:47
(Received via mailing list)
WWW::Mechanize can do most of what is needed, except for the dumping
of the HTML as text. As others have said, what we really need is some
kind of HTML to text renderer. There has got to be gobs of C or C++
code out there that does this...how hard would it be to make a Ruby C
extension for this? Hash anyone ever thought about making a nice Ruby
extension for Gecko or even the HTML renderers in lynx or w3m?

Ryan
This topic is locked and can not be replied to.