Forum: Ruby scrape data from Javascript inside HTML source

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Yaxm Y. (Guest)
on 2009-04-11 09:09
Hi,
is there any scraper that interact with Javascript function inside a
HTML page?  Sometimes, the data is returned from a  Javascript function
or a javascript variable.  so I wonder if there's a easy way to get data
out by evaluating and evaluating the javascript based on the context of
the page in Ruby?

Yaxm
Phlip (Guest)
on 2009-04-11 09:20
(Received via mailing list)
Yaxm Y. wrote:

> is there any scraper that interact with Javascript function inside a
> HTML page?  Sometimes, the data is returned from a  Javascript function
> or a javascript variable.  so I wonder if there's a easy way to get data
> out by evaluating and evaluating the javascript based on the context of
> the page in Ruby?

In test? Or "scraping" a target website to see what it's got?

Either way, I would use Nokogiri to rip the HTML and find <script> tags,
then
use racc and rkelly to interpret the JavaScript and find its variables.

By "would" I mean I already _do_ that. Here's the rkelly calls required:

       RKelly.parse(js).pointcut('TargetMethod()').  # with the ()
           matches.each do |updater|
         updater.grep(RKelly::Nodes::ArgumentsNode).each do |thang|
           p thang
         end
       end

However, if you are attacking other peoples' websites to scrape out
their data,
you might instead try Watir. It just runs a web browser and evaluates
its JS
directly.
This topic is locked and can not be replied to.