# Accessing dynamic javasript with Ruby

**URL:** <https://www.ruby-forum.com/t/accessing-dynamic-javasript-with-ruby/181280>\
**Category:** Rails\
**Created:** [January 8, 2010, 6:01pm UTC](https://www.ruby-forum.com/t/accessing-dynamic-javasript-with-ruby/181280 "2010-01-08T18:01:41Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dr\_gavin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c37758/32.png) [@dr\_gavin](https://www.ruby-forum.com/u/dr_gavin)\
**Post date:** [January 8, 2010, 6:01pm UTC](https://www.ruby-forum.com/t/accessing-dynamic-javasript-with-ruby/181280/1 "2010-01-08T18:01:41Z")

</div>

Hey all

I’m experimenting with writing a scraper at the moment and have hit a  
major hump.

Part of the DOM is added after the page has loaded via javascript.

This means when I make an a request the HTML response I receive back  
doesn’t accurately represent the page.

Here’s a simplified example:

@http\_obj = Net::HTTP.new(“[targetdomain.com](http://targetdomain.com)”)

response, page\_data = @http\_obj.request\_get( “/” )

# page data doesn’t contain all of the HTML that is actually shown

Is there anyway library or gem that could simulate the browser  
updating the DOM with the Javascript or any other way I could approach  
this short of decoding the obfuscated Javascript file?

Thanks in advance

Gav

---

<div class="post-metadata">

**Author:** ![dr\_gavin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c37758/32.png) [@dr\_gavin](https://www.ruby-forum.com/u/dr_gavin)\
**Post date:** [January 14, 2010, 12:01am UTC](https://www.ruby-forum.com/t/accessing-dynamic-javasript-with-ruby/181280/2 "2010-01-14T00:01:46Z")

</div>

Gavin M. wrote:

> Hey all
> 
> I’m experimenting with writing a scraper at the moment and have hit a  
> major hump.
> 
> Part of the DOM is added after the page has loaded via javascript.
> 
> This means when I make an a request the HTML response I receive back  
> doesn’t accurately represent the page.
> 
> Here’s a simplified example:
> 
> @http\_obj = Net::HTTP.new(“[targetdomain.com](http://targetdomain.com)”)
> 
> response, page\_data = @http\_obj.request\_get( “/” )
> 
> # page data doesn’t contain all of the HTML that is actually shown
> 
> Is there anyway library or gem that could simulate the browser  
> updating the DOM with the Javascript or any other way I could approach  
> this short of decoding the obfuscated Javascript file?

Try Selenium or some other remote browser control.

> Thanks in advance
> 
> Gav

## Best,

Marnen Laibow-Koser  
[http://www.marnen.org](http://www.marnen.org)  
removed\_email\_address@domain.invalid