# Using an existing session to grab a page

**URL:** <https://www.ruby-forum.com/t/using-an-existing-session-to-grab-a-page/51087>\
**Category:** Ruby\
**Created:** [December 25, 2005, 11:55pm UTC](https://www.ruby-forum.com/t/using-an-existing-session-to-grab-a-page/51087 "2005-12-25T23:55:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chad\_Layton](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/48db29/32.png) [@Chad\_Layton](https://www.ruby-forum.com/u/Chad_Layton)\
**Post date:** [December 25, 2005, 11:55pm UTC](https://www.ruby-forum.com/t/using-an-existing-session-to-grab-a-page/51087/1 "2005-12-25T23:55:49Z")

</div>

I’m rather new to both web programming and ruby so forgive me if my  
question is ill formed.

I’m trying to do some screen scraping on a website that requires a login  
. What I would like to have happen is for the user to login to the  
website normally, then run my script which uses the existing login  
session to grab the page and do whatever to it.

To illustrate my problem: If I use  
Net::HTTP.get\_response(URI.parse(“[http://foo.bar/baz.php](http://foo.bar/baz.php)”)).body, then  
it serves up the index asking for a login. How do I get contents of  
baz.php?

---

<div class="post-metadata">

**Author:** ![Chad\_Layton](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/48db29/32.png) [@Chad\_Layton](https://www.ruby-forum.com/u/Chad_Layton)\
**Post date:** [December 26, 2005, 12:37am UTC](https://www.ruby-forum.com/t/using-an-existing-session-to-grab-a-page/51087/2 "2005-12-26T00:37:54Z")

</div>

Chad Layton wrote:

> it serves up the index asking for a login. How do I get contents of  
> baz.php?

I suspect that the user agent (i.e., the code, as opposed to a browser)  
needs to include site cookies in the request headers.

After you sign in using a browser, you’ll need to find the cookie left  
by the site, or inspect a session cookie if the browser is not writing  
it to disk. Most browsers have a way to show cookies sent by a site.

James

–

[http://www.ruby-doc.org](http://www.ruby-doc.org) - Ruby Help & Documentation  
[Ruby Code & Style](http://www.artima.com/rubycs/) - Ruby Code & Style: Writers wanted  
[http://www.rubystuff.com](http://www.rubystuff.com) - The Ruby Store for Ruby Stuff  
[http://www.jamesbritt.com](http://www.jamesbritt.com) - Playing with Better Toys  
[http://www.30secondrule.com](http://www.30secondrule.com) - Building Better Tools

---

<div class="post-metadata">

**Author:** ![Chad\_Layton](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/48db29/32.png) [@Chad\_Layton](https://www.ruby-forum.com/u/Chad_Layton)\
**Post date:** [December 26, 2005, 5:44am UTC](https://www.ruby-forum.com/t/using-an-existing-session-to-grab-a-page/51087/3 "2005-12-26T05:44:19Z")

</div>

Don’t if this’ll help, but www-mechanize is able to login to a site  
that is using a html form to login.

You don’t need to use a browser at all.

[http://www.ntecs.de/blog/Blog/WWW-Mechanize.rdoc](http://www.ntecs.de/blog/Blog/WWW-Mechanize.rdoc)

---

<div class="post-metadata">

**Author:** ![Chad\_Layton](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/48db29/32.png) [@Chad\_Layton](https://www.ruby-forum.com/u/Chad_Layton)\
**Post date:** [December 26, 2005, 10:23am UTC](https://www.ruby-forum.com/t/using-an-existing-session-to-grab-a-page/51087/4 "2005-12-26T10:23:43Z")

</div>

James B. wrote:

> > To illustrate my problem: If I use  
> > it to disk. Most browsers have a way to show cookies sent by a site.
> 
> James

Thank you, James. I see that when I login to the site 4 cookies are set,  
how would I include them in the request headers?

---

<div class="post-metadata">

**Author:** ![Chad\_Layton](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/48db29/32.png) [@Chad\_Layton](https://www.ruby-forum.com/u/Chad_Layton)\
**Post date:** [December 29, 2005, 12:30am UTC](https://www.ruby-forum.com/t/using-an-existing-session-to-grab-a-page/51087/5 "2005-12-29T00:30:54Z")

</div>

Chad Layton wrote:

> > > login session to grab the page and do whatever to it.
> 
> Thank you, James. I see that when I login to the site 4 cookies are set,  
> how would I include them in the request headers?

I _think_ you pass a hash into the Net::HTTP initializer, or perhaps as  
a parameter to ‘get’ but I can’t find docs or examples to prove this.

James

> 

–

[http://www.ruby-doc.org](http://www.ruby-doc.org) - Ruby Help & Documentation  
[Ruby Code & Style](http://www.artima.com/rubycs/) - Ruby Code & Style: Writers wanted  
[http://www.rubystuff.com](http://www.rubystuff.com) - The Ruby Store for Ruby Stuff  
[http://www.jamesbritt.com](http://www.jamesbritt.com) - Playing with Better Toys  
[http://www.30secondrule.com](http://www.30secondrule.com) - Building Better Tools