Forum: Ruby Using an existing session to grab a page

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Chad Layton (Guest)
on 2005-12-26 00:55
(Received via mailing list)
I'm rather new to both web programming and ruby so forgive me if my
question is ill formed.

I'm trying to do some screen scraping on a website that requires a login
  . What I would like to have happen is for the user to login to the
website normally, then run my script which uses the existing login
session to grab the page and do whatever to it.

To illustrate my problem: If I use
Net::HTTP.get_response(URI.parse("http://foo.bar/baz.php")).body, then
it  serves up the index asking for a login. How do I get contents of
baz.php?
James B. (Guest)
on 2005-12-26 01:37
(Received via mailing list)
Chad Layton wrote:
> it  serves up the index asking for a login. How do I get contents of
> baz.php?

I suspect that the user agent (i.e., the code, as opposed to a browser)
needs to include site cookies in the request headers.

After you sign in using a browser, you'll need to find the cookie left
by the site, or inspect a session cookie if the browser is not writing
it to disk.  Most browsers have a way to show cookies sent by a site.



James


--

http://www.ruby-doc.org       - Ruby Help & Documentation
http://www.artima.com/rubycs/ - Ruby Code & Style: Writers wanted
http://www.rubystuff.com      - The Ruby Store for Ruby Stuff
http://www.jamesbritt.com     - Playing with Better Toys
http://www.30secondrule.com   - Building Better Tools
Jim (Guest)
on 2005-12-26 06:44
(Received via mailing list)
Don't if this'll help, but www-mechanize is able to login to a site
that is using a html form to login.

You don't need to use a browser at all.

http://www.ntecs.de/blog/Blog/WWW-Mechanize.rdoc
Chad Layton (Guest)
on 2005-12-26 11:23
(Received via mailing list)
James B. wrote:
>> To illustrate my problem: If I use
> it to disk.  Most browsers have a way to show cookies sent by a site.
>
>
>
> James
>
>

Thank you, James. I see that when I login to the site 4 cookies are set,
   how would I include them in the request headers?
James B. (Guest)
on 2005-12-29 01:30
(Received via mailing list)
Chad Layton wrote:
>>> login session to grab the page and do whatever to it.
>>
> Thank you, James. I see that when I login to the site 4 cookies are set,
>   how would I include them in the request headers?

I *think* you pass a hash into the Net::HTTP initializer, or perhaps as
a parameter to 'get' but I can't find docs or examples to prove this.



James
>
>


--

http://www.ruby-doc.org       - Ruby Help & Documentation
http://www.artima.com/rubycs/ - Ruby Code & Style: Writers wanted
http://www.rubystuff.com      - The Ruby Store for Ruby Stuff
http://www.jamesbritt.com     - Playing with Better Toys
http://www.30secondrule.com   - Building Better Tools
This topic is locked and can not be replied to.