Forum: Ruby Using an existing session to grab a page

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
16891a11b3c6e8b08dd738471e1fa3f1?d=identicon&s=25 Chad Layton (Guest)
on 2005-12-25 23:55
(Received via mailing list)
I'm rather new to both web programming and ruby so forgive me if my
question is ill formed.

I'm trying to do some screen scraping on a website that requires a login
  . What I would like to have happen is for the user to login to the
website normally, then run my script which uses the existing login
session to grab the page and do whatever to it.

To illustrate my problem: If I use
Net::HTTP.get_response(URI.parse("http://foo.bar/baz.php")).body, then
it  serves up the index asking for a login. How do I get contents of
baz.php?
Bc6d88907ce09158581fbb9b469a35a3?d=identicon&s=25 James Britt (Guest)
on 2005-12-26 00:37
(Received via mailing list)
Chad Layton wrote:
> it  serves up the index asking for a login. How do I get contents of
> baz.php?

I suspect that the user agent (i.e., the code, as opposed to a browser)
needs to include site cookies in the request headers.

After you sign in using a browser, you'll need to find the cookie left
by the site, or inspect a session cookie if the browser is not writing
it to disk.  Most browsers have a way to show cookies sent by a site.



James


--

http://www.ruby-doc.org       - Ruby Help & Documentation
http://www.artima.com/rubycs/ - Ruby Code & Style: Writers wanted
http://www.rubystuff.com      - The Ruby Store for Ruby Stuff
http://www.jamesbritt.com     - Playing with Better Toys
http://www.30secondrule.com   - Building Better Tools
7c3a0706ce454a5e7c405a63fdcd2c21?d=identicon&s=25 Jim (Guest)
on 2005-12-26 05:44
(Received via mailing list)
Don't if this'll help, but www-mechanize is able to login to a site
that is using a html form to login.

You don't need to use a browser at all.

http://www.ntecs.de/blog/Blog/WWW-Mechanize.rdoc
16891a11b3c6e8b08dd738471e1fa3f1?d=identicon&s=25 Chad Layton (Guest)
on 2005-12-26 10:23
(Received via mailing list)
James Britt wrote:
>> To illustrate my problem: If I use
> it to disk.  Most browsers have a way to show cookies sent by a site.
>
>
>
> James
>
>

Thank you, James. I see that when I login to the site 4 cookies are set,
   how would I include them in the request headers?
Bc6d88907ce09158581fbb9b469a35a3?d=identicon&s=25 James Britt (Guest)
on 2005-12-29 00:30
(Received via mailing list)
Chad Layton wrote:
>>> login session to grab the page and do whatever to it.
>>
> Thank you, James. I see that when I login to the site 4 cookies are set,
>   how would I include them in the request headers?

I *think* you pass a hash into the Net::HTTP initializer, or perhaps as
a parameter to 'get' but I can't find docs or examples to prove this.



James
>
>


--

http://www.ruby-doc.org       - Ruby Help & Documentation
http://www.artima.com/rubycs/ - Ruby Code & Style: Writers wanted
http://www.rubystuff.com      - The Ruby Store for Ruby Stuff
http://www.jamesbritt.com     - Playing with Better Toys
http://www.30secondrule.com   - Building Better Tools
This topic is locked and can not be replied to.