\
**Category:** Ruby\
**Created:** [August 19, 2008, 2:23pm UTC](https://www.ruby-forum.com/t/grabbing-html-from-pages-that-require-login/146251 "2008-08-19T14:23:59Z")\
**Posts on this page:** 5\
**Page:** 1
**Author:**  [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [August 19, 2008, 2:23pm UTC](https://www.ruby-forum.com/t/grabbing-html-from-pages-that-require-login/146251/1 "2008-08-19T14:23:59Z")
Hi ive searched the forums already but couldnt come up with any concrete
information.
Ive written a script that accesses certain pages on ebay and scapes
information. Its using open-uri to grab html and just standard regexps
to grab the data i want. When building the script I used ebay webpages I
saved to my harddisk. All worked fine.
However when the script attmepts to access the actual online webpages I
get an “file or directory not found …” error.
What I think is happening is that ebay is automatically redirecting the
script to the user login. Even if im already logged in via firefox i
guess the fact that the request is coming from something else outside of
firefox makes ebay suspect its from another location and thus for
security reasons requests the user to reenter their detials. Im assuming
this is to do with cookies???
So how do get round this issue? Is there a way to get avoid this login
requirement or somehow providing my script with the info it needs to
login and continue from there?
---
**Author:**  [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [August 19, 2008, 6:23pm UTC](https://www.ruby-forum.com/t/grabbing-html-from-pages-that-require-login/146251/2 "2008-08-19T18:23:03Z")
You might be able to use httpclient for this. It manages a session
much like a browser might and can track cookies that the site might
send. I was able to use this for doing some pretty ugly manipulation
of some pop-up windows that was clearly intended to be used only by a
human with a browser.
gem install httpclient
-Rob
On Aug 19, 2008, at 8:20 AM, Adam A. wrote:
> However when the script attmepts to access the actual online
> this is to do with cookies???
>
> ## So how do get round this issue? Is there a way to get avoid this login requirement or somehow providing my script with the info it needs to login and continue from there?
Rob B. [http://agileconsultingllc.com](http://agileconsultingllc.com)
removed\_email\_address@domain.invalid
---
**Author:**  [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [August 20, 2008, 5:10am UTC](https://www.ruby-forum.com/t/grabbing-html-from-pages-that-require-login/146251/3 "2008-08-20T05:10:22Z")
Thank you for that rob, ill look into it now.
---
**Author:**  [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [August 20, 2008, 5:23am UTC](https://www.ruby-forum.com/t/grabbing-html-from-pages-that-require-login/146251/4 "2008-08-20T05:23:36Z")
On Tue, Aug 19, 2008 at 12:19 PM, Rob B.
[removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> You might be able to use httpclient for this. It manages a session much
> like a browser might and can track cookies that the site might send. I was
> able to use this for doing some pretty ugly manipulation of some pop-up
> windows that was clearly intended to be used only by a human with a browser.
>
> gem install httpclient
You may also wish to look at: mechanize, scrubyt, firewatir, etc.
---
**Author:**  [@scudco](https://www.ruby-forum.com/u/scudco)\
**Post date:** [August 20, 2008, 3:34pm UTC](https://www.ruby-forum.com/t/grabbing-html-from-pages-that-require-login/146251/5 "2008-08-20T15:34:48Z")
Use mechanize, it makes logins very easy