# How to imitate a browser?

**URL:** <https://www.ruby-forum.com/t/how-to-imitate-a-browser/195940>\
**Category:** Ruby\
**Created:** [October 3, 2010, 4:27pm UTC](https://www.ruby-forum.com/t/how-to-imitate-a-browser/195940 "2010-10-03T16:27:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![drekorian](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@drekorian](https://www.ruby-forum.com/u/drekorian)\
**Post date:** [October 3, 2010, 4:27pm UTC](https://www.ruby-forum.com/t/how-to-imitate-a-browser/195940/1 "2010-10-03T16:27:43Z")

</div>

Hey guys,

so I’m trying to write a simple script for extraction of my profile data  
from one site. In order to get the data I have to log in and then  
download a page.

At first I tried to use Ruby’s Net::HTTP library, but the problem is  
that the site runs PHP and logs users in via storing cookies (therefore  
the low-level network approach doesn’t work).

Any ideas?

---

<div class="post-metadata">

**Author:** ![drekorian](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@drekorian](https://www.ruby-forum.com/u/drekorian)\
**Post date:** [October 3, 2010, 4:43pm UTC](https://www.ruby-forum.com/t/how-to-imitate-a-browser/195940/2 "2010-10-03T16:43:25Z")

</div>

On Sun, Oct 3, 2010 at 4:28 PM, Marek O. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Any ideas?  
> Use Mechanize:

[http://mechanize.rubyforge.org/mechanize/](http://mechanize.rubyforge.org/mechanize/)

It manages cookies and allows you to fill forms, click links and so on.

Jesus.

---

<div class="post-metadata">

**Author:** ![drekorian](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@drekorian](https://www.ruby-forum.com/u/drekorian)\
**Post date:** [October 3, 2010, 9:25pm UTC](https://www.ruby-forum.com/t/how-to-imitate-a-browser/195940/3 "2010-10-03T21:25:34Z")

</div>

Hi Marek,

Have you tried the combo Nokogiri ([http://nokogiri.org/](http://nokogiri.org/)) / Mechanize  
([http://mechanize.rubyforge.org/mechanize/](http://mechanize.rubyforge.org/mechanize/)) to do the job?

I recommend you to take a look at:

- [#190 Screen Scraping with Nokogiri - RailsCasts](http://railscasts.com/episodes/190-screen-scraping-with-nokogiri)
- [#191 Mechanize - RailsCasts](http://railscasts.com/episodes/191-mechanize)  
Adriano

* * *

From: Marek O. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
To: ruby-talk ML [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
Sent: Sun, October 3, 2010 11:28:38 AM  
Subject: How to imitate a browser?

Hey guys,

so I’m trying to write a simple script for extraction of my profile data  
from one site. In order to get the data I have to log in and then  
download a page.

At first I tried to use Ruby’s Net::HTTP library, but the problem is  
that the site runs PHP and logs users in via storing cookies (therefore  
the low-level network approach doesn’t work).

Any ideas?

---

<div class="post-metadata">

**Author:** ![drekorian](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/f4b2a3/32.png) [@drekorian](https://www.ruby-forum.com/u/drekorian)\
**Post date:** [October 4, 2010, 7:45pm UTC](https://www.ruby-forum.com/t/how-to-imitate-a-browser/195940/4 "2010-10-04T19:45:49Z")

</div>

> so I’m trying to write a simple script for extraction of my profile data  
> from one site.

> **[Category: HTTP clients - The Ruby Toolbox](https://www.ruby-toolbox.com/categories/http_clients.html)**
>
> Explore and compare open source Ruby libraries

[Category: Browser testing - The Ruby Toolbox](http://ruby-toolbox.com/categories/browser_testing.html) (watir,  
especially)