# Parsing problems using https and redirects

**URL:** <https://www.ruby-forum.com/t/parsing-problems-using-https-and-redirects/123216>\
**Category:** Ruby\
**Created:** [December 14, 2007, 11:58pm UTC](https://www.ruby-forum.com/t/parsing-problems-using-https-and-redirects/123216 "2007-12-14T23:58:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ramiro\_Diaz\_Trepat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/3ab097/32.png) [@Ramiro\_Diaz\_Trepat](https://www.ruby-forum.com/u/Ramiro_Diaz_Trepat)\
**Post date:** [December 14, 2007, 11:58pm UTC](https://www.ruby-forum.com/t/parsing-problems-using-https-and-redirects/123216/1 "2007-12-14T23:58:31Z")

</div>

Hello list,  
I have to develop a simple script to parse some parts of a web site  
and I  
thought it could be a good opportunity to start trying Ruby.  
I found that there are two network libraries that I could supposedly  
use  
to retrieve the contents of the web site: open-uri and net-http.

_First problem_  
This web site is accessed only with https and has a self issued  
certificate. This has made it impossible so far for me to access the  
contents of the web site.  
Simple examples from the Hpricot html parsing library like this one:

require ‘hpricot’  
require ‘open-uri’  
doc = Hpricot(open(“[https://xxxxxx](https://xxxxxx)”))

will not work because the open will fail because of problems due to  
https.

_Second problem_  
I need to know also how to handle redirection and cookies. But to be  
fair, I still can do some further reading myself on these issues.

Thank you very much.

---

<div class="post-metadata">

**Author:** ![Ramiro\_Diaz\_Trepat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/3ab097/32.png) [@Ramiro\_Diaz\_Trepat](https://www.ruby-forum.com/u/Ramiro_Diaz_Trepat)\
**Post date:** [December 15, 2007, 2:24am UTC](https://www.ruby-forum.com/t/parsing-problems-using-https-and-redirects/123216/2 "2007-12-15T02:24:46Z")

</div>

Quoth Ramiro Diaz Trepat:

> Simple examples from the Hpricot html parsing library like this one:  
> fair, I still can do some further reading myself on these issues.
> 
> Thank you very much.

1. Look at mechanize.

2. Look at http-access2 (or whatever it’s been renamed to).

Regards,

---

<div class="post-metadata">

**Author:** ![Ramiro\_Diaz\_Trepat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/3ab097/32.png) [@Ramiro\_Diaz\_Trepat](https://www.ruby-forum.com/u/Ramiro_Diaz_Trepat)\
**Post date:** [December 15, 2007, 6:49am UTC](https://www.ruby-forum.com/t/parsing-problems-using-https-and-redirects/123216/3 "2007-12-15T06:49:00Z")

</div>

Thank you very much Konrad, it seems that I am on my way now.  
The only weird thing that happened now with Mechanize is that it all  
works  
perfectly on my Linux but it doesn´t on my Mac/Leopard.  
Both have Ruby 1.8.6

On the mac I get the following error while trying to execute the first  
Mechanize example:

./mechanize.rb:4: uninitialized constant WWW (NameError)  
from /Library/Ruby/Site/1.8/rubygems/custom\_require.rb:27:in  
`gem_original_require' from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:27:in `require’  
from goog.rb:2

and the code is the first example of machanize:

require ‘rubygems’  
require ‘mechanize’

agent = WWW::Mechanize.new  
agent.user\_agent\_alias = ‘Mac Safari’  
page = agent.get(“[http://www.google.com/](http://www.google.com/)”)  
search\_form = page.forms.with.name(“f”).first  
search\_form.q = “Hello”  
search\_results = agent.submit(search\_form)  
puts search\_results.body

I really don’t know why this constant is uninitialized and how could I  
initialize it. Besides it worries my that on Linux, after installing  
the  
mechanize gem, everything worked out of the box.

Thanks again