# Ruby Mechanize issues

**URL:** <https://www.ruby-forum.com/t/ruby-mechanize-issues/110048>\
**Category:** Ruby\
**Created:** [July 22, 2007, 2:43am UTC](https://www.ruby-forum.com/t/ruby-mechanize-issues/110048 "2007-07-22T02:43:39Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Timothy\_Brown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@Timothy\_Brown](https://www.ruby-forum.com/u/Timothy_Brown)\
**Post date:** [July 22, 2007, 2:43am UTC](https://www.ruby-forum.com/t/ruby-mechanize-issues/110048/1 "2007-07-22T02:43:39Z")

</div>

I’m having a devil of a time trying to scrape an onerous website with  
Scrubyt/Mechanize.

Changing the site’s policies is out of scope, so I need to try to find  
some way to work around them.

I think the problem is centered around the site’s use of schemes.  
HTTPS is often used in place of https, for instance. I’m not a Ruby  
wizard, but I’ve patched Mechanize at line 429, line 439, and line  
466. On 466 i’ve put

```
if uri.scheme == 'https|HTTPS' && ! http_obj.started?

```

I assume this is right but like I said, i’m a Nuby. 🙂

Anyway, the site does the following. After the below output, Ruby  
times out in rbuf\_fill. I assume it’s not really following the  
redirect, and my patching is wrong, but I can’t be certain.  
Regardless of whether I use scrubyt or mechanize natively, the  
symptoms are the same. My script looks like:

page = agent.get(‘[http://accounts.xxxx.org](http://accounts.xxxx.org)’)  
form = page.forms.find {|f| f.name == “frmInput”}  
form.fields.find {|f| f.name == ‘Name’}.value = ‘xxxx’  
form.fields.find {|f| f.name == ‘Password’}.value = ‘xxxx’  
puts page.forms  
page = agent.submit(form, form.buttons.first)

after the obvious initialization.

Does anyone have any clues? I’m at wits end here.

[Parked at Loopia](http://pastie.caboo.se/81051) is the output.