# Open-uri bug **URL:** \ **Category:** Ruby\ **Created:** [February 26, 2008, 8:51pm UTC](https://www.ruby-forum.com/t/open-uri-bug/130256 "2008-02-26T20:51:14Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![steveybaby](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/c57346/32.png) [@steveybaby](https://www.ruby-forum.com/u/steveybaby)\ **Post date:** [February 26, 2008, 8:51pm UTC](https://www.ruby-forum.com/t/open-uri-bug/130256/1 "2008-02-26T20:51:14Z")
Hello all, I’m using open-uri combined with hpricot to make a basic web crawler that scrapes for different links that I need. It seems to be working perfectly, but I have encountered the following bug when this type of link is encountered: irb(main):015:0\> URI.parse(‘[http://hello.com/a.php?%1](http://hello.com/a.php?%251)’) URI::InvalidURIError: bad URI(is not URI?): [http://hello.com/a.php?%1](http://hello.com/a.php?%251) from c:/ruby/lib/ruby/1.8/uri/common.rb:436:in `split' from c:/ruby/lib/ruby/1.8/uri/common.rb:485:in `parse’ from (irb):15 Can anyone illuminate why this is a problem? Thanks! ---
**Author:** ![steveybaby](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/c57346/32.png) [@steveybaby](https://www.ruby-forum.com/u/steveybaby)\ **Post date:** [February 26, 2008, 9:21pm UTC](https://www.ruby-forum.com/t/open-uri-bug/130256/2 "2008-02-26T21:21:20Z")
On Feb 26, 2008, at 2:50 PM, Steve H. wrote: > Can anyone illuminate why this is a problem? Thanks! Probably because %1 looks like a partially escaped character. Try: ?%251 Where %25 is an escaped % -Rob Rob B. [http://agileconsultingllc.com](http://agileconsultingllc.com) removed\_email\_address@domain.invalid ---
**Author:** ![steveybaby](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/c57346/32.png) [@steveybaby](https://www.ruby-forum.com/u/steveybaby)\ **Post date:** [February 26, 2008, 10:22pm UTC](https://www.ruby-forum.com/t/open-uri-bug/130256/3 "2008-02-26T22:22:03Z")
On Feb 26, 12:20 pm, Rob B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Probably because %1 looks like a partially escaped character. Try: > > ``` > ?%251 > > ``` > > Where %25 is an escaped % > > -Rob I appreciate the reply. This is a bit unfortunate, I am developing a tool which has to handle URIs the same way the browser does. While I realize that is not a “correct” URI, the browser still fetches the pages without a problem. In some sense, I wish I could mirror the functionality of the browser fetch using the URI module. Anyhow, thank you for your help! ---
**Author:** ![steveybaby](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/c57346/32.png) [@steveybaby](https://www.ruby-forum.com/u/steveybaby)\ **Post date:** [February 26, 2008, 10:43pm UTC](https://www.ruby-forum.com/t/open-uri-bug/130256/4 "2008-02-26T22:43:51Z")
Steve H. wrote: > On Feb 26, 12:20 pm, Rob B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > wrote: > > > Probably because %1 looks like a partially escaped character. Try: > > > > ``` > > ?%251 > > > > ``` > > > > Where %25 is an escaped % > > > > -Rob > > I appreciate the reply. This is a bit unfortunate, I am developing a > tool which has to handle URIs the same way the browser does. While I > realize that is not a “correct” URI, the browser still fetches the > pages without a problem. In some sense, I wish I could mirror the > functionality of the browser fetch using the URI module. Anyhow, thank > you for your help! Maybe this helps: URI.escape(‘[http://hello.com/a.php?%1](http://hello.com/a.php?%251)’) =\> “[http://hello.com/a.php?%251](http://hello.com/a.php?%251)” Regards, Siep ---
**Author:** ![steveybaby](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/c57346/32.png) [@steveybaby](https://www.ruby-forum.com/u/steveybaby)\ **Post date:** [March 1, 2008, 9:46am UTC](https://www.ruby-forum.com/t/open-uri-bug/130256/5 "2008-03-01T09:46:31Z")
I too want to know how to handle invalid URIs in mechanize. Is there any way to override url checking ? ---
**Author:** ![steveybaby](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/c57346/32.png) [@steveybaby](https://www.ruby-forum.com/u/steveybaby)\ **Post date:** [February 27, 2008, 5:52am UTC](https://www.ruby-forum.com/t/open-uri-bug/130256/6 "2008-02-27T05:52:44Z")
On Feb 26, 2008, at 13:20 PM, Steve H. wrote: > I appreciate the reply. This is a bit unfortunate, I am developing a > tool which has to handle URIs the same way the browser does. While I > realize that is not a “correct” URI, the browser still fetches the > pages without a problem. In some sense, I wish I could mirror the > functionality of the browser fetch using the URI module. Anyhow, thank > you for your help! What about Mechanize?