How to parse a unicode url?

Ruby
1

I would really like to be able to do the following. Is this even
possible?

Thanks,
nerdytenor

uri = URI.parse(‘http://www.hÃren.com’) # not a real url (that I know
of)
URI::InvalidURIError: bad URI(is not URI?): http://www.hÃren.com
from /usr/lib/ruby/1.8/uri/common.rb:432:in split' from /usr/lib/ruby/1.8/uri/common.rb:481:in parse’
from (irb):26

2

Dan The man wrote:

I would really like to be able to do the following. Is this even
possible?

Thanks,
nerdytenor

uri = URI.parse(‘http://www.hÃren.com’) # not a real url (that I know
of)
URI::InvalidURIError: bad URI(is not URI?): http://www.hÃren.com
from /usr/lib/ruby/1.8/uri/common.rb:432:in split' from /usr/lib/ruby/1.8/uri/common.rb:481:in parse’
from (irb):26

You can do this:

require “uri”

url = "http://www.hÃren.co"ö
enc_url = URI.encode(url)
puts enc_url

to get this:

http://www.h%C3%B6ren.co

which according to wikipedia here:

is a legal uri. But when I do this:

require “uri”

url = “http://www.hÃren.co”
enc_url = URI.encode(url)
puts enc_url

uri = URI.parse(enc_url)

I get this:

http://www.h%C3%B6ren.co
/usr/lib/ruby/1.8/uri/generic.rb:194:in initialize': the scheme http does not accept registry part: www.h%C3%B6ren.co (or bad hostname?) (URI::InvalidURIError) from /usr/lib/ruby/1.8/uri/http.rb:46:in initialize’
from /usr/lib/ruby/1.8/uri/common.rb:484:in new' from /usr/lib/ruby/1.8/uri/common.rb:484:in parse’
from r3test.rb:7

which as far as I can tell means that URI.parse() is broken.

3

Robert K. wrote:

which as far as I can tell means that URI.parse() is broken.

I don’t think so. There are invalid characters in the domain name (as
the exception indicates).

Can you identify which character is invalid in:

http://www.h%C3%B6ren.co

According to wikipedia, all those characters are valid for a uri.

4

On 20.09.2007 07:19, 7stud – wrote:

    from /usr/lib/ruby/1.8/uri/common.rb:432:in `split'
    from /usr/lib/ruby/1.8/uri/common.rb:481:in `parse'
    from (irb):26

There is no such thing as a Unicode URL. The RFC for URI and URL
specify the charset as 7Bit ASCII AFAIK.

The legal form of that URL is this: http://www.hören.com/

See IDNA for details, for example:

Quick searching revealed this - maybe it can help:
http://rubyforge.org/pipermail/idn-discuss/2005-September/000000.html

http://www.h%C3%B6ren.co
url = “http://www.hören.co”
(URI::InvalidURIError)
from /usr/lib/ruby/1.8/uri/http.rb:46:in initialize' from /usr/lib/ruby/1.8/uri/common.rb:484:in new’
from /usr/lib/ruby/1.8/uri/common.rb:484:in `parse’
from r3test.rb:7

which as far as I can tell means that URI.parse() is broken.

I don’t think so. There are invalid characters in the domain name (as
the exception indicates).

Kind regards

robert

5

Ollivier R. wrote:

In article [email protected],
7stud – [email protected] wrote:

Can you identify which character is invalid in:

http://www.h%C3%B6ren.co

According to wikipedia, all those characters are valid for a uri.

They are but host & domain names do not accept Unicode characters at all
and are limited to 7 bits ASCII. Search for IDN for more information.

I thought this might be the case. However, typing the following into
firefox gets me a real live page (after trying a few random domains)

http://www.hören.at/

Hmmm…

6

In article [email protected],
7stud – [email protected] wrote:

Can you identify which character is invalid in:

http://www.h%C3%B6ren.co

According to wikipedia, all those characters are valid for a uri.

They are but host & domain names do not accept Unicode characters at all
and are limited to 7 bits ASCII. Search for IDN for more information.

7

On Sep 20, 2007, at 24:31 , 7stud – wrote:

http://www.h%C3%B6ren.co

According to wikipedia, all those characters are valid for a uri.

It says that what characters are valid for each piece of a URI is
dependent on the URI scheme. The characters valid for the hostname
part of the http URI scheme is goverened by the DNS system, so you
need to use an IDN.

I believe there is a ruby wrapper for libidn.

8

I thought this might be the case. However, typing the following into
firefox gets me a real live page (after trying a few random domains)

http://www.hören.at/

Hmmm…

That’s because Firefox automatically translates into the equivalent IDN;
attempting to lookup the domain www.hören.at (or hören.at) gives
NXDOMAIN - it will never work. www.hören.at is translated into the
punycode name www.xn–hren-5qa.at. Try visiting that in Firefox - it
actually rewrote the URL as www.hören.at for me, but rest assured the
domain is one and the same.

7 bit ASCII is to be used in domain names only; please use IDN to
represent international characters, as it’s the accepted way to do it,
and implemented as you see in Firefox.

Cheers.

Arlen