# Creating a canonicalized url **URL:** \ **Category:** Ruby\ **Created:** [January 24, 2008, 1:14pm UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036 "2008-01-24T13:14:23Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![thechak](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/aeb1de/32.png) [@thechak](https://www.ruby-forum.com/u/thechak)\ **Post date:** [January 24, 2008, 1:14pm UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036/1 "2008-01-24T13:14:23Z")
Hello there guys, I’m trying to track down an easy way to canonicalize a URL from with ruby. I’ve been looking around for this but all I can find are some procedure hacks sure as # canonicalize the url if ($url -notmatch “^[a-z]+://”) { $url = “http://$url” } which isn’t going to take into account everything according to RFC 2396 ``` * Remove all leading and trailing dots * Replace consecutive dots with a single dot. * If the hostname can be parsed as an IP address, it should be ``` normalized to 4 dot-separated decimal values. The client should handle any legal IP address encoding, including octal, hex, and fewer than 4 components. \* Lowercase the whole string. # The sequences “/…/” and “/./” in the path should be resolved, by replacing “/./” with “/”, and removing “/…/” along with the preceding path component. # Runs of consecutive slashes should be replaced with a single slash character. So is there a method out there for this? ---
**Author:** ![thechak](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/aeb1de/32.png) [@thechak](https://www.ruby-forum.com/u/thechak)\ **Post date:** [January 24, 2008, 2:24pm UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036/2 "2008-01-24T14:24:14Z")
On Jan 24, 2008, at 7:14 AM, Dan Cuddeford wrote: > - Remove all leading and trailing dots > path component. > > # Runs of consecutive slashes should be replaced with a single slash > > character. > > So is there a method out there for this? I’d start looking at URI, in particular, URI#parse. ## $ fri URI#parse ------------------------------------------------------------- URI::parse URI::parse(uri) ``` Synopsis URI::parse(uri_str) Args +uri_str+: String with URI. Description Creates one of the URI's subclasses instance from the string. Raises URI::InvalidURIError Raised if URI given is not a correct one. Usage require 'uri' uri = URI.parse("http://www.ruby-lang.org/") p uri # => # p uri.scheme # => "http" p uri.host # => "www.ruby-lang.org" ``` As for the “Lowercase the whole string” part, only the domain is required to be case-insensitive. It is possible for the underlying web server to ignore case when finding a path, but the URI is not necessarily a reference to the same resource if the case is altered. -Rob Rob B. [http://agileconsultingllc.com](http://agileconsultingllc.com) removed\_email\_address@domain.invalid ---
**Author:** ![thechak](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/aeb1de/32.png) [@thechak](https://www.ruby-forum.com/u/thechak)\ **Post date:** [January 24, 2008, 3:12pm UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036/3 "2008-01-24T15:12:09Z")
Thanks for your help - I’ll let you know how I get on ---
**Author:** ![thechak](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/aeb1de/32.png) [@thechak](https://www.ruby-forum.com/u/thechak)\ **Post date:** [January 24, 2008, 3:07pm UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036/4 "2008-01-24T15:07:49Z")
2008/1/24, Rob B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > As for the “Lowercase the whole string” part, only the domain is > required to be case-insensitive. It is possible for the underlying > web server to ignore case when finding a path, but the URI is not > necessarily a reference to the same resource if the case is altered. There’s URI#normalize and URI#normalize! to downcase the host part of the url. – Jean-François. ---
**Author:** ![thechak](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/aeb1de/32.png) [@thechak](https://www.ruby-forum.com/u/thechak)\ **Post date:** [January 24, 2008, 3:23pm UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036/5 "2008-01-24T15:23:20Z")
So it seems using the two together require ‘uri’ ``` uri = URI.parse("http://www.ruBy-lang.org/ARSE") ``` can = uri.normalize p can p can.host p can.path means the path keeps it’s case sensitivity but the host is normalized. I think that’s it - however, try it with ruby-lang…org and /usr/lib/ruby/1.8/uri/generic.rb:195:in `initialize': the scheme http does not accept registry part: www.ruBy-lang..org (or bad hostname?) (URI::InvalidURIError) from /usr/lib/ruby/1.8/uri/http.rb:78:in`initialize’ from /usr/lib/ruby/1.8/uri/common.rb:488:in `new' from /usr/lib/ruby/1.8/uri/common.rb:488:in`parse’ from canon.rb:3 So I guess it needs a bit or error checking before hand. ---
**Author:** ![thechak](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/aeb1de/32.png) [@thechak](https://www.ruby-forum.com/u/thechak)\ **Post date:** [January 24, 2008, 5:38pm UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036/6 "2008-01-24T17:38:48Z")
Wow - thanks for the answer mate! ---
**Author:** ![thechak](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/aeb1de/32.png) [@thechak](https://www.ruby-forum.com/u/thechak)\ **Post date:** [January 24, 2008, 5:06pm UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036/7 "2008-01-24T17:06:08Z")
On Jan 24, 2008, at 9:23 AM, Dan Cuddeford wrote: > p can.host > /usr/lib/ruby/1.8/uri/generic.rb:195:in `initialize': the scheme http does not accept registry part: www.ruBy-lang..org (or bad hostname?) (URI::InvalidURIError) from /usr/lib/ruby/1.8/uri/http.rb:78:in `initialize’ > from /usr/lib/ruby/1.8/uri/common.rb:488:in `new' from /usr/lib/ruby/1.8/uri/common.rb:488:in `parse’ > from canon.rb:3 > > So I guess it needs a bit or error checking before hand. require ‘uri’ def canonicalize(uri) u = uri.kind\_of?(URI) ? uri : URI.parse(uri.to\_s) u.normalize! newpath = u.path while newpath.gsub!(%r{([^/]+)/../?}) { |match| $1 == ‘…’ ? match : ‘’ } do end newpath = newpath.gsub(%r{/./}, ‘/’).sub(%r{/.\z}, ‘/’) u.path = newpath u.to\_s end canonicalize(‘[http://www.Ruby-Lang.ORG/ARSE/done/../../rear/./end/.](http://www.Ruby-Lang.ORG/ARSE/done/../../rear/./end/.)’) =\> “[http://www.ruby-lang.org/rear/end/](http://www.ruby-lang.org/rear/end/)” -Rob Rob B. [http://agileconsultingllc.com](http://agileconsultingllc.com) removed\_email\_address@domain.invalid ---
**Author:** ![thechak](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/aeb1de/32.png) [@thechak](https://www.ruby-forum.com/u/thechak)\ **Post date:** [January 24, 2008, 8:30pm UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036/8 "2008-01-24T20:30:47Z")
Dan Cuddeford wrote: > Wow - thanks for the answer mate! There’s also the Addressable Gem: [http://Addressable.RubyForge.Org/](http://Addressable.RubyForge.Org/). It’s intended as a standards compliant replacement for the stdlib’s URI library. Take a look into the test directory of that sucker: over 440 Unit Tests (actually, Object Examples) for a frickin’ URI parser! (See: [http://Addressable.RubyForge.Org/specdoc/](http://Addressable.RubyForge.Org/specdoc/)) That guy is nuts! That code’s gotta be as rock-solid as it gets. Oh, and back to the topic at hand: it has a normalize method built in: begin require ‘rubygems’ gem ‘addressable’ rescue LoadError; end require ‘addressable/uri’ uri = Addressable::URI.heuristic\_parse(‘www.Ruby-Lang…ORG/ARSE/done/…/…/r e a r/./end/.#exit’) uri.normalize! puts uri.display\_uri # =\> [http://www.ruby-lang](http://www.ruby-lang)…org/r%20e%20a%20r/end/#exit jwm ---
**Author:** ![thechak](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/aeb1de/32.png) [@thechak](https://www.ruby-forum.com/u/thechak)\ **Post date:** [January 26, 2008, 1:25am UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036/9 "2008-01-26T01:25:32Z")
Dan Cuddeford wrote: > Jörg W Mittag wrote: > > > puts uri.display\_uri # =\> > > [http://www.ruby-lang](http://www.ruby-lang)…org/r%20e%20a%20r/end/#exit > > Nice but shouldn’t it go to [ruby-lang.org](http://ruby-lang.org)? I’m not sure. I just scanned RfC3986 and RfC1034 and I’m not even sure that’s a valid URI host part to begin with. _If_ it’s invalid, then there’s not much a URI normalizer can do, right? However, I could be wrong. Reading RfCs is not exactly my specialty. jwm ---
**Author:** ![thechak](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/aeb1de/32.png) [@thechak](https://www.ruby-forum.com/u/thechak)\ **Post date:** [January 25, 2008, 4:27pm UTC](https://www.ruby-forum.com/t/creating-a-canonicalized-url/127036/10 "2008-01-25T16:27:20Z")
Jörg W Mittag wrote: > puts uri.display\_uri # =\> > [http://www.ruby-lang](http://www.ruby-lang)…org/r%20e%20a%20r/end/#exit > > jwm Nice but shouldn’t it go to [ruby-lang.org](http://ruby-lang.org)?