# Open-uri and utf8

**URL:** <https://www.ruby-forum.com/t/open-uri-and-utf8/86954>\
**Category:** Ruby\
**Created:** [February 27, 2007, 2:50am UTC](https://www.ruby-forum.com/t/open-uri-and-utf8/86954 "2007-02-27T02:50:43Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kjell\_Olsen](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/c89c15/32.png) [@Kjell\_Olsen](https://www.ruby-forum.com/u/Kjell_Olsen)\
**Post date:** [February 27, 2007, 2:50am UTC](https://www.ruby-forum.com/t/open-uri-and-utf8/86954/1 "2007-02-27T02:50:43Z")

</div>

I’m trying to make requests to google translate [[http://google.com/](http://google.com/)  
translate\_t] to translate words through a little ruby script. I can’t  
get open-uri to work on URL’s with accented characters (åéîòü etc).  
Just calling open() gives:

URI::InvalidURIError: bad URI(is not URI?): [http://google.com/](http://google.com/)  
translate\_t?langpair=en|fr&text=élire  
from /usr/local/lib/ruby/1.8/uri/common.rb:432:in `split' from /usr/local/lib/ruby/1.8/uri/common.rb:481:in `parse’  
from /usr/local/lib/ruby/1.8/open-uri.rb:29:in `open’  
from (irb):2  
from :0

URI.encode()ing the url breaks the characters down into gunk (é =\> %  
C3%A9) which the translator doesn’t understand.

I’ve tried fooling with $KCODE and switching to net/http, neither to  
any avail. Anyone have a hand to give me? I’ll paste the code in full  
below in case anyone wants to see it.

-kjell

— translate.rb ------------

#!/usr/local/bin/ruby  
%w[rubygems open-uri hpricot active\_support readline].each {|lib|  
require lib}  
$KCODE = ‘u’

# can’t handle â/é/utf8 chars because open-uri won’t let us have

special chars in a query, nor will net/http  
class GoogleTranslator  
attr\_reader :doc  
@@langs = ‘fr|en’

def initialize(text, langs=@@langs)  
@text, @langs = text.chomp, langs  
@doc = Hpricot(open(URI.encode(“[http://google.com/translate\_t](http://google.com/translate_t)?  
langpair=#{@langs}&text=#{@text}”)))  
end

def result  
@result ||= @doc.search(‘#result\_box’).inner\_html  
write\_history\_line  
@result  
end

def write\_history\_line # keep a file with all the translations,  
just for kicks.  
`echo "#{"[#{@langs}]\t#{@text}\t\t->\t#{@result}"}" | cat >> '/ Users/kjell/mess/2007/03/translation-history.txt'`  
end

class \<\< self  
include Readline

```
 def prompt; "[#{@@langs}] > "; end
 def interact!
   while text = readline(prompt, true)
     (text =~ /lang: (.*)/) ? @@langs = $1 : puts("#

```

{GoogleTranslator.new(text).result}") # either switch languages or  
spit out a translation  
end  
end  
end  
end

ARGV[0] ? puts(GoogleTranslator.new(\*ARGV[0…1]).result) :  
GoogleTranslator.interact! # if called with an argument, translate  
that argument; else set up for interaction

---

<div class="post-metadata">

**Author:** ![Kjell\_Olsen](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/c89c15/32.png) [@Kjell\_Olsen](https://www.ruby-forum.com/u/Kjell_Olsen)\
**Post date:** [February 27, 2007, 9:41am UTC](https://www.ruby-forum.com/t/open-uri-and-utf8/86954/2 "2007-02-27T09:41:15Z")

</div>

On Tue, Feb 27, 2007 at 10:50:11AM +0900, Kjell Olsen wrote:

> I’m trying to make requests to google translate [[http://google.com/](http://google.com/)  
> translate\_t] to translate words through a little ruby script. I can’t  
> get open-uri to work on URL’s with accented characters (åéîòü etc).

URLs cannot contain accented characters directly - they must be escaped  
into  
%xx hex form. See RFC 2396:

Data must be escaped if it does not have a representation using an  
unreserved character; this includes data that does not correspond to  
a printable character of the US-ASCII coded character set, or that  
corresponds to any US-ASCII character that is disallowed, as  
explained below.

> URI.encode()ing the url breaks the characters down into gunk (é =\> %  
> C3%A9) which the translator doesn’t understand.

Looks like you have a disagreement between yourself and Google about  
what  
character set to use. %C3%A9 looks like a two-byte UTF8 character to me.  
Perhaps Google expects ISO-8859-1; have a look at their API  
documentation if  
they provide any. You should be able to use iconv to convert from one to  
the  
other.

In ISO-8859-1, your accented character will almost certainly appear as a  
single byte, which will URI.encode to a single %xx

HTH,

Brian.