# Translating international characters

**URL:** <https://www.ruby-forum.com/t/translating-international-characters/134679>\
**Category:** Rails\
**Created:** [April 13, 2008, 11:01pm UTC](https://www.ruby-forum.com/t/translating-international-characters/134679 "2008-04-13T23:01:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![makire](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/fbc32d/32.png) [@makire](https://www.ruby-forum.com/u/makire)\
**Post date:** [April 13, 2008, 11:01pm UTC](https://www.ruby-forum.com/t/translating-international-characters/134679/1 "2008-04-13T23:01:58Z")

</div>

Hi

I need to convert strings with international characters to strings  
with corresponding ASCII codes. For example é, è, ë, and ê (and all  
other e-related versions) should convert to e and so on.

Does anyone have a good solution on this?

Kindest regards

Erik

---

<div class="post-metadata">

**Author:** ![makire](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/fbc32d/32.png) [@makire](https://www.ruby-forum.com/u/makire)\
**Post date:** [April 13, 2008, 11:56pm UTC](https://www.ruby-forum.com/t/translating-international-characters/134679/2 "2008-04-13T23:56:56Z")

</div>

On 13 Apr 2008, at 22:01, Erik L. wrote:

> Hi
> 
> I need to convert strings with international characters to strings  
> with corresponding ASCII codes. For example é, è, ë, and ê (and all  
> other e-related versions) should convert to e and so on.

I once did something very crude, which for your purpose would look  
something like this:

```
 def preprocess(query)
   normalized = query.chars.normalize :d
   processed = ""
   normalized.u_unpack.each do |c|
     if c >= 0x300 && c < 0x370 #combining marks
     else
       processed << [c].pack('U*')
     end
   processed
 end

```

Fred

---

<div class="post-metadata">

**Author:** ![makire](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/fbc32d/32.png) [@makire](https://www.ruby-forum.com/u/makire)\
**Post date:** [April 14, 2008, 7:21pm UTC](https://www.ruby-forum.com/t/translating-international-characters/134679/3 "2008-04-14T19:21:48Z")

</div>

On 13 Apr 2008, at 23:01, Erik L. wrote:

> Hi
> 
> I need to convert strings with international characters to strings  
> with corresponding ASCII codes. For example é, è, ë, and ê (and all  
> other e-related versions) should convert to e and so on.
> 
> Does anyone have a good solution on this?

Create a file core\_extensions.rb in /lib/ and stick this in:

require ‘iconv’

class String  
def to\_ascii  
Iconv.iconv(“ASCII//IGNORE//TRANSLIT”, “UTF-8”, self).join.sanitize  
rescue  
self.sanitize  
end

def sanitize  
self.gsub(/[^a-z.\_0-9 -]/i, “”).downcase  
end  
end

Restart your rails server to load the file. Then when you want to  
convert the string, you just do something like “Thïs ïs à  
téststrïng”.to\_ascii and it will convert the characters to their ascii  
equivalent.

Best regards

Peter De Berdt

---

<div class="post-metadata">

**Author:** ![makire](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/fbc32d/32.png) [@makire](https://www.ruby-forum.com/u/makire)\
**Post date:** [April 22, 2008, 4:10pm UTC](https://www.ruby-forum.com/t/translating-international-characters/134679/4 "2008-04-22T16:10:25Z")

</div>

Great advice from everybody. I will try these and see how they work.  
Thanks.

Erik

---

<div class="post-metadata">

**Author:** ![makire](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/fbc32d/32.png) [@makire](https://www.ruby-forum.com/u/makire)\
**Post date:** [April 18, 2008, 11:08pm UTC](https://www.ruby-forum.com/t/translating-international-characters/134679/5 "2008-04-18T23:08:45Z")

</div>

> convert é, è, ë, and ê … to e, etc…

Try  
str = DiacriticsFu::escape(source)  
with

file /lib/diacritic\_fu.rb :  
module DiacriticsFu  
def self.escape(str)  
ActiveSupport::Multibyte::Handlers::UTF8Handler.  
normalize(str,:d).  
split(//u).  
reject { |e| e.length \> 1 }.  
join  
end  
end

, by Thibaut Barrère  
(found here :  
[http://groups.google.ca/group/MephistoBlog/browse\_thread/thread/afe817a4a594ddde](http://groups.google.ca/group/MephistoBlog/browse_thread/thread/afe817a4a594ddde)  
there’s even a test suite)

For example, I extended String with  
class String

# “Un été À la maison”.to\_slug(true) == “un-ete-a-la-maison”

def to\_slug(force\_downcase=false)  
str = DiacriticsFu::escape(self)  
str.gsub!(/[^a-zA-Z0-9]/,“”)  
str.gsub!(/[]+/," “)  
str.gsub!(/ /,”-")  
force\_downcase ? str.downcase : str  
end  
end

Alain