# Character substitution using tr()

**URL:** <https://www.ruby-forum.com/t/character-substitution-using-tr/135457>\
**Category:** Ruby\
**Created:** [April 22, 2008, 11:46am UTC](https://www.ruby-forum.com/t/character-substitution-using-tr/135457 "2008-04-22T11:46:20Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cmaxvv](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/85f322/32.png) [@cmaxvv](https://www.ruby-forum.com/u/cmaxvv)\
**Post date:** [April 22, 2008, 11:46am UTC](https://www.ruby-forum.com/t/character-substitution-using-tr/135457/1 "2008-04-22T11:46:20Z")

</div>

I’m using a method that i found at the acts as ferret site:

[http://projects.jkraemer.net/acts\_as\_ferret/#UTF-8support](http://projects.jkraemer.net/acts_as_ferret/#UTF-8support)

which is intended to strip accents out of strings, turning for example  
“La Bohème” into “La Boheme”. Here’s the method:

def strip\_diacritics(s)  
# latin1 subset only  
s.tr(“ÀÁÂÃÄÅÇÈÉÊËÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝà áâãäåçèéêëìíîïñòóôõöøùúûüýÿ”,  
“AAAAAACEEEEIIIINOOOOOOUUUUYaaaaaaceeeeiiiinoooooouuuuyy”).  
gsub(/Æ/, “AE”).  
gsub(/Ð/, “Eth”).  
gsub(/Þ/, “THORN”).  
gsub(/ß/, “ss”).  
gsub(/æ/, “ae”).  
gsub(/ð/, “eth”).  
gsub(/þ/, “thorn”)  
end

However, it’s breaking for me: è is turned into “yy”. I think this is  
to do with the number of bytes used: the first string passed to tr()  
uses 2 bytes per character while the second uses 1 byte per character:

“ÀÁÂÃÄÅÇÈÉÊËÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝà áâãäåçèéêëìíîïñòóôõöøùúûüýÿ”.size  
=\> 110

“AAAAAACEEEEIIIINOOOOOOUUUUYaaaaaaceeeeiiiinoooooouuuuyy”.size  
=\> 55

Assuming this is the problem, can anyone tell me how to get around it?  
I know next to nothing about character encoding: i tried converting both  
translation strings to utf8 with String#toutf8, but that didn’t make any  
difference.

thanks, max

---

<div class="post-metadata">

**Author:** ![cmaxvv](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/85f322/32.png) [@cmaxvv](https://www.ruby-forum.com/u/cmaxvv)\
**Post date:** [April 22, 2008, 6:45pm UTC](https://www.ruby-forum.com/t/character-substitution-using-tr/135457/2 "2008-04-22T18:45:04Z")

</div>

On Tuesday 22 April 2008 11:46:23 Max W. wrote:

> ```
> "AAAAAACEEEEIIIINOOOOOOUUUUYaaaaaaceeeeiiiinoooooouuuuyy").
> gsub(/Æ/, "AE").
> gsub(/Ð/, "Eth").
> gsub(/Þ/, "THORN").
> gsub(/ß/, "ss").
> gsub(/æ/, "ae").
> gsub(/ð/, "eth").
> gsub(/þ/, "thorn")
> 
> ```
> 
> end

With ruby 1.9 your code works fine without modifications, with ruby 1.8  
and  
it’s support for unicode (or lack of thereof) it might be quite a  
problem to  
get it working.

> Assuming this is the problem, can anyone tell me how to get around it?  
> I know next to nothing about character encoding: i tried converting both  
> translation strings to utf8 with String#toutf8, but that didn’t make any  
> difference.

UTF-8 is variable length encoding, the first half of ascii (which  
includes  
a-zA-Z) is not encoded at all (=1 byte), anything other is encoded as  
2-4  
byte chars. Both of the strings are therefore valid UTF-8, but ruby  
1.8’s tr  
can’t operate on character level, only on byte level.

Jan

---

<div class="post-metadata">

**Author:** ![cmaxvv](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/85f322/32.png) [@cmaxvv](https://www.ruby-forum.com/u/cmaxvv)\
**Post date:** [April 22, 2008, 7:05pm UTC](https://www.ruby-forum.com/t/character-substitution-using-tr/135457/3 "2008-04-22T19:05:46Z")

</div>

Jan D. wrote:

> With ruby 1.9 your code works fine without modifications, with ruby 1.8  
> and  
> it’s support for unicode (or lack of thereof) it might be quite a  
> problem to  
> get it working.

ah…i’m a bit scared to change our project over to ruby 1.9 (i didn’t  
know there was a 1.9) to solve this problem. I ended up just picking  
the most commonly used accents and doing individual gsubs on the strings  
to swap them out. Feels dirty but it works.

Thanks a lot for the info!  
max

---

<div class="post-metadata">

**Author:** ![cmaxvv](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/85f322/32.png) [@cmaxvv](https://www.ruby-forum.com/u/cmaxvv)\
**Post date:** [April 22, 2008, 7:28pm UTC](https://www.ruby-forum.com/t/character-substitution-using-tr/135457/4 "2008-04-22T19:28:47Z")

</div>

Max W. wrote:

> However, it’s breaking for me: è is turned into “yy”.

It works if you require ‘jcode’ first.

HTH,  
Sebastian

---

<div class="post-metadata">

**Author:** ![cmaxvv](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/85f322/32.png) [@cmaxvv](https://www.ruby-forum.com/u/cmaxvv)\
**Post date:** [April 23, 2008, 10:48am UTC](https://www.ruby-forum.com/t/character-substitution-using-tr/135457/5 "2008-04-23T10:48:34Z")

</div>

Sebastian H. wrote:

> Max W. wrote:
> 
> > However, it’s breaking for me: è is turned into “yy”.
> 
> It works if you require ‘jcode’ first.
> 
> HTH,  
> Sebastian

Perfect, thanks! That’s much more palatable than upgrading ruby.

cheers  
max