# International character search

**URL:** <https://www.ruby-forum.com/t/international-character-search/71877>\
**Category:** Rails\
**Created:** [October 12, 2006, 4:24am UTC](https://www.ruby-forum.com/t/international-character-search/71877 "2006-10-12T04:24:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![soulhunter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/7feea3/32.png) [@soulhunter](https://www.ruby-forum.com/u/soulhunter)\
**Post date:** [October 12, 2006, 4:24am UTC](https://www.ruby-forum.com/t/international-character-search/71877/1 "2006-10-12T04:24:02Z")

</div>

Hi!

I’m working on a program, and I need to do case-insensitive search with  
international characters on it, like:

ñáéíóúàèìòùäëïöü and so on.

Anyway, I found a way of implementing it, but I don’t quite like it  
because it would implies create the autocomplete function for _each_  
autocomplete I have in my project.

The way of doing so I found is to change the condition from:

LOWER(column) like ‘%thing\_downcased%’

to

column ~\* ‘thing\_downcased’

and replacing the international characters for the [ñÑ] kind of  
expression, like this:

name ~\* ‘la [ñÑ]apa’

and it actually works (at least with postgresql), but then, I would  
need to do the substitution everytime I do a search, and I would need  
to reimplement the autocomplete function for each autocompletion with  
the new schema.

Any better idea?,

Sincerely,

Ildefonso Camargo

---

<div class="post-metadata">

**Author:** ![soulhunter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/7feea3/32.png) [@soulhunter](https://www.ruby-forum.com/u/soulhunter)\
**Post date:** [October 18, 2006, 10:39am UTC](https://www.ruby-forum.com/t/international-character-search/71877/2 "2006-10-18T10:39:28Z")

</div>

On Oct 12, 2006, at 4:19 AM, soulhunter wrote:

> Anyway, I found a way of implementing it, but I don’t quite like it  
> because it would implies create the autocomplete function for _each_  
> autocomplete I have in my project.
> 
> The way of doing so I found is to change the condition from:
> 
> LOWER(column) like ‘%thing\_downcased%’

Just to share a different approach, since you can’t expect users to  
type accented words correctly, I usually store a normalized extra  
column (say name\_normalized) for searches maintained in some Rails-  
way like filters, or store just the normalization of them in ferret.  
Then any query has to be normalized.

– fxn

# Utility method that retursn an ASCIIfied, downcased, and

sanitized string.

# It relies on the Unicode Hacks plugin by means of String#chars.

We assume

# $KCODE is ‘u’ in environment.rb. By now we support a wide range

of latin

# accented letters, based on the Unicode Character Palette bundled

in Macs.  
def self.normalize(str)  
n = str.chars.downcase.strip.to\_s  
n.gsub!(/[à áâãäåāÄ?Ä?]/u, ‘a’)  
n.gsub!(/\s+/, ’ ')  
n.gsub!(/æ/u, ‘ae’)  
n.gsub!(/[ďÄ?]/u, ‘d’)  
n.gsub!(/[çÄ?čÄ?Ä?]/u, ‘c’)  
n.gsub!(/[èéêëÄ?Ä?Ä?Ä?Ä?]/u, ‘e’)  
n.gsub!(/Æ?/u, ‘f’)  
n.gsub!(/[ĝÄ?Ä¡Ä£]/u, ‘g’)  
n.gsub!(/[ĥħ]/, ‘h’)  
n.gsub!(/[ììíîïīĩĭ]/u, ‘i’)  
n.gsub!(/[įıijĵ]/u, ‘j’)  
n.gsub!(/[ķĸ]/u, ‘k’)  
n.gsub!(/[Å?ľĺļÅ?]/u, ‘l’)  
n.gsub!(/[ñÅ?Å?Å?Å?Å?]/u, ‘n’)  
n.gsub!(/[òóôõöøōÅ?ŏŏ]/u, ‘o’)  
n.gsub!(/Å?/u, ‘oe’)  
n.gsub!(/[Å?Å?Å?]/u, ‘r’)  
n.gsub!(/[Å?Å¡Å?ŝÈ?]/u, ‘s’)  
n.gsub!(/[ťţŧÈ?]/u, ‘t’)  
n.gsub!(/[ùúûüūůűŭũų]/u, ‘u’)  
n.gsub!(/ŵ/u, ‘w’)  
n.gsub!(/[ýÿŷ]/u, ‘y’)  
n.gsub!(/[žżź]/u, ‘z’)  
n.gsub!(/[^\sa-z0-9\_-]/, ‘’)  
n  
end

---

<div class="post-metadata">

**Author:** ![soulhunter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/7feea3/32.png) [@soulhunter](https://www.ruby-forum.com/u/soulhunter)\
**Post date:** [October 18, 2006, 10:39am UTC](https://www.ruby-forum.com/t/international-character-search/71877/3 "2006-10-18T10:39:38Z")

</div>

On 10/13/06, Xavier N. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> # It relies on the Unicode Hacks plugin by means of String#chars.
> 
> ```
> n.gsub!(/[ďÄ?]/u, 'd')
> n.gsub!(/[òóôõöøōÅ?ŏŏ]/u, 'o')
> 
> ```
> 
> end

Sweet! I’ve just been looking for a character conversion chart like  
this to add a filter to Ferret. In a future version of Ferret (coming  
very soon) this will be a lot easier and faster. I’ll probably put an  
option on the StandardAnalyzer called :normalize\_unicode or something.

Thanks Xavier,  
Dave

---

<div class="post-metadata">

**Author:** ![soulhunter](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/7feea3/32.png) [@soulhunter](https://www.ruby-forum.com/u/soulhunter)\
**Post date:** [October 18, 2006, 10:39am UTC](https://www.ruby-forum.com/t/international-character-search/71877/4 "2006-10-18T10:39:42Z")

</div>

On Oct 13, 2006, at 2:47 AM, David B. wrote:

> Sweet! I’ve just been looking for a character conversion chart like  
> this to add a filter to Ferret. In a future version of Ferret (coming  
> very soon) this will be a lot easier and faster. I’ll probably put an  
> option on the StandardAnalyzer called :normalize\_unicode or something.

Excelent!

I noticed in the mail that a q-like character was among the a-like  
character class, I moved that out and send the normalizer again for  
the archives:

# Utility method that retursn an ASCIIfied, downcased, and sanitized

string.

# It relies on the Unicode Hacks plugin by means of String#chars. We

assume

# $KCODE is ‘u’ in environment.rb. By now we support a wide range of

latin

# accented letters, based on the Unicode Character Palette bundled in

Macs.  
def self.normalize(str)  
n = str.chars.downcase.strip.to\_s  
n.gsub!(/[à áâãäåāÄ?]/u, ‘a’)  
n.gsub!(/æ/u, ‘ae’)  
n.gsub!(/[ďÄ?]/u, ‘d’)  
n.gsub!(/[çÄ?čÄ?Ä?]/u, ‘c’)  
n.gsub!(/[èéêëÄ?Ä?Ä?Ä?Ä?]/u, ‘e’)  
n.gsub!(/Æ?/u, ‘f’)  
n.gsub!(/[ĝÄ?Ä¡Ä£]/u, ‘g’)  
n.gsub!(/[ĥħ]/, ‘h’)  
n.gsub!(/[ììíîïīĩĭ]/u, ‘i’)  
n.gsub!(/[įıijĵ]/u, ‘j’)  
n.gsub!(/[ķĸ]/u, ‘k’)  
n.gsub!(/[Å?ľĺļÅ?]/u, ‘l’)  
n.gsub!(/[ñÅ?Å?Å?Å?Å?]/u, ‘n’)  
n.gsub!(/[òóôõöøōÅ?ŏŏ]/u, ‘o’)  
n.gsub!(/Å?/u, ‘oe’)  
n.gsub!(/Ä?/u, ‘q’)  
n.gsub!(/[Å?Å?Å?]/u, ‘r’)  
n.gsub!(/[Å?Å¡Å?ŝÈ?]/u, ‘s’)  
n.gsub!(/[ťţŧÈ?]/u, ‘t’)  
n.gsub!(/[ùúûüūůűŭũų]/u, ‘u’)  
n.gsub!(/ŵ/u, ‘w’)  
n.gsub!(/[ýÿŷ]/u, ‘y’)  
n.gsub!(/[žżź]/u, ‘z’)  
n.gsub!(/\s+/, ’ ')  
n.gsub!(/[^\sa-z0-9\_-]/, ‘’)  
n  
end

– fxn