# How to have 'o' == 'ö'

**URL:** <https://www.ruby-forum.com/t/how-to-have-o-a/82684>\
**Category:** Ferret\
**Created:** [January 19, 2007, 6:12pm UTC](https://www.ruby-forum.com/t/how-to-have-o-a/82684 "2007-01-19T18:12:12Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnpetterson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/41988e/32.png) [@johnpetterson](https://www.ruby-forum.com/u/johnpetterson)\
**Post date:** [January 19, 2007, 6:12pm UTC](https://www.ruby-forum.com/t/how-to-have-o-a/82684/1 "2007-01-19T18:12:12Z")

</div>

Greetings,

(using acts\_as\_ferret)

So I have a book title “Möngrel „Horsemen“” in my index.

Searching for “Möngrel” retrieves the document.

But I would like searching for “Mongrel” to also retrieve the document.  
Which it does not currently.

Anyone have any good solutions to this problem?

I suppose I could filter the documents and queries first which something  
like:

(Iconv.new(‘US-ASCII//TRANSLIT’, ‘utf-8’).iconv “Möngrel  
„Horsemen“”).gsub(/[^a-zA-Z0-9/im,"")

But perhaps there is a better, or built in solution.

Thanks

---

<div class="post-metadata">

**Author:** ![johnpetterson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/41988e/32.png) [@johnpetterson](https://www.ruby-forum.com/u/johnpetterson)\
**Post date:** [January 22, 2007, 2:55pm UTC](https://www.ruby-forum.com/t/how-to-have-o-a/82684/2 "2007-01-22T14:55:58Z")

</div>

On Fri, Jan 19, 2007 at 06:12:12PM +0100, John Private wrote:

> Anyone have any good solutions to this problem?
> 
> I suppose I could filter the documents and queries first which something  
> like:
> 
> (Iconv.new(‘US-ASCII//TRANSLIT’, ‘utf-8’).iconv “Möngrel  
> „Horsemen“”).gsub(/[^a-zA-Z0-9/im,“”)
> 
> But perhaps there is a better, or built in solution.

I don’t think so - a custom Analyzer would be the right place for  
this.

Jens

–  
webit! Gesellschaft für neue Medien mbH [www.webit.de](http://www.webit.de)  
Dipl.-Wirtschaftsingenieur Jens Krämer removed\_email\_address@domain.invalid  
Schnorrstraße 76 Tel +49 351 46766 0  
D-01069 Dresden Fax +49 351 46766 66

---

<div class="post-metadata">

**Author:** ![johnpetterson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/41988e/32.png) [@johnpetterson](https://www.ruby-forum.com/u/johnpetterson)\
**Post date:** [January 22, 2007, 4:27pm UTC](https://www.ruby-forum.com/t/how-to-have-o-a/82684/3 "2007-01-22T16:27:31Z")

</div>

On Jan 22, 2007, at 2:49 PM, Jens K. wrote:

> > document.  
> > „Horsemen“").gsub(/[^a-zA-Z0-9/im,"")
> > 
> > But perhaps there is a better, or built in solution.
> 
> I don’t think so - a custom Analyzer would be the right place for  
> this.

We use a normalizer to store/query (to be revised for Rails 1.2):

# Utility method that retursn an ASCIIfied, downcased, and

sanitized string.

# It relies on the Unicode Hacks plugin by means of String#chars.

We assume

# $KCODE is ‘u’ in environment.rb. By now we support a wide range

of latin

# accented letters, based on the Unicode Character Palette bundled

in Macs.  
def self.normalize(str)  
n = str.chars.downcase.strip.to\_s  
n.gsub!(/[à áâãäåāă]/, ‘a’)  
n.gsub!(/æ/, ‘ae’)  
n.gsub!(/[ďđ]/, ‘d’)  
n.gsub!(/[çćčĉċ]/, ‘c’)  
n.gsub!(/[èéêëēęěĕė]/, ‘e’)  
n.gsub!(/Æ’/, ‘f’)  
n.gsub!(/[ĝğġģ]/, ‘g’)  
n.gsub!(/[ĥħ]/, ‘h’)  
n.gsub!(/[ììíîïīĩĭ]/, ‘i’)  
n.gsub!(/[įıijĵ]/, ‘j’)  
n.gsub!(/[ķĸ]/, ‘k’)  
n.gsub!(/[łľĺļŀ]/, ‘l’)  
n.gsub!(/[ñńňņʼnŋ]/, ‘n’)  
n.gsub!(/[òóôõöøōőŏŏ]/, ‘o’)  
n.gsub!(/Å“/, ‘oe’)  
n.gsub!(/Ä…/, ‘q’)  
n.gsub!(/[ŕřŗ]/, ‘r’)  
n.gsub!(/[śšşŝș]/, ‘s’)  
n.gsub!(/[ťţŧț]/, ‘t’)  
n.gsub!(/[ùúûüūůűŭũų]/, ‘u’)  
n.gsub!(/ŵ/, ‘w’)  
n.gsub!(/[ýÿŷ]/, ‘y’)  
n.gsub!(/[žżź]/, ‘z’)  
n.gsub!(/\s+/, ’ ')  
n.gsub!(/[^\sa-z0-9\_-]/, ‘’)  
n  
end

And this convenience class method to use in Rails models with  
acts\_as\_ferret (slightly edited):

# Wrapper function to normalize fields before calling acts\_as\_ferret

# 

# Usage: index\_fields [:field1, :field2], :option1

=\> …, :option2 =\> …

# 

# Please note that your queries should use a “\_normalized” suffix on

# each field, i.e: +field1\_normalized:foo

class ActiveRecord::Base  
def self.index\_fields(fields, \*options)  
aaf\_fields = []  
fields.each do |f|  
class\_eval \<\<-EOS  
def #{f}\_normalized  
MyAppUtils.normalize(#{f})  
end  
EOS  
aaf\_fields.push “:#{f}\_normalized”  
end  
aaf\_call = ‘acts\_as\_ferret :fields =\> [’ + aaf\_fields.join  
(’,’) + ‘]’  
options.each do |option\_pair|  
option\_pair.each do |key, value|  
aaf\_call \<\< “, :#{key} =\> #{value}”  
end  
end  
logger.info aaf\_call  
class\_eval(aaf\_call)  
end  
end

– fxn

---

<div class="post-metadata">

**Author:** ![johnpetterson](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/41988e/32.png) [@johnpetterson](https://www.ruby-forum.com/u/johnpetterson)\
**Post date:** [February 24, 2007, 1:58pm UTC](https://www.ruby-forum.com/t/how-to-have-o-a/82684/4 "2007-02-24T13:58:25Z")

</div>

On 1/23/07, Xavier N. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > >
> 
> # Utility method that retursn an ASCIIfied, downcased, and
> 
> ```
> n.gsub!(/æ/, 'ae')
> n.gsub!(/[ñńňņʼnŋ]/, 'n')
> n.gsub!(/\s+/, ' ')
> 
> ```
> 
> =\> …, :option2 =\> …  
> end  
> logger.info aaf\_call  
> class\_eval(aaf\_call)  
> end  
> end
> 
> – fxn

Sorry to bring this one back from the archives (I’m going through all  
the email I’ve missed in my long absence). Anyway, I thought that  
since not even Jens knew about this I should point out the existence  
of MappingFilter:

```
http://ferret.davebalmain.com/api/classes/Ferret/Analysis/MappingFilter.html

```

It essentially does the same thing as Xavier’s code above but it is  
much faster. It compiles the mappings to a single deterministic finite  
automaton (DFA):

```
http://en.wikipedia.org/wiki/Deterministic_finite_state_machine

```

Basically, this means the filter does a single pass through the string  
to do all the mappings rather than a pass for each mapping.

Hope that helps somebody,  
Dave