# Problem with case sensitivity

**URL:** <https://www.ruby-forum.com/t/problem-with-case-sensitivity/179504>\
**Category:** Ferret\
**Created:** [November 26, 2009, 12:13pm UTC](https://www.ruby-forum.com/t/problem-with-case-sensitivity/179504 "2009-11-26T12:13:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![cmaxvv](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/85f322/32.png) [@cmaxvv](https://www.ruby-forum.com/u/cmaxvv)\
**Post date:** [November 26, 2009, 12:13pm UTC](https://www.ruby-forum.com/t/problem-with-case-sensitivity/179504/1 "2009-11-26T12:13:32Z")

</div>

I’m using a custom stem analyser in my searches and my indexing. The  
analyser is defined thus:

module Ferret::Analysis  
class StemmingAnalyzer  
def token\_stream(field, text)  
text.downcase!  
RAILS\_DEFAULT\_LOGGER.debug “SEARCHING, field = #{field.inspect},  
text  
= #{text.inspect}”  
tokenizer = StandardTokenizer.new(text)  
filter = StemFilter.new(tokenizer)  
filter  
end  
end  
end

I use it in my indexing like this:

acts\_as\_ferret({ :store\_class\_name =\> true,  
:ferret =\> { :analyzer =\>  
Ferret::Analysis::StemmingAnalyzer.new },  
:fields =\> {:property\_names =\> { :boost =\> 3.0 },  
…etc  
}})

And in a search like this:

search\_class.find\_ids\_with\_ferret(search\_term, {:limit =\> 10000,  
:analyzer  
=\> Ferret::Analysis::StemmingAnalyzer.new}) do |model, r\_id, score|  
r\_id = r\_id.to\_i  
ferret\_ids \<\< r\_id  
self.scores\_hash[r\_id] = score  
end

I have a problem with case sensitivity - basically, searches only work  
when  
they are lowercase: even when it looks like the text stored in the index  
is  
uppercase. From the console -

> > resource.to\_doc  
> > =\> {:resource\_id=\>“59”, :property\_names=\>“Bb Clarinet Clarinet Family  
> > Woodwind Instrumental and Vocal Image Resources Types” }
> 
> > TeachingObject.find\_with\_ferret(“Vocal”, :page =\> 1, :per\_page =\>  
> > 1000).include?(resource)  
> > =\> false
> 
> > TeachingObject.find\_with\_ferret(“vocal”, :page =\> 1, :per\_page =\>  
> > 1000).include?(resource)  
> > =\> true

I think i have my stemming set up wrong, i’m not sure if it is even  
being  
used. I implemented it so that searches allowed pluralised and singular  
terms, and that seems to work, eg

> > TeachingObject.find\_with\_ferret(“vocals”, :page =\> 1, :per\_page =\>  
> > 1000).include?(resource)  
> > =\> true

But the case sensitivity thing has me stumped. I thought that the  
downcase!  
call on the search term would make case irrelevant for searching but  
that  
seems not to be the case. Can anyone set me straight?

---

<div class="post-metadata">

**Author:** ![cmaxvv](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/85f322/32.png) [@cmaxvv](https://www.ruby-forum.com/u/cmaxvv)\
**Post date:** [November 26, 2009, 12:58pm UTC](https://www.ruby-forum.com/t/problem-with-case-sensitivity/179504/2 "2009-11-26T12:58:33Z")

</div>

I think i fixed this. I did three things

- changed my custom analyser to inherit from Ferret::Analysis::Analyzer
- ditched the downcase! line
- instead of doing downcase!, I added LowerCaseFilter.new(filter) to my  
chain

module Ferret::Analysis  
class StemmingAnalyzer \< Ferret::Analysis::Analyzer  
def token\_stream(field, text)  
RAILS\_DEFAULT\_LOGGER.debug “SEARCHING, field = #{field.inspect},  
text = #{text.inspect}”  
tokenizer = StandardTokenizer.new(text)  
filter = StemFilter.new(tokenizer)  
low\_filter = LowerCaseFilter.new(filter)  
low\_filter  
end  
end  
end

After calling ferret\_update on the resource, i can now get it with  
‘vocal’ or ‘Vocal’.

I’d still welcome any further advice on this, in case i’m not doing  
something right.

thanks, max