# A few questions: Tweaking StemFilter, indexes,

**URL:** <https://www.ruby-forum.com/t/a-few-questions-tweaking-stemfilter-indexes/82837>\
**Category:** Ferret\
**Created:** [January 21, 2007, 6:11pm UTC](https://www.ruby-forum.com/t/a-few-questions-tweaking-stemfilter-indexes/82837 "2007-01-21T18:11:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![phillyloco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/258eb7/32.png) [@phillyloco](https://www.ruby-forum.com/u/phillyloco)\
**Post date:** [January 21, 2007, 6:11pm UTC](https://www.ruby-forum.com/t/a-few-questions-tweaking-stemfilter-indexes/82837/1 "2007-01-21T18:11:24Z")

</div>

Hello all,

I am new to the list, but I have been using ferret for a little bit  
already. I would first like to thank Dave for all his work on ferret.

I had a few questions that I haven’t been able to figure out after  
messing around with ferret and going through the documentation.

StemFilter ------

I am trying to improve the quality of my searches in context of the  
content of my application. I have created an analyzer using the  
following:

StemFilter.new StopFilter.new(  
LowerCaseFilter.new(StandardTokenizer.new(text)), @stop\_words )

This has been pretty good so far, however, I really would like to get  
a search for “plumber” match “plumbing” at maybe a lower score than it  
would match “plumbers”. The thing is that plumber(s) is filtered to  
“plumber” and plumbing is filtered to plumb, so it doesn’t match. Is  
there any way to tweak the filter to be able to do these matches? I  
would like to match all noun and verbs together (and ideally with a  
lower score than different verb conjugations would match). Another  
example would be driving and driver.

Worst case scenario, I could probably do some preprocessing to the  
search queries to expand “plumber” or “driving” to a query that  
includes both stems (for example expand the query for plumber to  
“plumber plumb”)

Indexes —

I was wondering how exactly indexes are implemented under the hood and  
if there is a way to give hints to ferret as to how our queries will  
be formed in order to optimize performance. Maybe I’m thinking of  
ferret too much as a database, but I am not too familiar with what’s  
under ferret’s hood.

The reason I ask is that for the project I am working on, I have huge  
amounts of text to search, but each item also has a location  
associated with it (longitude & lattitude) and each query will only  
want to search the text located in a specific area (point and radius).  
I can add ranged parameters to the query and that will work, but is  
that optimal? Hopefully I am making sense.

Donations —

I was wondering if there is a page that lists the total amount of  
donations so far?

Thanks,  
-carl

---

<div class="post-metadata">

**Author:** ![phillyloco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/258eb7/32.png) [@phillyloco](https://www.ruby-forum.com/u/phillyloco)\
**Post date:** [January 22, 2007, 1:38am UTC](https://www.ruby-forum.com/t/a-few-questions-tweaking-stemfilter-indexes/82837/2 "2007-01-22T01:38:32Z")

</div>

Hi,

You could use a FuzzyQuery, that will match words that have some degree  
of resemblance, with lower score.

---

<div class="post-metadata">

**Author:** ![phillyloco](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/258eb7/32.png) [@phillyloco](https://www.ruby-forum.com/u/phillyloco)\
**Post date:** [January 22, 2007, 6:13pm UTC](https://www.ruby-forum.com/t/a-few-questions-tweaking-stemfilter-indexes/82837/3 "2007-01-22T18:13:58Z")

</div>

Excerpts from Carl L.'s message of Sun Jan 21 09:09:59 -0800 2007:

> Worst case scenario, I could probably do some preprocessing to the  
> search queries to expand “plumber” or “driving” to a query that  
> includes both stems (for example expand the query for plumber to  
> “plumber plumb”)

You can either do query expansion or you can modify the stemmer. Query  
expansion is probably a little easier to experiment with because you  
don’t have to worry about reindexing, but it does come with a  
search-time cost which may or may not be negligible. (And it gets a  
little tricky with phrasal queries.)

> I can add ranged parameters to the query and that will work, but is  
> that optimal? Hopefully I am making sense.

I don’t know for sure whether Ferret is sophisticated enough to optimize  
retrieval based on multiple ranges, but it may very well be. In any  
case, I think you’re doing the right thing.