# Extending/Modifying QueryParser

**URL:** <https://www.ruby-forum.com/t/extending-modifying-queryparser/225955>\
**Category:** Ferret\
**Created:** [July 7, 2007, 5:29am UTC](https://www.ruby-forum.com/t/extending-modifying-queryparser/225955 "2007-07-07T05:29:06Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mitchell\_Curtis\_Hat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/e9c0ed/32.png) [@Mitchell\_Curtis\_Hat](https://www.ruby-forum.com/u/Mitchell_Curtis_Hat)\
**Post date:** [July 7, 2007, 5:29am UTC](https://www.ruby-forum.com/t/extending-modifying-queryparser/225955/1 "2007-07-07T05:29:06Z")

</div>

Hi,

I’ve implemented synonym searching in my rails application but have  
an idea I’d like to implement but can’t figure out how to do. The  
idea is that I’d like to give the end user the choice on whether to  
search for the synonym of a word or not. Preferably by extending the  
query language to parse a construct similar to ‘%word1’ and then have  
the word turned into a or list (i.e., word1|word2|word3|…).

Currently, the query parser constantly calls SynonymTokenFilter to  
get synonyms for each token. Is there a way I can go about achieving  
this functionality?

Here’s an overview of what I’ve done so far:

My model classes in my rails app use acts\_as\_ferret with a call that  
looks like:

acts\_as\_ferret(  
:fields =\> [:body],  
:store\_class\_name =\> true,  
:ferret =\> {  
:or\_default =\> false,  
:analyzer =\> SynonymAnalyzer.new(WordnetSynonymEngine.new, [])  
}  
)

I created a SynonymAnalyzer and SynonymTokenFilter:

class SynonymAnalyzer \< Ferret::Analysis::Analyzer  
include Ferret::Analysis

def initialize(synonym\_engine, stop\_words =  
FULL\_ENGLISH\_STOP\_WORDS, lower = true)  
@synonym\_engine = synonym\_engine  
@lower = lower  
@stop\_words = stop\_words  
end

def token\_stream(field, str)  
ts = StandardTokenizer.new(str)  
ts = LowerCaseFilter.new(ts) if @lower  
ts = StopFilter.new(ts, @stop\_words)  
ts = SynonymTokenFilter.new(ts, @synonym\_engine)  
end  
end

class SynonymTokenFilter \< Ferret::Analysis::TokenStream  
include Ferret::Analysis

def initialize(token\_stream, synonym\_engine)  
@token\_stream = token\_stream  
@synonym\_stack = []  
@synonym\_engine = synonym\_engine  
end

def text=(text)  
@token\_stream.text = text  
end

def next  
return @synonym\_stack.pop if @synonym\_stack.size \> 0

```
 if token = @token_stream.next
   add_synonyms_to_stack(token) unless token.nil?
 end

 return token

```

end

private  
def add\_synonyms\_to\_stack(token)  
synonyms = @synonym\_engine.get\_synonyms(token.text)

```
 return if synonyms.nil?

 synonyms.each do |s|
   @synonym_stack.push(
     Token.new(s, token.start, token.end, 0))
 end

```

end  
end

FInally a WordnetSynonymEngine that queries my wordnet index I created:

class WordnetSynonymEngine  
include Ferret::Search

def initialize(index\_name = “wordnet”)  
@searcher = Searcher.new("#{RAILS\_ROOT}/index/#{ENV  
[‘RAILS\_ENV’]}/#{index\_name}")  
end

def get\_synonyms(word)  
@searcher.search\_each(TermQuery.new(:word, word)) do |doc\_id,  
score|  
return @searcher[doc\_id][:syn]  
end

```
 return nil

```

end  
end

It works great except that I’d really like that ability to only run  
tokens through the SynonymTokenFilter when they are prepended by an  
unescaped % sign.

Also, if anyone is interested I can post the code for turning the  
wordnet prolog database into a ferret database (primarily recoding  
the java lucene program that did the same thing to ruby and ferret).

Thanks,  
Curtis

---

<div class="post-metadata">

**Author:** ![Mitchell\_Curtis\_Hat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/e9c0ed/32.png) [@Mitchell\_Curtis\_Hat](https://www.ruby-forum.com/u/Mitchell_Curtis_Hat)\
**Post date:** [July 10, 2007, 10:15am UTC](https://www.ruby-forum.com/t/extending-modifying-queryparser/225955/2 "2007-07-10T10:15:55Z")

</div>

On Fri, Jul 06, 2007 at 11:18:09PM -0400, Mitchell Curtis Hatter wrote:

> get synonyms for each token. Is there a way I can go about achieving  
> this functionality?

You have to extend Ferret’s Query Parser to achieve this. If you don’t  
want to mess around with the grammar stuff the parser code is generated  
from, you could also preprocess user queries to modify them accordingly  
before giving them to the QueryParser. Can get complicated, too 😉

Atm you’re doing the synonym stuff twice, once at indexing time and once  
when Queries are parsed. Because of the insertion of synonyms in the  
index at indexing time, adding synonyms to Queries is not really needed  
any more.

So you don’t really want to specify your SynonymAnalyzer for aaf as the  
analyzer to use for indexing and searching (aaf doesn’t support  
different analyzers for indexing/searching bec. in general it’s a good  
idea to use the same analyzer in both cases).

If you used plain Ferret and wanted Synonyms everywhere or in a specific  
field, but for ALL queries, you could use your Analyzer at indexing  
time,  
but not for Query parsing. In your case, using your WordnetEngine in a  
customized QueryParser or a custom query preprocessor would be the  
better way.

> Here’s an overview of what I’ve done so far:

[…]

That’s really cool stuff, would you mind posting this to Ferret’s Wiki  
so other people can more easily find it? If you included the  
WordnetSynonymEngine that would be even better 🙂

Cheers,  
Jens

–  
Jens Krämer  
webit! Gesellschaft für neue Medien mbH  
Schnorrstraße 76 | 01069 Dresden  
Telefon +49 351 46766-0 | Telefax +49 351 46766-66  
removed\_email\_address@domain.invalid | [www.webit.de](http://www.webit.de)

Amtsgericht Dresden | HRB 15422  
GF Sven Haubold, Hagen Malessa

---

<div class="post-metadata">

**Author:** ![Mitchell\_Curtis\_Hat](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/e9c0ed/32.png) [@Mitchell\_Curtis\_Hat](https://www.ruby-forum.com/u/Mitchell_Curtis_Hat)\
**Post date:** [July 10, 2007, 8:13pm UTC](https://www.ruby-forum.com/t/extending-modifying-queryparser/225955/3 "2007-07-10T20:13:48Z")

</div>

> You have to extend Ferret’s Query Parser to achieve this. If you don’t  
> want to mess around with the grammar stuff the parser code is  
> generated  
> from, you could also preprocess user queries to modify them  
> accordingly  
> before giving them to the QueryParser. Can get complicated, too 😉

I do not enjoy writing parsers, and am not especially good at it. I  
think first I’ll check out the grammar for the parser and see if I  
can modify that. Perhaps creating a SynonymQuery class?

I did consider preprocesing user queries and then just grouping the  
resulting or’d query in parens: ‘rabbit %{ferret}’ would parse to  
‘rabbit (ferret|“black-footed ferret”|etc|etc)’ but I’m sure there  
are situations where that would not be good but it’s an option.

> So you don’t really want to specify your SynonymAnalyzer for aaf as  
> the  
> analyzer to use for indexing and searching (aaf doesn’t support  
> different analyzers for indexing/searching bec. in general it’s a good  
> idea to use the same analyzer in both cases).

Thanks, I was looking at aaf wondering how I could specific a  
different analyzer to use for searches. I didn’t find anything that  
would really let me get a hold of the QueryParser to change the  
analyzer used. Glad I wasn’t just missing it.

> If you used plain Ferret and wanted Synonyms everywhere or in a  
> specific  
> field, but for ALL queries, you could use your Analyzer at indexing  
> time,  
> but not for Query parsing. In your case, using your WordnetEngine in a  
> customized QueryParser or a custom query preprocessor would be the  
> better way.

Since this isn’t for anything but fun right now (at work I’m stuck  
using Oracle’s full text engine which has its own set of problems)  
first I’ll try modifying the QueryParser grammar to account for a new  
query type. My C is not very good so hopefully won’t have to do much,  
but I like that solution better then having to write a pre-processor  
for queries.

> That’s really cool stuff, would you mind posting this to Ferret’s Wiki  
> so other people can more easily find it? If you included the  
> WordnetSynonymEngine that would be even better 🙂
> 
> Cheers,  
> Jens

Thanks, I’ve posted it to the Ferret wiki. It’s quite long but I hope  
that’s not a problem. I included the wordnetSynonymEngine and created  
a YAMLSynonymEngine just to show how it can be pluggable.

Thanks for the tips I’ll see what I can accomplish,  
Curtis