Forum: Ferret Using custom stem analyzer giving mongrel errors

I'm using the custom stem analyzer:

require 'rubygems'
require 'ferret'
include Ferret
module Ferret::Analysis
 class FerretAnalyzer
  def initialize(stop_words = FULL_ENGLISH_STOP_WORDS)
      @stop_words = stop_words
  end
   def token_stream(field, text)
     StemFilter.new(StopFilter.new(LowerCaseFilter.new(StandardTokenizer.new(text)),
@stop_words))
   end
 end
end

and I'm simply setting the :analyzer option in AAF.

However, I get odd behavior.  The first search that I do will go through
and display the proper results, but any subsequent request starts to
produce odd behavior.  For example when you are redirected, mongrel does
not redirect the browser but instead displays the message 'You are being
redirected'.  Here is the error produced in the Mongrel logs:

Thu Dec 07 17:49:06 PST 2006: Error calling Dispatcher.dispatch
#<NameError: cannot remove Object::QueryParser>
/Users/raymond/Rails/cart/config/../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:416:in
`remove_const'
/Users/raymond/Rails/cart/config/../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:416:in
`remove_constant'
/Users/raymond/Rails/cart/config/../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:270:in
`remove_unloadable_constants!'
/Users/raymond/Rails/cart/config/../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:270:in
`remove_unloadable_constants!'
/Users/raymond/Rails/cart/config/../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:73:in
`clear'
/Users/raymond/Rails/cart/config/../vendor/rails/railties/lib/dispatcher.rb:60:in
`reset_application!'
/Users/raymond/Rails/cart/config/../vendor/rails/railties/lib/dispatcher.rb:116:in
`reset_after_dispatch'
/Users/raymond/Rails/cart/config/../vendor/rails/railties/lib/dispatcher.rb:51:in
`dispatch'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel/rails.rb:84:in
`process'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/1.8/sync.rb:229:in
`synchronize'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel/rails.rb:83:in
`process'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel.rb:580:in
`process_client'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel.rb:579:in
`process_client'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel.rb:686:in
`run'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel.rb:686:in
`run'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel.rb:673:in
`run'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel/configurator.rb:267:in
`run'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel/configurator.rb:266:in
`run'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/bin/mongrel_rails:127:in
`run'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/lib/mongrel/command.rb:211:in
`run'
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/lib/ruby/gems/1.8/gems/mongrel-0.3.13.4/bin/mongrel_rails:231
/Applications/Locomotive2/Bundles/standardRailsSept2006.locobundle/i386/bin/mongrel_rails:18

Oh I forgot to add, that these errors all go away and everything works 
fine when I remove the :analyzer option from aaf.

Thanks,
Ray

Well after a few hours of playing with it, I finally gave up and decided 
to try a different stem analyzer from another post.  Specifically:
require 'rubygems'
require 'ferret'

class StemmedAnalyzer < Ferret::Analysis::Analyzer
  include Ferret::Analysis
  def initialize(stop_words = ENGLISH_STOP_WORDS)
    @stop_words = stop_words
  end
  def token_stream(field, str)
    StemFilter.new(StopFilter.new(LowerCaseFilter.new(StandardTokenizer.new(str)), 
@stop_words))
  end
end

The error seems to have disappeared now.  Very strange indeed.

On Fri, Dec 08, 2006 at 06:20:11AM +0100, Raymond O'connor wrote:
> Well after a few hours of playing with it, I finally gave up and decided 
> to try a different stem analyzer from another post.  Specifically:
> require 'rubygems'
> require 'ferret'
> 
> class StemmedAnalyzer < Ferret::Analysis::Analyzer

[..]

> The error seems to have disappeared now.  Very strange indeed.

looks like inheriting from Analyzer is essential...

Jens


--
webit! Gesellschaft für neue Medien mbH          www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer       kraemer@webit.de
Schnorrstraße 76                         Tel +49 351 46766  0
D-01069 Dresden                          Fax +49 351 46766 66

> looks like inheriting from Analyzer is essential...

Not necessarily.  I had a similar problem when I upgraded to Rails 1.2, 
and the solution seems to have been putting the 'include' statement 
inside the class, instead of outside.  It looks like the same change 
happened here, in addition to the change in inheritance.