# Re: Custom Tokenizer not working

**URL:** <https://www.ruby-forum.com/t/re-custom-tokenizer-not-working/135677>\
**Category:** Ferret\
**Created:** [April 24, 2008, 5:06am UTC](https://www.ruby-forum.com/t/re-custom-tokenizer-not-working/135677 "2008-04-24T05:06:12Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![S\_D](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@S\_D](https://www.ruby-forum.com/u/S_D)\
**Post date:** [April 24, 2008, 5:06am UTC](https://www.ruby-forum.com/t/re-custom-tokenizer-not-working/135677/1 "2008-04-24T05:06:12Z")

</div>

I changed XMLAnalyzer so that it does not inherit from  
Ferret::Analysis::Analyzer. That seemed to have no effect.

I successfully ran ferret-browser.As shown below, I am using two  
fields - :file and :content. When I browse through  
the “file” term everything appears fine; all of the filenames are  
found. The “content” term on the other hand is  
empty. Apparently I’m not stuffing the tokens in the index at all. One  
question I have is exactly what should happen  
in the Tokenizer#text method and when will this method be called?

Thanks,  
John

=====  
index = Index::Index.new(:analyzer =\> XMLAnalyzer.new(),  
:path =\> options.indexLocation,  
:create =\> true)

# Find.find(options.searchPath) do |path| if FileTest.file? path File.open(path) do |file| puts "Adding file to index: " + path index.add\_document :file =\> path, :content =\> file.readlines end end end

Hi!

On Wed, Apr 23, 2008 at 01:59:32PM -0400, S D wrote:

> - [unfortunately I received my messages as a batched digest…hence, I’m  
> _\>_ forced to respond in a new thread. I’ve requested the administrator  
> to  
> _\>_ change my config to receive each message on this list. Sorry for any  
> _\>_ inconvenience]  
> _\>_  
> _\>_ Thanks for the response below. Here is XMLAnalyzer (currently I’m  
> not using  
> _\>_ the stop or lower case filter):  
> _\>_  
> _\>_ class XMLAnalyzer \< Ferret::Analysis::Analyzer

- 

could you try if not inheriting from Ferret’s Analyzer changes anything?  
At least I usually don’t do that in my analyzers.

[…]

> - I just tried running ferret-browser by pointing to an index created with  
> _\>_ StandardTokenizer and got the error below in Firefox. Is there any  
> _\>_ configuration that is necessary? Presumably the defaults should  
> work.  
> \*[…]
> - Internal Server Error No such file or directory -  
> _\>_ /usr/local/lib/site\_ruby/1.8/ferret/browser/views/error/index.rhtml  
> _\>_ ------------------------------  
> _\>_ WEBrick/1.3.1 (Ruby/1.8.6/2007-06-07) at 127.0.0.1:3301

- 

works just fine here (Ferret 0.11.6 / Ubuntu), just tried it out. The  
location from the error message looks a bit strange to me, how did you  
install ferret?

Cheers,  
Jens

–  
Jens Krämer  
webit! Gesellschaft für neue Medien mbH  
Schnorrstraße 76 | 01069 Dresden  
Telefon +49 351 46766-0 | Telefax +49 351 46766-66  
kraemer at [webit.de](http://webit.de)  
[http://rubyforge.org/mailman/listinfo/ferret-talk](http://rubyforge.org/mailman/listinfo/ferret-talk) | [www.webit.de](http://www.webit.de)

Amtsgericht Dresden | HRB 15422  
GF Sven Haubold