# Custom Tokenizer not working

**URL:** <https://www.ruby-forum.com/t/custom-tokenizer-not-working/135625>\
**Category:** Ferret\
**Created:** [April 23, 2008, 6:25pm UTC](https://www.ruby-forum.com/t/custom-tokenizer-not-working/135625 "2008-04-23T18:25:12Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![S\_D](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@S\_D](https://www.ruby-forum.com/u/S_D)\
**Post date:** [April 23, 2008, 6:25pm UTC](https://www.ruby-forum.com/t/custom-tokenizer-not-working/135625/1 "2008-04-23T18:25:12Z")

</div>

First, thanks to Jens K. for pointing a stupid error on my part  
regarding  
the use of test\_token\_stream().

My current problem, a custom tokenizer I’ve written in Ruby does not  
properly create an index (or at least searches on the index don’t work).  
Using test\_token\_stream() I have verified that my tokenizer properly  
creates  
the token\_stream; certainly each Token’s attributes are set properly.  
Nevertheless, simple searches return zero results.

The essence of my tokenizer is to skip beyond XML tags in a file and  
break  
up and return text components as tokens. I use this approach as opposed  
to  
an Hpricot approach because I need to keep track of the location of the  
text  
with respect to XML tags since after a search for a phrase I’ll want to  
extract the nearby XML tags as they contain important context. My  
tokenizer  
(XMLTokenizer) contains a the obligatory initialize, next and text  
methods  
(shown below) as well as a lot of parsing methods that are called at the  
top  
level by the method XMLTokenizer.get\_next\_token which is the primary  
action  
within next. I didn’t add the details of get\_next\_token as I’m assuming  
that  
if each token produced by get\_next\_token has the proper attributes then  
it  
shouldn’t be the cause of the problem. What more should I be looking  
for?  
I’ve been looking for a custom tokenizer written in Ruby to model after;  
any  
suggestions?

```
def initialize(xmlText)
  @xmlText = xmlText.gsub(/[;,!]/, ' ')
  @currPtr = 0
  @currWordStart = nil
  @currTextStart = 0
  @nextTagStart = 0
  @startOfTextRegion = 0

  @currTextStart = \
    XMLTokenizer.skip_beyond_current_tag(@currPtr, @xmlText)
  @nextTagStart = \
    XMLTokenizer.skip_beyond_current_text(@currTextStart, @xmlText)
  @currPtr = @currTextStart
  @startOfTextRegion = 1
end

def next
  tkn = get_next_token
  if tkn != nil
    puts "%5d |%4d |%5d | %s" % [tkn.start, tkn.end, tkn.pos_inc,

```

tkn.text]  
end  
return tkn  
end

```
def text=(text)
  initialize(text)
  @xmlText
end

```

Below is text from a previous, related message that shows that  
StopFiltering  
is not working:

> - I’ve written a tokenizer/analyzer that parses a file extracting tokens and  
> _\>_ operate this analyzer/tokenizer on ASCII data consisting of XML  
> files (the  
> _\>_ tokenizer skips over XML elements but maintains relative  
> positioning). I’ve  
> _\>_ written many units tests to check the produced token stream and was  
> _\>_ confident that the tokenizer was working properly. Then I noticed  
> two  
> _\>_ problems:  
> _\>_  
> _\>_ 1. StopFilter (using English stop words) does not properly filter  
> the  
> _\>_ token stream output from my tokenizer. If I explicitly pass an  
> array of stop  
> _\>_ words to the stop filter it still doesn’t work. If I simply  
> switch my  
> _\>_ tokenizer to a StandardTokenizer the stop words are  
> appropriately filtered  
> _\>_ (of course the XML tags are treated differently).  
> \*\>
> - 
> 1. When I try a simple search no results come up. I can see that my  
> _\>_ tokenizer is adding files to the index but a simple search (using  
> _\>_ Ferret::Index::Index.search\_each) produces no results.

- 

Any suggestions are appreciated.

John

---

<div class="post-metadata">

**Author:** ![S\_D](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@S\_D](https://www.ruby-forum.com/u/S_D)\
**Post date:** [April 23, 2008, 7:09pm UTC](https://www.ruby-forum.com/t/custom-tokenizer-not-working/135625/2 "2008-04-23T19:09:05Z")

</div>

Hi!

On Wed, Apr 23, 2008 at 12:18:12PM -0400, S D wrote:  
[…]

> My current problem, a custom tokenizer I’ve written in Ruby does not  
> properly create an index (or at least searches on the index don’t work).  
> Using test\_token\_stream() I have verified that my tokenizer properly creates  
> the token\_stream; certainly each Token’s attributes are set properly.  
> Nevertheless, simple searches return zero results.

Could you have a look at your index with the ferret\_browser utility? It  
allows you to check what exactly has been indexed and that maybe leads  
to the root of your problem.

What does your analyzer, where you use the Tokenizer, look like? Is your  
next() method below being called and working correctly when test driving  
your analyzer i.e. in irb?

Cheers,  
Jens

> shouldn’t be the cause of the problem. What more should I be looking for?
> 
> ```
> if tkn != nil
> 
> ```
> 
> _\>_ 1. StopFilter (using English stop words) does not properly filter the  
> \*
> 
> Any suggestions are appreciated.
> 
> John

> * * *
> 
> Ferret-talk mailing list  
> removed\_email\_address@domain.invalid  
> [http://rubyforge.org/mailman/listinfo/ferret-talk](http://rubyforge.org/mailman/listinfo/ferret-talk)

–  
Jens Krämer  
Finkenlust 14, 06449 Aschersleben, Germany  
VAT Id DE251962952  
[http://www.jkraemer.net/](http://www.jkraemer.net/) - Blog  
[http://www.omdb.org/](http://www.omdb.org/) - The new free film database