# Using tokenizers?

**URL:** <https://www.ruby-forum.com/t/using-tokenizers/149691>\
**Category:** Ferret\
**Created:** [September 28, 2008, 12:20pm UTC](https://www.ruby-forum.com/t/using-tokenizers/149691 "2008-09-28T12:20:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lyes](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/ed8c4c/32.png) [@lyes](https://www.ruby-forum.com/u/lyes)\
**Post date:** [September 28, 2008, 12:20pm UTC](https://www.ruby-forum.com/t/using-tokenizers/149691/1 "2008-09-28T12:20:51Z")

</div>

Hi lis

I using Ferret to index some files for a specific usage.

I want to know how can I set a tokenizer for some of my index fields and  
if  
I can choose a different tokenizer for each field. For example:

If in my document I have two fields :F1 and :F2. What I have to do if I  
want  
that the field :F1 will be tokenized with a StandardTokenizer and :F2  
with  
the WhiteSpaceTokenizer??

thank you

---

<div class="post-metadata">

**Author:** ![lyes](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/ed8c4c/32.png) [@lyes](https://www.ruby-forum.com/u/lyes)\
**Post date:** [October 1, 2008, 9:55pm UTC](https://www.ruby-forum.com/t/using-tokenizers/149691/2 "2008-10-01T21:55:42Z")

</div>

Hi!

have a look at the PerFieldAnalyzer:  
[http://ferret.davebalmain.com/api/classes/Ferret/Analysis/PerFieldAnalyzer.html](http://ferret.davebalmain.com/api/classes/Ferret/Analysis/PerFieldAnalyzer.html)

cheers,  
Jens

On 28.09.2008, at 12:20, Lyes A. wrote:

> [http://rubyforge.org/mailman/listinfo/ferret-talk](http://rubyforge.org/mailman/listinfo/ferret-talk)  
> –  
> Jens Krämer  
> webit! Gesellschaft für neue Medien mbH  
> Schnorrstraße 76 | 01069 Dresden  
> Telefon +49351467660 | Telefax +493514676666  
> removed\_email\_address@domain.invalid | [www.webit.de](http://www.webit.de)

Amtsgericht Dresden | HRB 15422  
GF Sven Haubold

---

<div class="post-metadata">

**Author:** ![lyes](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/ed8c4c/32.png) [@lyes](https://www.ruby-forum.com/u/lyes)\
**Post date:** [October 2, 2008, 11:58am UTC](https://www.ruby-forum.com/t/using-tokenizers/149691/3 "2008-10-02T11:58:50Z")

</div>

Ok thank you, it was what I looked for