# Handling Carriage Returns

**URL:** <https://www.ruby-forum.com/t/handling-carriage-returns/136024>\
**Category:** Ferret\
**Created:** [April 28, 2008, 9:05am UTC](https://www.ruby-forum.com/t/handling-carriage-returns/136024 "2008-04-28T09:05:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![S\_D](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@S\_D](https://www.ruby-forum.com/u/S_D)\
**Post date:** [April 28, 2008, 9:05am UTC](https://www.ruby-forum.com/t/handling-carriage-returns/136024/1 "2008-04-28T09:05:04Z")

</div>

It’s my understanding that the tokens in a token\_stream consist of text  
along with start/stop positions that represent the byte positions of the  
text within the corresponding document field. The documentation I’ve  
been  
reading (i.e., O’Reilly - Ferret - page 67) suggests that these byte  
positions represent positions within the entire field but based on my  
testing it appears that the byte positions are with respect to the line  
that  
contains the corresponding text within the field. I read my fields  
following  
Brian McCallister:

```
  index.add_document :file => path,
                     :content => file.readlines

```

Hence, if I have a file that contains carriage returns, the token  
positions  
will be reset with each new line. For example, the following file  
contents  
(File A)  
this is a sentence  
will result in a token for the text “sentence” with start position equal  
to  
10 (assume “this” starts in position 0) while a file with a carriage  
return  
this is a  
sentence  
will result in a token for the text “sentence” with start position equal  
to  
0. I get the same results for my custom tokenizer as well as  
StandardTokenizer. The above does not seem consistent with the  
documentation  
but more importantly, it seems that global positions are more useful  
than  
line-based positions (e.g., for highlighting).

Digging a little deeper it seems that the tokenizer’s initialize method  
is  
called each time the token\_stream method of the containing analyzer is  
called:

class CustomAnalyzer  
def token\_stream(field, str)  
ts = StandardTokenizer.new(str)  
end  
end

Am I missing something here? Are the start/stop byte positions intended  
to  
be with respect to the line? Is there a way for token\_stream to only be  
called once for an entire string sequence (even if carriage returns are  
contained)?

Thanks,  
John

---

<div class="post-metadata">

**Author:** ![S\_D](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@S\_D](https://www.ruby-forum.com/u/S_D)\
**Post date:** [April 28, 2008, 12:37pm UTC](https://www.ruby-forum.com/t/handling-carriage-returns/136024/2 "2008-04-28T12:37:37Z")

</div>

Hi,

File.readlines returns an array which I think is the root cause of the  
problem.  
Just using File.read instead should solve your problem.

Cheers,  
Jens

On Mon, Apr 28, 2008 at 03:04:36AM -0400, S D wrote:

> ```
> :content => file.readlines
> 
> ```
> 
> will result in a token for the text “sentence” with start position equal to  
> def token\_stream(field, str)  
> John
> 
> * * *
> 
> Ferret-talk mailing list  
> removed\_email\_address@domain.invalid  
> [http://rubyforge.org/mailman/listinfo/ferret-talk](http://rubyforge.org/mailman/listinfo/ferret-talk)

–  
Jens Krämer  
Finkenlust 14, 06449 Aschersleben, Germany  
VAT Id DE251962952  
[http://www.jkraemer.net/](http://www.jkraemer.net/) - Blog  
[http://www.omdb.org/](http://www.omdb.org/) - The new free film database

---

<div class="post-metadata">

**Author:** ![S\_D](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/aca169/32.png) [@S\_D](https://www.ruby-forum.com/u/S_D)\
**Post date:** [April 30, 2008, 7:54am UTC](https://www.ruby-forum.com/t/handling-carriage-returns/136024/3 "2008-04-30T07:54:12Z")

</div>

That was it. Stupid mistake on my part.

Thanks!  
John