# Tokenizing text

**URL:** <https://www.ruby-forum.com/t/tokenizing-text/161276>\
**Category:** Ruby\
**Created:** [February 25, 2009, 12:07am UTC](https://www.ruby-forum.com/t/tokenizing-text/161276 "2009-02-25T00:07:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![alce](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/f07891/32.png) [@alce](https://www.ruby-forum.com/u/alce)\
**Post date:** [February 25, 2009, 12:07am UTC](https://www.ruby-forum.com/t/tokenizing-text/161276/1 "2009-02-25T00:07:49Z")

</div>

Hello,

I need to tokenize English text into sentences. I realize this is a very  
complex task to get right all of the time (if possible at all) but for  
the time being I’m only trying to implement a better solution than  
strintg.split(’.’).

Bowsing around I found this snippet:

string.scan( /\w.+?[.!?]+(?=\s|\Z)/ )

which almost works for what I need except for two cases: ellipses and at  
least most common abbreviations. Abbreviations are the hardest part and  
I’ve been tinkering with a couple possible solutions. How would you  
approach this?

Thanks in advance  
Juan