# Text parser (text into sentences) that works with UTF-8 and **URL:** \ **Category:** Ruby\ **Created:** [July 30, 2007, 10:52am UTC](https://www.ruby-forum.com/t/text-parser-text-into-sentences-that-works-with-utf-8-and/110726 "2007-07-30T10:52:12Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![asplake](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/2bfe46/32.png) [@asplake](https://www.ruby-forum.com/u/asplake)\ **Post date:** [July 30, 2007, 10:52am UTC](https://www.ruby-forum.com/t/text-parser-text-into-sentences-that-works-with-utf-8-and/110726/1 "2007-07-30T10:52:12Z")
Hi all, I have to parse about 2000 files that are written in multiple languages (some English, some Korean, some Arabic and some Japanese). I have to split these UTF-8 encoded into individual sentences. Has anyone written a good parser that can parse all these non-Latin character languages or can someone give me some advice on how to go about writing a parser that can handle all these fairly different languages? Thank you, Mike ---
**Author:** ![asplake](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/2bfe46/32.png) [@asplake](https://www.ruby-forum.com/u/asplake)\ **Post date:** [July 30, 2007, 11:27am UTC](https://www.ruby-forum.com/t/text-parser-text-into-sentences-that-works-with-utf-8-and/110726/2 "2007-07-30T11:27:16Z")
2007/7/30, mike b. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > I have to parse about 2000 files that are written in multiple > languages (some English, some Korean, some Arabic and some Japanese). > I have to split these UTF-8 encoded into individual sentences. Has > anyone written a good parser that can parse all these non-Latin > character languages or can someone give me some advice on how to go > about writing a parser that can handle all these fairly different > languages? I would consider doing this in Java, as Java’s regular expressions support Unicode. That might make the job much easier. OTOH, if all files use only dot, question mark etc. (i.e. ASCII chars) as sentence delimiters then Ruby’s regular expressions might as well do the job. Kind regards robert ---
**Author:** ![asplake](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/2bfe46/32.png) [@asplake](https://www.ruby-forum.com/u/asplake)\ **Post date:** [July 30, 2007, 12:02pm UTC](https://www.ruby-forum.com/t/text-parser-text-into-sentences-that-works-with-utf-8-and/110726/3 "2007-07-30T12:02:18Z")
On Jul 30, 11:26 am, “Robert K.” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I would consider doing this in Java, as Java’s regular expressions > support Unicode. That might make the job much easier. OTOH, if all > files use only dot, question mark etc. (i.e. ASCII chars) as sentence > delimiters then Ruby’s regular expressions might as well do the job. Ruby supports UTF-8 regular expressions: for example, /\w+|\W/u can be used to scan a string splitting it into words and non-words. There were some bugs with Unicode character classifications in older versions of Ruby, but I’m not aware of any in 1.8.6; OTOH I’ve never tried it with non-latin text so I don’t know if it works correctly in those cases too. ---
**Author:** ![asplake](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/2bfe46/32.png) [@asplake](https://www.ruby-forum.com/u/asplake)\ **Post date:** [July 30, 2007, 2:54pm UTC](https://www.ruby-forum.com/t/text-parser-text-into-sentences-that-works-with-utf-8-and/110726/4 "2007-07-30T14:54:43Z")
On Jul 30, 2007, at 3:50 AM, mike b. wrote: > I have to parse about 2000 files that are written in multiple > languages (some English, some Korean, some Arabic and some Japanese). > I have to split these UTF-8 encoded into individual sentences. As has been stated, Ruby’s regular expression engine has a Unicode mode and that may be all you need here, depending on how you recognize sentence boundaries. > Has anyone written a good parser that can parse all these non-Latin > character languages or can someone give me some advice on how to go > about writing a parser that can handle all these fairly different > languages? I’ve released an initial version of my Ghost Wheel parser generator library. It doesn’t have documentation yet, but it was built using TDD and you should be able to look over the tests to see how it works. I’m also happy to answer questions. My hope is that it works fine for non-Latin languages, but I’ll confess that I haven’t tested it that way yet. I would try to fix any issues you uncovered though. James Edward G. II