# Encounter troubles with Regex in Chinese text splitting

**URL:** <https://www.ruby-forum.com/t/encounter-troubles-with-regex-in-chinese-text-splitting/49383>\
**Category:** Ruby\
**Created:** [December 3, 2005, 6:43am UTC](https://www.ruby-forum.com/t/encounter-troubles-with-regex-in-chinese-text-splitting/49383 "2005-12-03T06:43:28Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![meng.yan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/df705f/32.png) [@meng.yan](https://www.ruby-forum.com/u/meng.yan)\
**Post date:** [December 3, 2005, 6:43am UTC](https://www.ruby-forum.com/t/encounter-troubles-with-regex-in-chinese-text-splitting/49383/1 "2005-12-03T06:43:28Z")

</div>

Hi All,  
I’m a Ruby newbie. I’m writting a program to process a big chunk of  
Chinese text. The first step is to split the chunk of text into a list  
of sentences. In Chinese, all the characters are listed one by one  
without any natural boundary tag like space in English. Sentences are  
separated by one of three special characters(ã??ï¼?！). So at the  
first glance, I thought it’s a simple task:

# $chunk stores the text body

$sentenses = $chunk.split(/ã??|ï¼?|！/)

# now $sentenses holds the list of sentences.

By when I checked the result, I found some of the sentenses didn’t  
split well. For instance, here is a sentense:  
“ä½ æ²¡ç??ï¼?ä»?å?¢ï¼?” (means “You are not sick, how about him?”) . In  
GB2312, “ç??ï¼?” is encoded to (hex) b2a1 a3ac, and “ã??” happens to be  
encoded to (hex) a1a3. So the String#split method finds there is a  
“ã??” in the middle of the sentense and incorrectly do the splitting.

Certainly this is because the String#split (and the Ruby regex  
engine) is byte-oriented instead of true character-oriented, and it’s a  
frequent problem in i18n domain. Is there any ways in Ruby to correct  
split Chinese text?

Thanks in advance.

myan

---

<div class="post-metadata">

**Author:** ![meng.yan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/df705f/32.png) [@meng.yan](https://www.ruby-forum.com/u/meng.yan)\
**Post date:** [December 3, 2005, 7:15am UTC](https://www.ruby-forum.com/t/encounter-troubles-with-regex-in-chinese-text-splitting/49383/2 "2005-12-03T07:15:50Z")

</div>

Hi,

> without any natural boundary tag like space in English. Sentences are  
> GB2312, “??ï¼?” is encoded to (hex) b2a1 a3ac, and "??quot; happens to be  
> myan

Try the script with $KCODE = “E”

Hope this help,

Park H.

---

<div class="post-metadata">

**Author:** ![meng.yan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/df705f/32.png) [@meng.yan](https://www.ruby-forum.com/u/meng.yan)\
**Post date:** [December 3, 2005, 2:11pm UTC](https://www.ruby-forum.com/t/encounter-troubles-with-regex-in-chinese-text-splitting/49383/3 "2005-12-03T14:11:21Z")

</div>

Hi Park,  
It works. Thank you very much!

Could you please tell me the reason and where can I find relevant  
documents?

Thank you.

myan

---

<div class="post-metadata">

**Author:** ![meng.yan](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/df705f/32.png) [@meng.yan](https://www.ruby-forum.com/u/meng.yan)\
**Post date:** [December 3, 2005, 3:12pm UTC](https://www.ruby-forum.com/t/encounter-troubles-with-regex-in-chinese-text-splitting/49383/4 "2005-12-03T15:12:42Z")

</div>

Hi,  
----- Original Message -----  
From: “Mike Meng” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
Newsgroups: comp.lang.ruby  
To: “ruby-talk ML” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
Sent: Saturday, December 03, 2005 10:07 PM  
Subject: Re: Encounter troubles with Regex in Chinese text splitting

> 

$KCODE is the character coding system Ruby handles. If the first  
character  
of $KCODE is `e' or `E’, Ruby handles EUC. If it is `s' or `S’, Ruby  
handles  
Shift\_JIS. If it is `u' or `U’, Ruby handles UTF-8. If it is `n' or `N’,  
Ruby doesn’t handle multi-byte characters. The default value is “NONE”.

Regards,

Park H.