# Ruby1.9 Encoding

**URL:** <https://www.ruby-forum.com/t/ruby1-9-encoding/175265>\
**Category:** Ruby\
**Created:** [September 10, 2009, 11:20am UTC](https://www.ruby-forum.com/t/ruby1-9-encoding/175265 "2009-09-10T11:20:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Juliano\_SSS\_SSSS\_SS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/77aa72/32.png) [@Juliano\_SSS\_SSSS\_SS](https://www.ruby-forum.com/u/Juliano_SSS_SSSS_SS)\
**Post date:** [September 10, 2009, 11:20am UTC](https://www.ruby-forum.com/t/ruby1-9-encoding/175265/1 "2009-09-10T11:20:57Z")

</div>

Hey, guys!

I’ve just started learning Ruby from Python and recently posted a  
question here that was promptly and effectively answered (thanks,  
Glenn!), so I decided to come here once more. I hope I’ll be able to  
be answering some of the questions on my own soon. 😉

Here it is.

I’m writing a wrapper for a Korean Morphological Parser that only  
works with EUC-KR encoding and has some trouble with longer texts. So,  
first, I have to preprocess the input text to divide it into  
sentences, remove unicode characters which are not related to Korean  
and save them for further reinclusion in the postprocessing stage.  
This has worked out wonderfully and, I should say, easier than what  
I’d done in Python.

The problem I’m having now is converting the string from UTF-8 (I’m  
running Ubuntu with pt\_BR.UTF-8 locale) into EUC-KR, run the  
Morphological Parser, read its output and process it. I have to parse  
a whole bunch of data “spidered” from the internet and it works great  
until the encoder comes across a Korean typo… Let’s make this point  
clearer: the EUC-KR encoding does not cover all the possible  
combinations of initial and final consonants + vowels, as does  
unicode. Explaining: Unicode has 21708 codepoints for hangul whereas  
EUC-KR has only 11172. In fact, most of the extra chars are not used  
in day by day life, but still they can be used as abbreviations,  
slang, smileys or typos…

In Python, I would simply throw these chars away but I really didn’t  
manage to understand the “Ruby encoding way”… I know I’m missing  
something, but I can’t seem to find enough info around… Google  
doesn’t seem to know much of this either… So, I’m coming here to ask  
for your enlightenment, dear rubyist friends!

The part of my code which deals with this is as follows:

def run(txt)  
txt = txt.encode(“EUC-KR”)  
kts\_file = Tempfile::new(‘kts\_text’)  
kts\_file = open(kts\_file.path, “w:EUC-KR”)  
kts\_file \<\< “#{txt}\n”  
kts\_file.close  
cmd = “ktspell \< #{kts\_file.path}” # 2\> /dev/null"  
IO::popen(cmd, “r:EUC-KR”).read.encode(“UTF-8”)  
end

I found something about “ignoring” the non-existent codepoints, but it  
doesn’t work… I’m even thinking that my Ruby installation might have  
gotten corrupted somehow… Everytime I think I did it right, I still  
get The Exception popping up on the screen…

Thanks for your patience reading this looong post.

Juliano

---

<div class="post-metadata">

**Author:** ![Juliano\_SSS\_SSSS\_SS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/77aa72/32.png) [@Juliano\_SSS\_SSSS\_SS](https://www.ruby-forum.com/u/Juliano_SSS_SSSS_SS)\
**Post date:** [September 10, 2009, 12:06pm UTC](https://www.ruby-forum.com/t/ruby1-9-encoding/175265/2 "2009-09-10T12:06:37Z")

</div>

-------- Original-Nachricht --------

> Datum: Thu, 10 Sep 2009 18:20:06 +0900  
> Von: “Juliano 준호” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> An: removed\_email\_address@domain.invalid  
> Betreff: Ruby1.9 Encoding

> works with EUC-KR encoding and has some trouble with longer texts. So,  
> until the encoder comes across a Korean typo… Let’s make this point  
> doesn’t seem to know much of this either… So, I’m coming here to ask  
> cmd = “ktspell \< #{kts\_file.path}” # 2\> /dev/null"  
> Juliano  
> Dear Juliano,

a disclaimer first: I know no Korean, so what’s below might not work.

I’ve had to do some coding to resolve Arabic ligatures (combinations  
of two letters) recently. Similarly as what you describe, there is most  
of the time no need to use a special combined form, and unluckily, the  
same word is sometimes spelled in this and sometimes in that way, giving  
a list of duplicate words.

I used a list of Unicode characters with names of the individual  
characters  
to solve that problem.

You might find the table below on this page useful :

[http://www.kfunigraz.ac.at/~katzer/korean\_hangul\_unicode.html](http://www.kfunigraz.ac.at/~katzer/korean_hangul_unicode.html)

I don’t know if that list is exhaustive, but you may try to individually  
convert each of the syllables listed there from Unicode to EUC::KR, and  
if that doesn’t work, decide what to do with the particular combination  
of signs, based on the Latin transcription, creating a transform hash  
for these encodings yourself.

There might also be some locale or OS-related problems with  
Iconv::IGNORE .  
There’s some discussion of this here :

> **[ActiveState Community](https://community.activestate.com)**
>
> Community support for users of the ActiveState Platform.

Best regards,

Axel

---

<div class="post-metadata">

**Author:** ![Juliano\_SSS\_SSSS\_SS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/77aa72/32.png) [@Juliano\_SSS\_SSSS\_SS](https://www.ruby-forum.com/u/Juliano_SSS_SSSS_SS)\
**Post date:** [September 10, 2009, 3:40pm UTC](https://www.ruby-forum.com/t/ruby1-9-encoding/175265/3 "2009-09-10T15:40:19Z")

</div>

Hi,

In message “Re: Ruby1.9 Encoding”  
on Thu, 10 Sep 2009 18:20:06 +0900, Juliano ÁØÈ£ [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
writes:

|The part of my code which deals with this is as follows:  
|  
| def run(txt)  
| txt = txt.encode(“EUC-KR”)  
| kts\_file = Tempfile::new(‘kts\_text’)  
| kts\_file = open(kts\_file.path, “w:EUC-KR”)  
| kts\_file \<\< “#{txt}\n”  
| kts\_file.close  
| cmd = “ktspell \< #{kts\_file.path}” # 2\> /dev/null"  
| IO::popen(cmd, “r:EUC-KR”).read.encode(“UTF-8”)  
| end  
|  
|I found something about “ignoring” the non-existent codepoints, but it  
|doesn’t work… I’m even thinking that my Ruby installation might have  
|gotten corrupted somehow… Everytime I think I did it right, I still  
|get The Exception popping up on the screen…

I had some difficulty to see your intention from the code. Could you  
show us the exception messages you’ve got?

```
          matz.

```

---

<div class="post-metadata">

**Author:** ![Juliano\_SSS\_SSSS\_SS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/77aa72/32.png) [@Juliano\_SSS\_SSSS\_SS](https://www.ruby-forum.com/u/Juliano_SSS_SSSS_SS)\
**Post date:** [September 10, 2009, 3:45pm UTC](https://www.ruby-forum.com/t/ruby1-9-encoding/175265/4 "2009-09-10T15:45:00Z")

</div>

On Sep 10, 2009, at 4:20 AM, Juliano 준호 wrote:

> Hey, guys!
> 
> I’ve just started learning Ruby from Python and recently posted a  
> question here that was promptly and effectively answered (thanks,  
> Glenn!), so I decided to come here once more. I hope I’ll be able to  
> be answering some of the questions on my own soon. 😉

Welcome to Ruby.

> slang, smileys or typos…
> 
> In Python, I would simply throw these chars away but I really didn’t  
> manage to understand the “Ruby encoding way”…

I think we can throw them away in Ruby too. See below.

> I know I’m missing something, but I can’t seem to find enough info  
> around… Google  
> doesn’t seem to know much of this either…

I wrote a lot about Ruby’s encoding engine on my blog:

[http://blog.grayproductions.net/articles/understanding\_m17n](http://blog.grayproductions.net/articles/understanding_m17n)

> The part of my code which deals with this is as follows:
> 
> def run(txt)  
> txt = txt.encode(“EUC-KR”)

Try replacing the above line with:

txt = txt.encode(“EUC-KR”, invalid: :replace, undef: :replace,  
replace: “”)

> kts\_file = Tempfile::new(‘kts\_text’)  
> kts\_file = open(kts\_file.path, “w:EUC-KR”)  
> kts\_file \<\< “#{txt}\n”  
> kts\_file.close  
> cmd = “ktspell \< #{kts\_file.path}” # 2\> /dev/null"  
> IO::popen(cmd, “r:EUC-KR”).read.encode(“UTF-8”)  
> end

Hope that helps.

James Edward G. II

---

<div class="post-metadata">

**Author:** ![Juliano\_SSS\_SSSS\_SS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/77aa72/32.png) [@Juliano\_SSS\_SSSS\_SS](https://www.ruby-forum.com/u/Juliano_SSS_SSSS_SS)\
**Post date:** [September 10, 2009, 5:09pm UTC](https://www.ruby-forum.com/t/ruby1-9-encoding/175265/5 "2009-09-10T17:09:17Z")

</div>

On Sep 10, 2009, at 09:39, Yukihiro M. wrote:

> I had some difficulty to see your intention from the code. Could you  
> show us the exception messages you’ve got?

My interpretation of the code is:

def run(txt)  
# translate the string into EUC-KR encoding  
txt = txt.encode(“EUC-KR”)

```
# Create a temp file to store the data and
# write it to the file, using the EUC-KR encoding
kts_file = Tempfile::new('kts_text')
kts_file = open(kts_file.path, "w:EUC-KR")
kts_file << "#{txt}\n"
kts_file.close

# Run ktspell, feeding it the data from the file
cmd = "ktspell < #{kts_file.path}" # 2> /dev/null"

# Read the result and translate it into UTF-8
IO::popen(cmd, "r:EUC-KR").read.encode("UTF-8")

```

end

I don’t know much about 1.9’s encodings, but can suggest a more  
rubyish way of writing the method:

def run(txt)  
euc\_txt = txt.encode(“EUC-KR”)

# Tempfile::new actually returns a filehandle, so there’s no need

to re-open

# the file based on the path. Since you seem to want to open the

file,

# write to it, use the data, and then immediately do away with the

file,

# the ‘open’ block form is probably more appropriate

kts\_file = Tempfile::open(“kts\_text”) do |kts\_file|

```
 # The more common way to write a newline-terminated string to a

```

file is  
# file.puts(foo) rather than file \<\< “#{foo}\n”  
kts\_file.puts(euc\_txt)  
cmd = “ktspell \< #{kts\_file.path}”

```
 # You could do all the rest on one line as you did, but this lets

```

you  
# look at the data first using ‘p processed\_euc\_txt’ or something  
processed\_euc\_txt = IO::popen(cmd, “r:EUC-KR”).read

```
 # Again, a temporary variable to let you see if the data looks

```

right.  
processed\_utf\_txt = processed\_euc\_txt.encode(“UTF-8”)  
end

processed\_utf\_txt  
end

Ben