# Using unicode in YAML

**URL:** <https://www.ruby-forum.com/t/using-unicode-in-yaml/122889>\
**Category:** Ruby\
**Created:** [December 12, 2007, 7:26pm UTC](https://www.ruby-forum.com/t/using-unicode-in-yaml/122889 "2007-12-12T19:26:37Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [December 12, 2007, 7:26pm UTC](https://www.ruby-forum.com/t/using-unicode-in-yaml/122889/1 "2007-12-12T19:26:37Z")

</div>

I’ve done my research, and it appears that the current ruby YAML  
implementation doesn’t really grok unicode. What I want to know is  
whether anything has changed in this regard, or is likely to in the  
future. It doesn’t appear that syck has been updated since May '05,  
but if it’s something I could fix, I’d be willing to do it.

In any case, here’s what I need to do: read in YAML files containing  
strings in various languages including Japanese and write the same  
strings back out unmolested. UTF-8 seems like the natural choice, but  
the encoding could be different, so long as I can do some processing  
and keep the strings human readable when I spit them back out. I don’t  
even need to modify the strings themselves, just modify sets of them  
and output.

Are ruby and YAML just not an option here? Any other suggestions?

If you’re not familiar, here’s the basic problem. I have the following  
YAML:

jp: “e$B$O$$e(B”

(If that doesn’t display right, it’s just the Japanese characters for  
the word “yes”.)

I read it in via YAML.load and I get:

{"\357\273\277jp"=\>"\343\201\257\343\201\204"}

OK, not so bad, the UTF-8 indicator is on the front there, but I can  
deal with that, and the six bytes in octal do indeed correspond to the  
UTF-8 codes for the two characters I expect. The problem is when I try  
to put this back out, and YAML decides to take my string and convert  
it to binary data:

* * *

“\xEF\xBB\xBFjp”: !binary |  
44Gv44GE

Blah.

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [December 30, 2007, 10:40pm UTC](https://www.ruby-forum.com/t/using-unicode-in-yaml/122889/2 "2007-12-30T22:40:45Z")

</div>

Hello baumanj,

\xEF\xBB\xBF from your example above is the byte order mark (BOM)  
that is needed to identify UTF-8-files. \357\273\277 are just the  
octal numbers that mean \xEF\xBB\xBF in hex. Have you tried to remove  
these three bytes manually before writing your string as YAML? I  
currently have a similar problem, and I just found out that these  
three bytes are imported (wrongly, I think) via YAML#load\_file into  
one of my objects (in your case: the “jp” key).

So try this:

```
  f = File.open("jp.txt", "r")
  raw = f.read
  f.close
  #remove bom
  raw_without_bom = raw[3..-1]
  #now change to yaml
  hash = YAML::load(raw_without_bom)

```

Hope that helps.

Happy new year!

Rainer