When I run it by launching ir.exe without any option I get 9 as an
output (each character in that string is actually made up of 3 bytes
with UTF-8 encoding), and when I do the same with the -KU option being
passed to ir.exe I get 3. Aside from the fact that I think that 3 is
to be considered the right behaviour here, shouldn’t the sole $KCODE =
“U” have the same effect of starting ir.exe with the -KU option?
Setting $KCODE = “U” doesn’t actually affect the encoding of the literal
in the same compilation unit. It only affects literals that are parsed
after the KCODE is set.
$KCODE = “U”
x = “日本語”
p x.Encoding # => ASCII-8BIT since the current compilation unit (a
file) was parsed using BINARY encoding
p x.size # => 9 bytes
y = eval(’“日本語”’)
p y.Encoding # => KCODE: UTF8
p y.size # => 9 since String#size in MRI 1.8.6 doesn’t understand
encodings, it counts in bytes
c = x.to_clr_string # this is essentially creating a string whose non
ASCII characters are not correctly encoded in UTF8 (they are UTF8 bytes
widened to 16bits)
p c.size # => 9 characters
p c.Encoding # => UTF-8 since CLR string doesn’t hold on an
encoding. When you ask for its bytes we need to use some encoding.
# Maybe we could choose UTF16 but MRI 1.8.6 has at least some
support for.
d = y.to_clr_string # correctly encoded string
d c.Encoding # UTF-8
p d.size # 3 characters
Encodings in 1.8.6 are not very well supported and it is difficult to
implement good interop between CLR and MRI strings. It would get better
in the next version of IronRuby which will target compatibility with
1.9.
Setting $KCODE = “U” doesn’t actually affect the encoding of the literal in the same compilation unit. It only affects literals that are parsed after the KCODE is set.
I was missing the subtle difference, now it makes sense to me. Thanks
