Actually the 1.8 parser is somewhat influenced by the current $KCODE.
Multi-byte characters could be part of identifiers and also the decision
of where a string literal ends needs to deal with multi-byte characters.
However, the resulting literals are just plain byte arrays with no
knowledge of encoding so String#size method is still broken.
To achieve a better .NET interop in IronRuby, we will honor KCODE when
creating MutableStrings. The representation of the string will be byte[]
if it contains any non-ascii characters and KCODE is set to a non-ascii
encoding. We will also attach the KCODE encoding to the MutableString at
creation time. This doesn’t affect Ruby 1.8 functionality, it only
affects conversions to CLR string. So if you use KCODE = “U†the CLR
strings should be correctly encoded (they are not now as you are
experiencing). I’ll implement this feature as soon as possible.
Tomas
From: [email protected]
[mailto:[email protected]] On Behalf Of Tomas M.
Sent: Tuesday, March 03, 2009 10:36 AM
To: [email protected]
Subject: Re: [Ironruby-core] Issue with accents (UTF-8) - is it supposed
to work ?
If I run this in Ruby 1.8.6:
ruby –Ku uni.rb
And uni.rb is UTF-8 encoded w/o BOM:
puts $KCODE
puts ‘hèllo’.size
I’ll get output:
UTF-8
6
So that clearly doesn’t work as one might expect. String literals in MRI
1.8 are always binary (ie. the accented character is stored as any other
2 bytes in the string).
AFAIK $KCODE only affects some built-in and library methods – for
example String#inspect, regular expression, conversion libraries, etc.
Although IronRuby stores string literals in UTF16 .NET strings, to be
fully compatible with MRI 1.8 we use a custom BinaryEncoding for these
strings. When a string is converted to an array of bytes using this
encoding, only 8 bits of each character are used (the other bits are
required to be 0). This works fine for encodings that use a single byte
per character. It’s broken for multi-byte encodings but that’s a problem
with Ruby 1.8 in general.
If you want to use Unicode you should not use 1.8 semantics. You should
use -19 switch to run your script in 1.9 mode and either add a UTF8 BOM
preamble or Ruby encoding magic comment:
#encoding: UTF-8
puts ‘hèllo’.size
ruby19 uni.rb
5
ir.exe -19 uni.rb
5
In a hosted app you can set 1.9 compat mode when creating the
ScriptEngine/Runtime:
var ruby = IronRuby.Ruby.CreateEngine((setup) => {
setup.Options[“Compatibility”] = RubyCompatibility.Ruby19
});
Tomas
From: [email protected]
[mailto:[email protected]] On Behalf Of Tomas M.
Sent: Tuesday, March 03, 2009 9:56 AM
To: [email protected]
Subject: Re: [Ironruby-core] Issue with accents (UTF-8) - is it supposed
to work ?
I’ll take a look.
Tomas
From: [email protected]
[mailto:[email protected]] On Behalf Of Ivan Porto
Carrero
Sent: Tuesday, March 03, 2009 6:58 AM
To: [email protected]
Subject: Re: [Ironruby-core] Issue with accents (UTF-8) - is it supposed
to work ?
No not a mono related issue. I get the same results when i run your
sample on windows with MS.NEThttp://MS.NET
It must be an encoding thing. When I set the $KCODE to “UTF-8” it still
has the same behavior which is weird I guess
On Tue, Mar 3, 2009 at 3:35 PM, Thibaut Barrère
<[email protected]mailto:[email protected]> wrote:
Hi,
not sure if it’s an oddity in my code, a bug or non-implemented feature in
IronRuby or Mono - so I’m reporting it here. When using accents inside
strings (“Barrère”) that I pass to either buttons or datagridviews, they
translate into “BarrA¨re”. Here’s a sample (also available on github):
Bumping this one - do you have some idea of what’s happening there ?
Is it a mono related issue ?
– Thibaut
@grid = data_grid_view :dock => DockStyle.fill
009-03-01 11:48:36.927 mono[5512:10b] WARNING:
CFSTR(“Barr\37777777703\37777777603\37777777702\37777777650re”) has non-7
bit chars, interpreting using MacOS Roman encoding for now, but this will
change. Please eliminate usages of non-7 bit chars (including escaped
characters above \177 octal) in CFSTR().
So I guess the issue probably boils down to non-MacOS Roman support in Mono.
What do you think ?
– Thibaut
Ironruby-core mailing list
[email protected]mailto:[email protected]
http://rubyforge.org/mailman/listinfo/ironruby-core