# MutableString encoding issue **URL:** \ **Category:** IronRuby\ **Created:** [July 14, 2008, 8:21am UTC](https://www.ruby-forum.com/t/mutablestring-encoding-issue/142983 "2008-07-14T08:21:59Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Oleg\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/o/f19dbf/32.png) [@Oleg\_T](https://www.ruby-forum.com/u/Oleg_T)\ **Post date:** [July 14, 2008, 8:21am UTC](https://www.ruby-forum.com/t/mutablestring-encoding-issue/142983/1 "2008-07-14T08:21:59Z")
Stumbled on this when testing yaml. > > > “\204” > > > =\> “?” While irb(main):005:0\>"\204" =\> “\204” I believe Ruby string can hold arbitrary byte values, but as we are storing content as a string we are obviously losing all values that cannot be represented in default encoding. Tomas, what do you think? ---
**Author:** ![Oleg\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/o/f19dbf/32.png) [@Oleg\_T](https://www.ruby-forum.com/u/Oleg_T)\ **Post date:** [July 14, 2008, 3:25pm UTC](https://www.ruby-forum.com/t/mutablestring-encoding-issue/142983/2 "2008-07-14T15:25:07Z")
This was a known issue a while back, it’s the reason the Zlib library didn’t work well with binary files. I’m fairly certain there was work being done on making String be backed by a byte array… and in fact I thought this was already done. On Mon, Jul 14, 2008 at 2:20 AM, Oleg T. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) ---
**Author:** ![Oleg\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/o/f19dbf/32.png) [@Oleg\_T](https://www.ruby-forum.com/u/Oleg_T)\ **Post date:** [July 14, 2008, 3:28pm UTC](https://www.ruby-forum.com/t/mutablestring-encoding-issue/142983/3 "2008-07-14T15:28:02Z")
MutableString can have one of three internal representations, depending on how it was last used. One of these is a byte array. This particular problem may be in the scanner or parser and not in the actual string class, as we don’t otherwise have a problem storing the character: > > > $s = “\204” > > > =\> “?” > > > $s[0] > > > =\> 63 > > > $s[0] = 132 > > > =\> 132 > > > $s > > > =\> “\204” From: Michael L. [[mailto:removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)] Sent: Monday, July 14, 2008 6:21 AM To: removed\_email\_address@domain.invalid Cc: IronRuby Team Subject: Re: [Ironruby-core] MutableString encoding issue This was a known issue a while back, it’s the reason the Zlib library didn’t work well with binary files. I’m fairly certain there was work being done on making String be backed by a byte array… and in fact I thought this was already done. On Mon, Jul 14, 2008 at 2:20 AM, Oleg T. \ wrote: Stumbled on this when testing yaml. > > > “\204” =\> “?” While irb(main):005:0\>“\204” =\> “\204” I believe Ruby string can hold arbitrary byte values, but as we are storing content as a string we are obviously losing all values that cannot be represented in default encoding. Tomas, what do you think? – Oleg ---
**Author:** ![Oleg\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/o/f19dbf/32.png) [@Oleg\_T](https://www.ruby-forum.com/u/Oleg_T)\ **Post date:** [July 14, 2008, 6:20pm UTC](https://www.ruby-forum.com/t/mutablestring-encoding-issue/142983/4 "2008-07-14T18:20:45Z")
This problem is probably StringContent.ToByteArray(). It uses Encoding.GetBytes(string) which obeys .NET encoding semantics and by default replaces any nonconvertible characters to ‘?’. And then MutableStringOps.Dump() is using it to create string representation. We could make StringContent.ToByteArray() not replacing nonconvertible characters by using EncodingFallback. BinaryContent.ToString()/ToStringBuilder() also has the same issue. – Oleg From: Curt H. Sent: Monday, July 14, 2008 6:27 AM To: Michael L.; removed\_email\_address@domain.invalid Cc: IronRuby Team Subject: RE: [Ironruby-core] MutableString encoding issue MutableString can have one of three internal representations, depending on how it was last used. One of these is a byte array. This particular problem may be in the scanner or parser and not in the actual string class, as we don’t otherwise have a problem storing the character: > > > $s = “\204” > > > =\> “?” > > > $s[0] > > > =\> 63 > > > $s[0] = 132 > > > =\> 132 > > > $s > > > =\> “\204” From: Michael L. [[mailto:removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)] Sent: Monday, July 14, 2008 6:21 AM To: removed\_email\_address@domain.invalid Cc: IronRuby Team Subject: Re: [Ironruby-core] MutableString encoding issue This was a known issue a while back, it’s the reason the Zlib library didn’t work well with binary files. I’m fairly certain there was work being done on making String be backed by a byte array… and in fact I thought this was already done. On Mon, Jul 14, 2008 at 2:20 AM, Oleg T. \ wrote: Stumbled on this when testing yaml. > > > “\204” =\> “?” While irb(main):005:0\>“\204” =\> “\204” I believe Ruby string can hold arbitrary byte values, but as we are storing content as a string we are obviously losing all values that cannot be represented in default encoding. Tomas, what do you think? – Oleg ---
**Author:** ![Oleg\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/o/f19dbf/32.png) [@Oleg\_T](https://www.ruby-forum.com/u/Oleg_T)\ **Post date:** [July 14, 2008, 6:31pm UTC](https://www.ruby-forum.com/t/mutablestring-encoding-issue/142983/5 "2008-07-14T18:31:56Z")
I think it’s a bug :). Could you file it? If it’s something that’s blocking you I can look at it asap. Tomas From: Oleg T. Sent: Sunday, July 13, 2008 11:20 PM To: IronRuby Team Cc: removed\_email\_address@domain.invalid Subject: MutableString encoding issue Stumbled on this when testing yaml. > > > “\204” > > > =\> “?” While irb(main):005:0\>"\204" =\> “\204” I believe Ruby string can hold arbitrary byte values, but as we are storing content as a string we are obviously losing all values that cannot be represented in default encoding. Tomas, what do you think? ---
**Author:** ![Oleg\_T](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/o/f19dbf/32.png) [@Oleg\_T](https://www.ruby-forum.com/u/Oleg_T)\ **Post date:** [July 14, 2008, 6:38pm UTC](https://www.ruby-forum.com/t/mutablestring-encoding-issue/142983/6 "2008-07-14T18:38:22Z")
## Sure. Unfortunately this one blocks our yaml impl passing MRI’s test\_yaml.rb. Oleg From: Tomas M. Sent: Monday, July 14, 2008 9:14 AM To: Oleg T.; IronRuby Team Cc: removed\_email\_address@domain.invalid Subject: RE: MutableString encoding issue I think it’s a bug :). Could you file it? If it’s something that’s blocking you I can look at it asap. Tomas From: Oleg T. Sent: Sunday, July 13, 2008 11:20 PM To: IronRuby Team Cc: removed\_email\_address@domain.invalid Subject: MutableString encoding issue Stumbled on this when testing yaml. > > > “\204” > > > =\> “?” While irb(main):005:0\>"\204" =\> “\204” I believe Ruby string can hold arbitrary byte values, but as we are storing content as a string we are obviously losing all values that cannot be represented in default encoding. Tomas, what do you think?