Strange Encoding Behavior

The encoding of FILE is always the same as Encoding.default_external
even if there is a magic column. Sometimes it is necessary to convert
the string into another encoding. Here is some code to demonstrate the
issue:

#coding: utf-8
# put the script in a not-pure-ascii path to see the difference
path = File.expand_path File.dirname FILE

puts RUBY_VERSION + ' ' + RUBY_PLATFORM
  #=> "1.9.1 i386-mingw32" is my ruby version
puts path.encoding
  #=> "GB2312" on my OS

# usually this "string.encode to, from" works,
# but HERE the new string's content bytes seems unchanged
puts \
  path.encode 'utf-8', Encoding.default_external

path.force_encoding Encoding.default_external
puts path.encode 'utf-8'
  # changed at last

On Jan 17, 12:55 am, Lui K. [email protected] wrote:

  #=> "1.9.1 i386-mingw32" is my ruby version
  # changed at last

At 1.9.1, and some part of 1.9.2 still display certain issues with
path/folders with non-ascii characters:

http://redmine.ruby-lang.org/issues/show/1685

I think #1685 is a little bit different.
Maybe the following code is a bit clearer:

# coding: ascii-8bit
puts Encoding.default_external  #=> GBK

def enc s
  s.encode 'utf-8', Encoding.default_external
end
p1 = File.expand_path File.dirname __FILE__
p2 = p1.dup
p2.force_encoding p2.encoding   # strange, but makes it different

puts p1 == p2                   #=> true
puts p1.encoding == p2.encoding #=> true
puts enc(p1) == enc(p2)         #=> sometimes false ???

Run in console:

D:\其他>ruby t.rb
GBK
true
true
false

put it in another folder:

D:\other>ruby t.rb
GBK
true
true
true

Luis L. wrote:

On Jan 17, 12:55�am, Lui K. [email protected] wrote:

� � � #=> “1.9.1 i386-mingw32” is my ruby version
� � � # changed at last
At 1.9.1, and some part of 1.9.2 still display certain issues with
path/folders with non-ascii characters:

http://redmine.ruby-lang.org/issues/show/1685

On 01/17/2010 04:55 AM, Lui K. wrote:

  #=> "1.9.1 i386-mingw32" is my ruby version
  # changed at last

I believe the point you are missing is that String#encode does not
change the String but it returns a new String with the desired encoding.
If you want inplace modification you need to use String#encode! which
does just that.

irb(main):006:0> s=“foo”
=> “foo”
irb(main):007:0> s.encoding
=> #Encoding:UTF-8
irb(main):008:0> x = s.encode “ASCII”
=> “foo”
irb(main):009:0> s.encoding
=> #Encoding:UTF-8
irb(main):010:0> x.encoding
=> #Encoding:US-ASCII
irb(main):011:0>

Kind regards

robert

2010/1/18 Lui K. [email protected]:

To describe it in a “encode!” version:
For some path p1 and p2,
when p1 == p2 and p1.encoding == p2.encoding,
p1.encode!(‘utf-8’)
p2.encode!(‘utf-8’)
p1 == p2 is still not always true

Apparently I misread your posting, sorry. Is UTF-8 capable of
representing those Japanese or Chinese characters? I believe I
remember Matz saying that UTF-8 is insufficient to properly represent
Japanese characters. If this is the case then I guess all bets are
off and you get undefined behavior. Although it might be desirable to
get the same garbage it may not be worthwhile to ensure this purely
for efficiency reasons.

Kind regards

robert

I know String#encode doesn’t change the original string, but the result
is encoded.

To understand the problem, you should try in a gbk/shift-jis environment
with some Chinese or Japanese path.

The point is:
For some path p1 and p2,
when p1 == p2 and p1.encoding == p2.encoding,
p1.encode(‘utf-8’) == p2.encode(‘utf-8’) is not always true.

To describe it in a “encode!” version:
For some path p1 and p2,
when p1 == p2 and p1.encoding == p2.encoding,
p1.encode!(‘utf-8’)
p2.encode!(‘utf-8’)
p1 == p2 is still not always true

Robert K. wrote:

On 01/17/2010 04:55 AM, Lui K. wrote:

  #=> "1.9.1 i386-mingw32" is my ruby version
  # changed at last

I believe the point you are missing is that String#encode does not
change the String but it returns a new String with the desired encoding.
If you want inplace modification you need to use String#encode! which
does just that.

irb(main):006:0> s=“foo”
=> “foo”
irb(main):007:0> s.encoding
=> #Encoding:UTF-8
irb(main):008:0> x = s.encode “ASCII”
=> “foo”
irb(main):009:0> s.encoding
=> #Encoding:UTF-8
irb(main):010:0> x.encoding
=> #Encoding:US-ASCII
irb(main):011:0>

Kind regards

robert

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs