# Strange Encoding Behavior

**URL:** <https://www.ruby-forum.com/t/strange-encoding-behavior/181729>\
**Category:** Ruby\
**Created:** [January 17, 2010, 4:55am UTC](https://www.ruby-forum.com/t/strange-encoding-behavior/181729 "2010-01-17T04:55:51Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![night\_stalker](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/f0a364/32.png) [@night\_stalker](https://www.ruby-forum.com/u/night_stalker)\
**Post date:** [January 17, 2010, 4:55am UTC](https://www.ruby-forum.com/t/strange-encoding-behavior/181729/1 "2010-01-17T04:55:51Z")

</div>

The encoding of **FILE** is always the same as Encoding.default\_external  
even if there is a magic column. Sometimes it is necessary to convert  
the string into another encoding. Here is some code to demonstrate the  
issue:

#coding: utf-8  
# put the script in a not-pure-ascii path to see the difference  
path = File.expand\_path File.dirname **FILE**

```
puts RUBY_VERSION + ' ' + RUBY_PLATFORM
  #=> "1.9.1 i386-mingw32" is my ruby version
puts path.encoding
  #=> "GB2312" on my OS

# usually this "string.encode to, from" works,
# but HERE the new string's content bytes seems unchanged
puts \
  path.encode 'utf-8', Encoding.default_external

path.force_encoding Encoding.default_external
puts path.encode 'utf-8'
  # changed at last
```

---

<div class="post-metadata">

**Author:** ![night\_stalker](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/f0a364/32.png) [@night\_stalker](https://www.ruby-forum.com/u/night_stalker)\
**Post date:** [January 17, 2010, 5:25am UTC](https://www.ruby-forum.com/t/strange-encoding-behavior/181729/2 "2010-01-17T05:25:53Z")

</div>

On Jan 17, 12:55 am, Lui K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> ```
> #=> "1.9.1 i386-mingw32" is my ruby version
> # changed at last
> 
> ```

At 1.9.1, and some part of 1.9.2 still display certain issues with  
path/folders with non-ascii characters:

[http://redmine.ruby-lang.org/issues/show/1685](http://redmine.ruby-lang.org/issues/show/1685)

---

<div class="post-metadata">

**Author:** ![night\_stalker](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/f0a364/32.png) [@night\_stalker](https://www.ruby-forum.com/u/night_stalker)\
**Post date:** [January 17, 2010, 7:13am UTC](https://www.ruby-forum.com/t/strange-encoding-behavior/181729/3 "2010-01-17T07:13:24Z")

</div>

I think #1685 is a little bit different.  
Maybe the following code is a bit clearer:

```
# coding: ascii-8bit
puts Encoding.default_external #=> GBK

def enc s
  s.encode 'utf-8', Encoding.default_external
end
p1 = File.expand_path File.dirname __FILE__
p2 = p1.dup
p2.force_encoding p2.encoding # strange, but makes it different

puts p1 == p2 #=> true
puts p1.encoding == p2.encoding #=> true
puts enc(p1) == enc(p2) #=> sometimes false ???

```

Run in console:

```
D:\其他>ruby t.rb
GBK
true
true
false

```

put it in another folder:

```
D:\other>ruby t.rb
GBK
true
true
true

```

Luis L. wrote:

> On Jan 17, 12:55�am, Lui K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> 
> > � � � #=\> “1.9.1 i386-mingw32” is my ruby version  
> > � � � # changed at last  
> > At 1.9.1, and some part of 1.9.2 still display certain issues with  
> > path/folders with non-ascii characters:
> 
> [http://redmine.ruby-lang.org/issues/show/1685](http://redmine.ruby-lang.org/issues/show/1685)

---

<div class="post-metadata">

**Author:** ![night\_stalker](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/f0a364/32.png) [@night\_stalker](https://www.ruby-forum.com/u/night_stalker)\
**Post date:** [January 17, 2010, 8:30pm UTC](https://www.ruby-forum.com/t/strange-encoding-behavior/181729/4 "2010-01-17T20:30:32Z")

</div>

On 01/17/2010 04:55 AM, Lui K. wrote:

> ```
> #=> "1.9.1 i386-mingw32" is my ruby version
> # changed at last
> 
> ```

I believe the point you are missing is that String#encode does not  
change the String but it returns a new String with the desired encoding.  
If you want inplace modification you need to use String#encode! which  
does just that.

irb(main):006:0\> s=“foo”  
=\> “foo”  
irb(main):007:0\> s.encoding  
=\> #Encoding:UTF-8  
irb(main):008:0\> x = s.encode “ASCII”  
=\> “foo”  
irb(main):009:0\> s.encoding  
=\> #Encoding:UTF-8  
irb(main):010:0\> x.encoding  
=\> #Encoding:US-ASCII  
irb(main):011:0\>

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![night\_stalker](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/f0a364/32.png) [@night\_stalker](https://www.ruby-forum.com/u/night_stalker)\
**Post date:** [January 18, 2010, 9:38am UTC](https://www.ruby-forum.com/t/strange-encoding-behavior/181729/5 "2010-01-18T09:38:22Z")

</div>

2010/1/18 Lui K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> To describe it in a “encode!” version:  
> For some path p1 and p2,  
> when p1 == p2 and p1.encoding == p2.encoding,  
> p1.encode!(‘utf-8’)  
> p2.encode!(‘utf-8’)  
> p1 == p2 is still not always true

Apparently I misread your posting, sorry. Is UTF-8 capable of  
representing those Japanese or Chinese characters? I believe I  
remember Matz saying that UTF-8 is insufficient to properly represent  
Japanese characters. If this is the case then I guess all bets are  
off and you get undefined behavior. Although it might be desirable to  
get the same garbage it may not be worthwhile to ensure this purely  
for efficiency reasons.

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![night\_stalker](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/f0a364/32.png) [@night\_stalker](https://www.ruby-forum.com/u/night_stalker)\
**Post date:** [January 18, 2010, 5:23am UTC](https://www.ruby-forum.com/t/strange-encoding-behavior/181729/6 "2010-01-18T05:23:12Z")

</div>

I know String#encode doesn’t change the original string, but the result  
is encoded.

To understand the problem, you should try in a gbk/shift-jis environment  
with some Chinese or Japanese path.

The point is:  
For some path p1 and p2,  
when p1 == p2 and p1.encoding == p2.encoding,  
p1.encode(‘utf-8’) == p2.encode(‘utf-8’) is not always true.

To describe it in a “encode!” version:  
For some path p1 and p2,  
when p1 == p2 and p1.encoding == p2.encoding,  
p1.encode!(‘utf-8’)  
p2.encode!(‘utf-8’)  
p1 == p2 is still not always true

Robert K. wrote:

> On 01/17/2010 04:55 AM, Lui K. wrote:
> 
> > ```
> > #=> "1.9.1 i386-mingw32" is my ruby version
> > # changed at last
> > 
> > ```
> 
> I believe the point you are missing is that String#encode does not  
> change the String but it returns a new String with the desired encoding.  
> If you want inplace modification you need to use String#encode! which  
> does just that.
> 
> irb(main):006:0\> s=“foo”  
> =\> “foo”  
> irb(main):007:0\> s.encoding  
> =\> #Encoding:UTF-8  
> irb(main):008:0\> x = s.encode “ASCII”  
> =\> “foo”  
> irb(main):009:0\> s.encoding  
> =\> #Encoding:UTF-8  
> irb(main):010:0\> x.encoding  
> =\> #Encoding:US-ASCII  
> irb(main):011:0\>
> 
> Kind regards
> 
> robert