# Bug in how Ruby 2.1,2.2 handles Encoding::ConverterNotFoundError

**URL:** <https://www.ruby-forum.com/t/bug-in-how-ruby-2-1-2-2-handles-encoding-converternotfounderror/234889>\
**Category:** Ruby\
**Created:** [January 9, 2015, 5:08am UTC](https://www.ruby-forum.com/t/bug-in-how-ruby-2-1-2-2-handles-encoding-converternotfounderror/234889 "2015-01-09T05:08:21Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![addis\_a](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/c4cdca/32.png) [@addis\_a](https://www.ruby-forum.com/u/addis_a)\
**Post date:** [January 9, 2015, 5:08am UTC](https://www.ruby-forum.com/t/bug-in-how-ruby-2-1-2-2-handles-encoding-converternotfounderror/234889/1 "2015-01-09T05:08:21Z")

</div>

Given:

“\x80”.force\_encoding(“ASCII-8BIT”).encode( Encoding::Emacs\_Mule)

raises an Encoding::ConverterNotFoundError on all rubies

On all rubies except for mri 2.1 and 2.2, encoding with the invalid  
option has no effect. But on 2.1 and 2.2, it replaces the “\x80” ==  
128.chr with the replace string (‘?’, 63.chr)

for each ruby version with UTF-8 encoding  
[“1.9.2”, “ruby”, #Encoding:UTF-8, #Encoding:UTF-8, 128]  
[“1.9.3”, “ruby”, #Encoding:UTF-8, #Encoding:UTF-8, 128]  
[“2.0.0”, “ruby”, #Encoding:UTF-8, #Encoding:UTF-8, 128]  
[“2.1.5”, “ruby”, #Encoding:UTF-8, #Encoding:UTF-8, 63]  
[“2.2.0”, “ruby”, #Encoding:UTF-8, #Encoding:UTF-8, 63]  
[“2.0.0”, “jruby”, #Encoding:UTF-8, #Encoding:UTF-8, 128]  
[“2.1.0”, “rbx”, #Encoding:UTF-8, #Encoding:UTF-8, 128]

[“1.9.3”, “jruby”, #Encoding:UTF-8, #Encoding:UTF-8, 128]  
[“2.0.0”, “jruby”, #Encoding:UTF-8, #Encoding:UTF-8, 128]

Here’s my test code

LANG=en\_US.UTF-8 LC\_ALL=en\_US.UTF-8   
rvm  
ruby-1.9.2-p330,ruby-1.9.3-p551,ruby-2.0.0-p598,ruby-2.1.5,ruby-2.2.0,jruby-1.7.18,rbx-2.2.2  
do   
ruby -e ‘p [RUBY\_VERSION, RUBY\_ENGINE, Encoding.default\_external,  
**ENCODING** ] +  
“\x80”.force\_encoding(“ASCII-8BIT”).force\_encoding(“Emacs-Mule”).encode(:invalid  
=\> :replace).bytes.to\_a’

for version in 1.9 2.0; do   
export JRUBY\_OPTS=“-Xcompat.version=${version}” ;   
LANG=en\_US.UTF-8 LC\_ALL=en\_US.UTF-8   
rvm jruby-1.7.18 do   
ruby -e ‘p [RUBY\_VERSION, RUBY\_ENGINE, Encoding.default\_external,  
**ENCODING** ] +  
“\x80”.force\_encoding(“ASCII-8BIT”).force\_encoding(“Emacs-Mule”).encode(:invalid  
=\> :replace).bytes.to\_a’ ; done

For the particularly curious, this is relevant to a PR I have for  
rspec-support

> <https://github.com/rspec/rspec-support/pull/151#discussion_r22573031>
>
> Why does it do nothing on 1.9 and 2.0? I didn't think 2.1 had new encoding feat…ures...shouldn't it act the same on those rubies?

---

<div class="post-metadata">

**Author:** ![beninem](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/aca169/32.png) [@beninem](https://www.ruby-forum.com/u/beninem)\
**Post date:** [January 9, 2015, 5:48am UTC](https://www.ruby-forum.com/t/bug-in-how-ruby-2-1-2-2-handles-encoding-converternotfounderror/234889/2 "2015-01-09T05:48:20Z")

</div>

String#encode converts to Encoding.default\_internal by default, not  
default\_exteranl.  
And default\_internal is nil by default, it doesn’t change the encoding.  
In this case, it had done nothing, except for just making a copy, till  
2.0.

But as many people made same mistake like yours, expecting invalid  
chars were removed/replaced, the behavior has changed since 2.1 to  
replace such chars if  
:replace is given.

---

<div class="post-metadata">

**Author:** ![beninem](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/aca169/32.png) [@beninem](https://www.ruby-forum.com/u/beninem)\
**Post date:** [January 9, 2015, 6:10am UTC](https://www.ruby-forum.com/t/bug-in-how-ruby-2-1-2-2-handles-encoding-converternotfounderror/234889/3 "2015-01-09T06:10:00Z")

</div>

Nobuyoshi N. wrote in post #1166362:

Thank you! That is very helpful!

> But as many people made same mistake like yours, expecting invalid  
> chars were removed/replaced,

Actually, I’ve just been reading code and testing and observing what  
Ruby does.

I was asked why the behavior changed and I didn’t know, since it is  
reasonable for Ruby to ignore the :invalid directive when no converter  
is found,

Thank you so much!

-Benjamin

p.s. Now I just need to figure out how all the Rubies reconcile the  
:undef, :invalid, and :replace with :fallback

I don’t really know C, but it appears to me from the code that :invalid  
and :undef are called before :fallback (ecflags?)

> <https://github.com/ruby/ruby/blob/34fbf57aaafee9390a0f7427eb90efac099e33ec/transcode.c#L2677-L2733>

> <https://github.com/ruby/ruby/blob/34fbf57aaafee9390a0f7427eb90efac099e33ec/transcode.c#L2838-L2858>

> <https://github.com/ruby/ruby/blob/34fbf57aaafee9390a0f7427eb90efac099e33ec/transcode.c#L2282-L2290>
