[ruby-trunk - Bug #8445][Open] IO.open and IO#set_enconding does not support :fallback option

Issue #8445 has been reported by pjmtdw (Haruhiro Yoshimoto).


Bug #8445: IO.open and IO#set_enconding does not support :fallback
option

Author: pjmtdw (Haruhiro Yoshimoto)
Status: Open
Priority: Normal
Assignee:
Category:
Target version:
ruby -v: trunk(ruby 2.1.0dev)
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN

RubyDoc says that IO.open and IO#set_encoding supports optional argument
defined in String#encode.

In fact, :invalid, :undef and :replace works as expected.

However, :fallback option does not work neither for IO.open and
IO#set_encoding.
Following is the example code which does not work.
f(x) is never called even if hoge.txt contains non convertible
character.

File.open(“./hoge.txt”,“r:Shift_JIS:utf-8”, :fallback =>
lambda{|x|f(x)}){|f|

}
File.open(“./hoge.txt”){|f|
f.set_encoding(“Shift_JIS”,“utf-8”,:fallback => lambda{|x|f(x)})

}

I Think this is because fill_cbuf() in io.c calls rb_econv_convert()
from transcode.c directly.
On the other hand, fallback_func is called in transcode_loop(), which is
called by str_encode().

Since transcode_loop() also calls rb_econv_convert(), I wrote a small
patch which moves some codes from
transcode_loop() to rb_econv_convert() to fix the problem.

The attached file is the patch. Hope this helps.

Issue #8445 has been updated by naruse (Yui NARUSE).

Category set to M17N
Status changed from Open to Assigned
Assignee set to akr (Akira T.)
Target version set to current: 2.1.0

Is this intentional or just unexpected inconsistency?

Bug #8445: IO.open and IO#set_enconding does not support :fallback
option

Author: pjmtdw (Haruhiro Yoshimoto)
Status: Assigned
Priority: Normal
Assignee: akr (Akira T.)
Category: M17N
Target version: current: 2.1.0
ruby -v: trunk(ruby 2.1.0dev)
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN

RubyDoc says that IO.open and IO#set_encoding supports optional argument
defined in String#encode.

In fact, :invalid, :undef and :replace works as expected.

However, :fallback option does not work neither for IO.open and
IO#set_encoding.
Following is the example code which does not work.
f(x) is never called even if hoge.txt contains non convertible
character.

File.open(“./hoge.txt”,“r:Shift_JIS:utf-8”, :fallback =>
lambda{|x|f(x)}){|f|

}
File.open(“./hoge.txt”){|f|
f.set_encoding(“Shift_JIS”,“utf-8”,:fallback => lambda{|x|f(x)})

}

I Think this is because fill_cbuf() in io.c calls rb_econv_convert()
from transcode.c directly.
On the other hand, fallback_func is called in transcode_loop(), which is
called by str_encode().

Since transcode_loop() also calls rb_econv_convert(), I wrote a small
patch which moves some codes from
transcode_loop() to rb_econv_convert() to fix the problem.

The attached file is the patch. Hope this helps.

Issue #8445 has been updated by Hiroshi SHIBATA.

Target version changed from 2.1.0 to current: 2.2.0


Bug #8445: IO.open and IO#set_enconding does not support :fallback
option

  • Author: Haruhiro Yoshimoto
  • Status: Assigned
  • Priority: Normal
  • Assignee: Akira T.
  • Category: M17N
  • Target version: current: 2.2.0
  • ruby -v: trunk(ruby 2.1.0dev)
  • Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN

RubyDoc says that IO.open and IO#set_encoding supports optional argument
defined in String#encode.

In fact, :invalid, :undef and :replace works as expected.

However, :fallback option does not work neither for IO.open and
IO#set_encoding.
Following is the example code which does not work.
f(x) is never called even if hoge.txt contains non convertible
character.

File.open(“./hoge.txt”,“r:Shift_JIS:utf-8”, :fallback =>
lambda{|x|f(x)}){|f|

}
File.open(“./hoge.txt”){|f|
f.set_encoding(“Shift_JIS”,“utf-8”,:fallback => lambda{|x|f(x)})

}

I Think this is because fill_cbuf() in io.c calls rb_econv_convert()
from transcode.c directly.
On the other hand, fallback_func is called in transcode_loop(), which is
called by str_encode().

Since transcode_loop() also calls rb_econv_convert(), I wrote a small
patch which moves some codes from
transcode_loop() to rb_econv_convert() to fix the problem.

The attached file is the patch. Hope this helps.

—Files--------------------------------
support-fallback-for-io.patch (3.9 KB)