> Where the perl script above
> can read text and binary files,

While true, I don't think it does so correctly.  split() becomes unicode
aware when a string has been encoded with utf-8, and you've told perl to
automatically encode every line read in with utf-8.  That means split()
is splitting on characters--not bytes.  That means if your binary data
happens to have the sequence:

\x{E2}
\x{82}
\x{AC}

then that sequence will not be split into three bytes because
that sequence is the utf8 encoding for a Euro Sign, therefore
all three bytes will be split off as one character.  There are lots
of combinations of random bytes that can make up a utf-8 character.
Proof:

    use strict;
    use warnings;
    use 5.016;

    my $fname =  'data.txt';

    #Write 3 bytes to file: ---------

    open my $OUTFILE, '>', $fname
        or die "Couldn't open $fname: $!\n";

    #my $str = "\N{EURO SIGN}";   #UTF-8 encoding: E2 82 AC
    print {$OUTFILE} "\x{E2}\x{82}\x{AC}";

    close $OUTFILE;

    #---------------------------


    use open IO => ':utf8';   #Now read file as utf8

    open my $INFILE, '<', $fname
        or die "Couldn't open $fname: $!\n";

    while ( my $line = <$INFILE> ) {
      my @chars = split //, $line;
      say scalar @chars;   #=> 1
    }

    close $INFILE;


As for ruby, Strings have the following methods:

each_byte()
each_char()

And you can specify the encoding of the file that you are reading when
you create the filehandle:


    #encoding: UTF-8

    #The previous comment line is so the string on line 9 will be
    #encoded with UTF-8.  The encoding in the comment applies
    #only to Strings in the source file.

    fname = 'data.txt'

    File.open(fname, 'w') do |f|
      f.write("\u20AC")  #LINE 9, Euro Sign
    end

    File.open(fname, 'r', external_encoding: 'UTF-8') do |f|
      f.each_line do |line|

        line.each_char do |char|   #UTF-8 chars
          puts char
        end

        line.each_byte do |byte|
          printf "%x \n", byte
        end

      end
    end

    --output:--
    €  (I see a Euro Sign)
    e2
    82
    ac

On Mon, Jun 30, 2014 at 7:14 PM, Eric Peterson <epeterson@rhapsody.com>
wrote:

> I though of doing "rb:utf-8:-" on the File.open, but that didn't work
> either.
>
> Any ideas?

You were pretty close:

$ ruby -e 'File.open("xx", "rb") {|io| p io.external_encoding}'
#<Encoding:ASCII-8BIT>

ASCII-8BIT is binary:

$ ruby -e 'p Encoding::BINARY'
#<Encoding:ASCII-8BIT>

For comparison

$ ruby -e 'File.open("xx", "r") {|io| p io.external_encoding}'
#<Encoding:UTF-8>

Does that help?

Kind regards

robert

Robert Klemme wrote in post #1151291:

> For comparison
>
> $ ruby -e 'File.open("xx", "r") {|io| p io.external_encoding}'
> #<Encoding:UTF-8>
>
> Does that help?
>

Well, he certainly can't rely on that.

===
The default external Encoding is pulled from your environment, much like
the source Encoding is for code given on the command-line
===
(James Edward Gray II)

And has been discussed recently here:

https://www.ruby-forum.com/topic/4980931#new

...the default external_encoding that ruby slaps on Strings read in from
IO objects does not mean those String are actually encoded in that
encoding.