do |file|
file.readlines.map do |i|
begin
i = i.gsub(/[\u0022]/,’’)
rescue
p $!
p i
p i.encode(‘UTF-16’, :invalid => :replace, :replace =>
‘’).encode(‘utf-8’)
end
end
end
Now problem is the line i.encode('UTF-16', :invalid => :replace, :replace => '').encode('utf-8') handling it properly, but for invalid
byte it is replacing it with "". As you can see, I got Cte d'Ivoire\t\tCte.. where the character Ù is missing. But can this be
placed, with some logic in the line ...:replace => ''. I am looking
for instead of "", the dynamic charater for which error happned, with
some processing replacement character should be that Ù or any …
I have a (humble) guess about your problem…
I think you are trying to open an ISO_8859_1 encoded file as if it was
an UTF_8.
If so, you don’t need to recreate the logic of the character/encoding
translation the way you are trying to do. Ruby will kindly do that for
you.
You just have to open the file telling Ruby that it is an ISO_8859_1
file.
Try…
File.open(’/home/kirti/workspace/Ruby/project_free/data/2013_diamonds.txt’,
external_encoding: Encoding::ISO_8859_1) do |file|
I think no error will be raised.
This worked for me, at least with the two lines that you gave as
example.
Just tell me if it worked with the whole file.
You can read more at the Encoding, IO and File ruby documentation (see
internal and external encoding).
Try…
File.open(’/home/kirti/workspace/Ruby/project_free/data/2013_diamonds.txt’,
external_encoding: Encoding::ISO_8859_1) do |file|
Very good suggestion indeed.
Just one topic in Ruby, always troubled me to understand the rationality
about this encoding. When to think of internal_encoding and external_encoding. Why not only encoding? Sometimes in this
situation I also used force_encoding… This all I just used an an
trial and error. No I didn’t really aware of what I was doing. My goal
was to fix the error. First try encoding, then try force_encoding…
Feel free to ask if it’s not clear yet.
The encoding problem is complex (but the solution we have in Ruby is
simple IMHO).
I’m a native portuguese speaker and have to rely on good encoding
support.
One line from the file gave me so much trouble, once. I fixed it as
below
Can you give me the line?
I can try to help you.
Most of the problem come from the following: One open a file telling
Ruby
(implicitly or explicitly) the encoding, for example utf-8. But, the
byte
representation inside it is another encoding.
Em 01/02/2014 11:16, “Arup R.” [email protected] escreveu:
For you to try to understand the rationality of it just relax and
think about why you (not me ) were trying to “encoding” or
“force_enconding” a string coming from a file that has a different
encoding than the internal one in your program.
Perhaps you will notice that you are receiving data (external data) in
an encoding different than that used internally.
Ruby does exactly what you were trying to accomplish, but in a more
elegant/fashioned way ;-).
When you set the external encoding of a file, Ruby tries to translate
all data coming from the file from the external encoding to the
internal one.
And when you try to write to the file it does the reverse.
So that you can preserve the original encoding of the file and don’t
have to worry about encoding compatibility inside your program.
Go for at
Feel free to ask if it’s not clear yet.
The encoding problem is complex (but the solution we have in Ruby is
simple IMHO).
I’m a native portuguese speaker and have to rely on good encoding
support.
As Ruby has its roots on japanese programmers, I think they’re really
concerned on good encoding support with a rich set of features to deal
with it.
Kind regards,
Abinoam Jr.
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
) were trying to “encoding” or