Which works on both environments. I fixed it by reading about encoding
and trial and error, so I’m left with a working solution, but not
knowing why it works on the Mac but not in Linux. Could someone please
explain?
At Thu, 23 Apr 2009 10:10:45 +0900,
Daly wrote in [ruby-talk:334732]:
I figured it out and fixed it by doing:
Iconv.iconv(“UTF8”, “UCS-2BE”, row)
Which works on both environments. I fixed it by reading about encoding
and trial and error, so I’m left with a working solution, but not
knowing why it works on the Mac but not in Linux. Could someone please
explain?
It seems a bug of Ubuntu iconv. According to Unicode
Consortium, UCS-2 is defaulted to big endian if no BOM exists.
Explicit endianness anyway. Ruby 1.9 does not provide UCS
names without endians.
I fixed it by reading about encoding
and trial and error, so I’m left with a working solution, but not
knowing why it works on the Mac but not in Linux. Could someone please
explain?
I’ve been trying to covering character encodings with a heavy Ruby
slant on my blog for just this reason:
My coverage is about 98% complete now, in case you want to browse a bit.
Hi James,
I was wondering if you wanted to add a note on how to deal with
potentially unknown character encodings. This was one of the more
annoying problems that I hit with trying to use Iconv directly for ruby
1.8. In the past, I’d used the Mozilla character detection library (in
Java) when doing processing of XML in order to ensure that the file
encoding matched the declaration in the <?xml > header.
Fortunately, I recently found the rchardet gem which is the port of this
library to Ruby, and it’s helped me deal with giving more appropriate
encoding information to Iconv.
Usage goes something like this:
91 cd = CharDet.detect(text)
92 encoding = cd[‘encoding’]
93 puts “Reading detected encoding ‘#{encoding}’ text with
confidence: %. 2f%%” % [cd[‘confidence’] * 100]
94 iconv = Iconv.new(“UTF-8”, encoding)
95 puts “Conversion to UTF-8 successful.”
This time, I needed this sort of thing when trying to ensure I could
load arbitrary text files from unknown sources into GTK+ widgets.
I’ve actually rarely been in the case where I knew the encoding of the
input I was trying to deal with if it wasn’t the same as the system
default, but maybe that’s just me…
The referenced blog post looks really good. Thanks for your efforts.
