# Unicode string conversion

**URL:** <https://www.ruby-forum.com/t/unicode-string-conversion/103393>\
**Category:** Ruby\
**Created:** [May 5, 2007, 2:01am UTC](https://www.ruby-forum.com/t/unicode-string-conversion/103393 "2007-05-05T02:01:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![airmind](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ecd19e/32.png) [@airmind](https://www.ruby-forum.com/u/airmind)\
**Post date:** [May 5, 2007, 2:01am UTC](https://www.ruby-forum.com/t/unicode-string-conversion/103393/1 "2007-05-05T02:01:06Z")

</div>

I’m reading a binary file in my program. It contains strings in the  
Windows Unicode format, which it says is stored as little-endian in the  
spefication. I’m loading it and trying to convert using Iconv, but I’m  
getting a invalid character exception, on any string. Now I’m just  
stripping the \000 character from it and it works, but I know it’s not  
an ideal solution and it only works in some cases.  
So, how can I get the string in a format Ruby can understand? By the  
way, I’ll load these string in GTK (with Ruby bindings), anyone knows if  
it can show Unicode strings?

---

<div class="post-metadata">

**Author:** ![airmind](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ecd19e/32.png) [@airmind](https://www.ruby-forum.com/u/airmind)\
**Post date:** [May 5, 2007, 6:51am UTC](https://www.ruby-forum.com/t/unicode-string-conversion/103393/2 "2007-05-05T06:51:19Z")

</div>

On May 5, 2007, at 9:01 AM, Alexandre R. wrote:

> it can show Unicode strings?
> 
> –  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

Stripping the BOM? (byte order mark)  
Should be fine. Unicode works just as well w/ no BOM, actually better  
with no BOM.  
The first thing you should check for though is the presence of the  
BOM and read the BOM.

---

<div class="post-metadata">

**Author:** ![airmind](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ecd19e/32.png) [@airmind](https://www.ruby-forum.com/u/airmind)\
**Post date:** [May 6, 2007, 4:05pm UTC](https://www.ruby-forum.com/t/unicode-string-conversion/103393/3 "2007-05-06T16:05:27Z")

</div>

John J. wrote:

> On May 5, 2007, at 9:01 AM, Alexandre R. wrote:
> 
> > it can show Unicode strings?
> > 
> > –  
> > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).
> 
> Stripping the BOM? (byte order mark)  
> Should be fine. Unicode works just as well w/ no BOM, actually better  
> with no BOM.  
> The first thing you should check for though is the presence of the  
> BOM and read the BOM.

There is no BOM. The specifications clearly states it “uses UTF-16,  
little endian, and the Byte-Order Marker (BOM) character is not present”

What I’m confused is to why Iconv coudlnt convert it. Does Iconv expects  
for the BOM, even when I specify UTF16LE, which would make it explicit  
the byte order?

---

<div class="post-metadata">

**Author:** ![airmind](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/ecd19e/32.png) [@airmind](https://www.ruby-forum.com/u/airmind)\
**Post date:** [May 9, 2007, 1:53am UTC](https://www.ruby-forum.com/t/unicode-string-conversion/103393/4 "2007-05-09T01:53:15Z")

</div>

Hi,

At Sun, 6 May 2007 23:05:37 +0900,  
Alexandre R. wrote in [ruby-talk:250503]:

> What I’m confused is to why Iconv coudlnt convert it. Does Iconv expects  
> for the BOM, even when I specify UTF16LE, which would make it explicit  
> the byte order?

BOM is a “ZERO WIDTH NON-BREAKING SPACE” at the beginning of  
a text. Almost iconv(3) should be possible to deal with it.  
Can’t you show minimal data to reproduce the error?