# Reliable character encodings conversion

**URL:** <https://www.ruby-forum.com/t/reliable-character-encodings-conversion/149845>\
**Category:** Ruby\
**Created:** [September 30, 2008, 2:32pm UTC](https://www.ruby-forum.com/t/reliable-character-encodings-conversion/149845 "2008-09-30T14:32:20Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hubert\_SSSSSSSSSpic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/73ab20/32.png) [@Hubert\_SSSSSSSSSpic](https://www.ruby-forum.com/u/Hubert_SSSSSSSSSpic)\
**Post date:** [September 30, 2008, 2:32pm UTC](https://www.ruby-forum.com/t/reliable-character-encodings-conversion/149845/1 "2008-09-30T14:32:20Z")

</div>

Hi,

I am looking for reliable and error-resistant way to convert character  
encodings to UTF8. Input encodings vary, and I have quite good input  
encodings detection in place.

I am using Iconv library wrapper to convert texts to UTF8, but it’s  
throwing “Iconv::IllegalSequence” exception. The problem is that input  
texts are user-generated and have sometimes mixed characters  
encodings.

Does anyone have any experience with these kind of situations, or can  
suggest alternative libraries?

Thanks,  
Hubert

---

<div class="post-metadata">

**Author:** ![Hubert\_SSSSSSSSSpic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/73ab20/32.png) [@Hubert\_SSSSSSSSSpic](https://www.ruby-forum.com/u/Hubert_SSSSSSSSSpic)\
**Post date:** [September 30, 2008, 3:05pm UTC](https://www.ruby-forum.com/t/reliable-character-encodings-conversion/149845/2 "2008-09-30T15:05:52Z")

</div>

On Sep 30, 2008, at 7:30 AM, Hubert Łępicki wrote:

> I am using Iconv library wrapper to convert texts to UTF8, but it’s  
> throwing “Iconv::IllegalSequence” exception.

You can add a //TRANSLIT to the end of the “to” encoding to have Iconv  
attempt to convert characters to reasonable equivalents in that  
encoding. This is usually more helpful when your input is all one  
encoding and just has some characters that won’t translate well (like  
a UTF-8 … going to ISO-8859-1).

Your case of mixed encodings is probably best handled with //IGNORE  
instead, which asks Iconv to skip over any characters that cannot be  
converted. You will loose some data with this, but it will convert  
what it can.

You can also use //TRANSLIT//IGNORE to convert what can be converted  
and skip the rest.

Hope that helps.

James Edward G. II

---

<div class="post-metadata">

**Author:** ![Hubert\_SSSSSSSSSpic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/73ab20/32.png) [@Hubert\_SSSSSSSSSpic](https://www.ruby-forum.com/u/Hubert_SSSSSSSSSpic)\
**Post date:** [September 30, 2008, 3:22pm UTC](https://www.ruby-forum.com/t/reliable-character-encodings-conversion/149845/3 "2008-09-30T15:22:18Z")

</div>

2008/9/30 James G. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> Your case of mixed encodings is probably best handled with //IGNORE instead,  
> which asks Iconv to skip over any characters that cannot be converted. You  
> will loose some data with this, but it will convert what it can.
> 
> You can also use //TRANSLIT//IGNORE to convert what can be converted and  
> skip the rest.

Thanks, //IGNORE//TRANSLIT seems to help a bit - but it’s not perfect.  
I am loosing characters like British pound that were placed in  
us-ascii encoding for example. Is there some smart library out there  
that can help with common problems like this one?

I have noticed that there is ICU ([http://www.icu-project.org/](http://www.icu-project.org/)) library  
for C++ that I could use if it’s any smarter - anyone had any  
experience with it?

Best,  
H.

---

<div class="post-metadata">

**Author:** ![Hubert\_SSSSSSSSSpic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/73ab20/32.png) [@Hubert\_SSSSSSSSSpic](https://www.ruby-forum.com/u/Hubert_SSSSSSSSSpic)\
**Post date:** [September 30, 2008, 4:00pm UTC](https://www.ruby-forum.com/t/reliable-character-encodings-conversion/149845/4 "2008-09-30T16:00:37Z")

</div>

On Sep 30, 2008, at 8:20 AM, Hubert Łępicki wrote:

> > This is usually more helpful when your input is all one encoding
> > 
> > You can also use //TRANSLIT//IGNORE to convert what can be  
> > converted and  
> > skip the rest.
> 
> Thanks, //IGNORE//TRANSLIT seems to help a bit - but it’s not perfect.

You listed those backwards. Is that really what you tried? Does  
reversing them make any difference?

James Edward G. II

---

<div class="post-metadata">

**Author:** ![Hubert\_SSSSSSSSSpic](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/73ab20/32.png) [@Hubert\_SSSSSSSSSpic](https://www.ruby-forum.com/u/Hubert_SSSSSSSSSpic)\
**Post date:** [September 30, 2008, 6:51pm UTC](https://www.ruby-forum.com/t/reliable-character-encodings-conversion/149845/5 "2008-09-30T18:51:50Z")

</div>

you can use RChardet library,

her’es what i use:

require ‘rchardet’

class String  
def encoding  
@encoding ||= guess\_encoding  
end

def encoding=(new)  
@encoding = new  
end

def convert\_to(new)  
self.replace(Iconv.iconv(new, encoding, self)[0])  
@encoding = new  
end

def guess\_encoding  
@encoding = CharDet.guess(self)[“encoding”]  
end

# this enables “foo”.convert :us-ascii =\> :utf8

def convert(hash)  
from = hash.keys[0]  
to = hash[from]  
self.replace(Iconv.iconv(to, from, self)[0])  
end  
end

it handles translating preatty well 🙂