# Text encodings

**URL:** <https://www.ruby-forum.com/t/text-encodings/97755>\
**Category:** Ruby\
**Created:** [July 10, 2006, 12:33pm UTC](https://www.ruby-forum.com/t/text-encodings/97755 "2006-07-10T12:33:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![xTRiM](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/ad7895/32.png) [@xTRiM](https://www.ruby-forum.com/u/xTRiM)\
**Post date:** [July 10, 2006, 12:33pm UTC](https://www.ruby-forum.com/t/text-encodings/97755/1 "2006-07-10T12:33:21Z")

</div>

Hello,

is there any way, to detect text encoding?  
For example, is it in utf8, or in win1251, or something else.

Thank you.

---

<div class="post-metadata">

**Author:** ![xTRiM](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/ad7895/32.png) [@xTRiM](https://www.ruby-forum.com/u/xTRiM)\
**Post date:** [July 10, 2006, 1:10pm UTC](https://www.ruby-forum.com/t/text-encodings/97755/2 "2006-07-10T13:10:19Z")

</div>

On 10/07/06, xTRiM [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> is there any way, to detect text encoding?  
> For example, is it in utf8, or in win1251, or something else.

You can’t detect one-byte-per-character encodings easily (i.e. without  
statistical analysis) but you can easily tell if something’s UTF-8 or  
not:

class String  
def is\_utf8?  
unpack(‘U\*’)  
return true  
rescue  
return false  
end  
end

“foo”.is\_utf8? #=\> true  
“foo\303”.is\_utf8? #=\> false

Not the most efficient way, necessarily, but probably the easiest.

Paul.

---

<div class="post-metadata">

**Author:** ![xTRiM](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/ad7895/32.png) [@xTRiM](https://www.ruby-forum.com/u/xTRiM)\
**Post date:** [July 10, 2006, 7:48pm UTC](https://www.ruby-forum.com/t/text-encodings/97755/3 "2006-07-10T19:48:29Z")

</div>

On Jul 10, 2006, at 4:47 AM, Takashi Sano wrote:

> > is there any way, to detect text encoding?  
> > For example, is it in utf8, or in win1251, or something else.
> 
> You can use the standard lib NKF’s guess or guess2 (ruby 1.8.2 or  
> later) method for that. Look up the NKF section in  
> [RDoc Documentation](http://www.ruby-doc.org/stdlib/).

In the general case, there’s _no safe way_ to do this, unless the  
data is XML or comes with an HTTP header from a reliable server (ha  
ha ha, I’m sure there must be one somewhere). Probably the best auto-  
detecter is Mark Pilgrim’s, but it’s in Python: http://  
[chardet.feedparser.org/](http://chardet.feedparser.org/)

-Tim

---

<div class="post-metadata">

**Author:** ![xTRiM](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/ad7895/32.png) [@xTRiM](https://www.ruby-forum.com/u/xTRiM)\
**Post date:** [July 10, 2006, 8:20pm UTC](https://www.ruby-forum.com/t/text-encodings/97755/4 "2006-07-10T20:20:41Z")

</div>

> data is XML or comes with an HTTP header from a reliable server (ha  
> ha ha, I’m sure there must be one somewhere). Probably the best auto-  
> detecter is Mark Pilgrim’s, but it’s in Python: http://  
> [chardet.feedparser.org/](http://chardet.feedparser.org/)
> 
> -Tim

Nice pointer, Tim. I’ll have to check that out. I did a quick web search  
and found a Ruby port incidentally (I have not evaluated it in any way  
though):  
[http://rubyforge.org/projects/chardet/](http://rubyforge.org/projects/chardet/) by Hui Zheng  
gem name is “chardet”

Jake

---

<div class="post-metadata">

**Author:** ![xTRiM](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/ad7895/32.png) [@xTRiM](https://www.ruby-forum.com/u/xTRiM)\
**Post date:** [July 10, 2006, 1:48pm UTC](https://www.ruby-forum.com/t/text-encodings/97755/5 "2006-07-10T13:48:01Z")

</div>

Hi,

2006/7/10, xTRiM [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> Hello,
> 
> is there any way, to detect text encoding?  
> For example, is it in utf8, or in win1251, or something else.

You can use the standard lib NKF’s guess or guess2 (ruby 1.8.2 or  
later) method for that. Look up the NKF section in  
[RDoc Documentation](http://www.ruby-doc.org/stdlib/).

Takashi Sano