# Charset Detection

**URL:** <https://www.ruby-forum.com/t/charset-detection/191088>\
**Category:** Ruby\
**Created:** [July 5, 2010, 7:05am UTC](https://www.ruby-forum.com/t/charset-detection/191088 "2010-07-05T07:05:11Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![voxom](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/e9a140/32.png) [@voxom](https://www.ruby-forum.com/u/voxom)\
**Post date:** [July 5, 2010, 7:05am UTC](https://www.ruby-forum.com/t/charset-detection/191088/1 "2010-07-05T07:05:11Z")

</div>

Is there any existing functions or external gems that can take a generic  
string and parse for charset markers, converting that parts of the  
string to their appropriate charsets?

For example, theres an email with the headers  
Date: Sat, 03 Jul 2010 04:00:29 EDT  
From: =?iso-8859-1?B?U0FU?= [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
Subject:  
=?iso-8859-1?B?VGhlIE9mZmljaWFsIFNBVCBRdWVzdGlvbiBvZiB0aGUgRGF5?=

How can I convert this to what it should be,  
Date: Sat, 03 Jul 2010 04:00:29 EDT  
From: SAT [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
Subject: SAT Question of the Day

---

<div class="post-metadata">

**Author:** ![voxom](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/e9a140/32.png) [@voxom](https://www.ruby-forum.com/u/voxom)\
**Post date:** [July 5, 2010, 12:15pm UTC](https://www.ruby-forum.com/t/charset-detection/191088/2 "2010-07-05T12:15:42Z")

</div>

Shea B. wrote:

> Is there any existing functions or external gems that can take a generic  
> string and parse for charset markers, converting that parts of the  
> string to their appropriate charsets?
> 
> For example, theres an email with the headers  
> Date: Sat, 03 Jul 2010 04:00:29 EDT  
> From: =?iso-8859-1?B?U0FU?= [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> Subject:  
> =?iso-8859-1?B?VGhlIE9mZmljaWFsIFNBVCBRdWVzdGlvbiBvZiB0aGUgRGF5?=
> 
> How can I convert this to what it should be,  
> Date: Sat, 03 Jul 2010 04:00:29 EDT  
> From: SAT [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> Subject: SAT Question of the Day

Not answering your question directly, but this syntax is specific to  
MIME encoding of E-mail headers:

> **[MIME | Encoded-Word](https://en.wikipedia.org/wiki/MIME#Encoded-Word)**
>
> Since RFC 2822, conforming message header field names and values use ASCII characters; values that contain non-ASCII data should use the MIME encoded-word syntax (RFC 2047) instead of a literal string. This syntax uses a string of ASCII characters indicating both the original character encoding (the "charset") and the content-transfer-encoding used to map the bytes of the charset into ASCII characters.
> The form is: "=?charset?encoding?encoded text?=".
> 1.charset may be any character set register...

So if you look at ruby MIME toolkits you may find what you’re looking  
for.

---

<div class="post-metadata">

**Author:** ![voxom](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/v/e9a140/32.png) [@voxom](https://www.ruby-forum.com/u/voxom)\
**Post date:** [August 28, 2010, 7:48pm UTC](https://www.ruby-forum.com/t/charset-detection/191088/3 "2010-08-28T19:48:11Z")

</div>

heres the method I created in case it is useful to anyone else…

def convert\_mime\_encoded\_word(mime\_encoded\_word)  
require ‘iconv’  
require ‘base64’  
from\_charset, from\_encoding, encoded\_word =  
mime\_encoded\_word.scan(/=?([^?]+)?([BQ])?([^?]+)?=/i).first  
if from\_encoding == “Q”  
decoded\_word = encoded\_word.unpack(“M”).first  
elsif from\_encoding == “B”  
decoded\_word = encoded\_word.unpack(“m”).first  
end  
Iconv.iconv(“UTF8”, from\_charset, decoded\_word).first  
end