# Multibyte Character References

**URL:** <https://www.ruby-forum.com/t/multibyte-character-references/114119>\
**Category:** Rails\
**Created:** [September 8, 2007, 3:02am UTC](https://www.ruby-forum.com/t/multibyte-character-references/114119 "2007-09-08T03:02:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdaoust](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/dbc845/32.png) [@mdaoust](https://www.ruby-forum.com/u/mdaoust)\
**Post date:** [September 8, 2007, 3:02am UTC](https://www.ruby-forum.com/t/multibyte-character-references/114119/1 "2007-09-08T03:02:12Z")

</div>

Hi,

I have a load of records in my database which were imported through  
processing a YAML file. These original YAML files were created from the  
‘to\_yaml’ function of an array of Hash objects.

The YAML file contains multibyte character references such as:

…and between them and today\xE2\x80\x99s College. The scope, r…

When I imported this data into my DB these character references have  
changed but are still there in the DB:

…and between them and today\342\200\231s College. The scope, r…

So I have two questions:

1. Are the original characters retreivable from the copy in the DB, or  
has it been mangled?

2. If the above answer is yes, then how!

Really appreciate any help on this one. Many thanks in advance.

~ Mark

---

<div class="post-metadata">

**Author:** ![mdaoust](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/dbc845/32.png) [@mdaoust](https://www.ruby-forum.com/u/mdaoust)\
**Post date:** [September 8, 2007, 3:07am UTC](https://www.ruby-forum.com/t/multibyte-character-references/114119/2 "2007-09-08T03:07:09Z")

</div>

Mark D. wrote:

> When I imported this data into my DB these character references have
> 
> Really appreciate any help on this one. Many thanks in advance.
> 
> ~ Mark

What’s the encoding in the YAML file (presumably UTF-8), what database  
are you using and what encoding is your database/table set to?

–  
Michael W.

---

<div class="post-metadata">

**Author:** ![mdaoust](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/dbc845/32.png) [@mdaoust](https://www.ruby-forum.com/u/mdaoust)\
**Post date:** [September 8, 2007, 3:15am UTC](https://www.ruby-forum.com/t/multibyte-character-references/114119/3 "2007-09-08T03:15:07Z")

</div>

Hi Michael,

The DB is ‘ISO Latin 1 (latin1)’ encoding.

I’m not sure about the original YAML file (do you know the default  
encoding for .to\_yaml?) - but when I open it directly with, say  
TextMate, it shows the character reference _not_ the actual character.

Thanks,

~ Mark

---

<div class="post-metadata">

**Author:** ![mdaoust](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/dbc845/32.png) [@mdaoust](https://www.ruby-forum.com/u/mdaoust)\
**Post date:** [September 8, 2007, 3:23am UTC](https://www.ruby-forum.com/t/multibyte-character-references/114119/4 "2007-09-08T03:23:10Z")

</div>

Mark D. wrote:

> ~ Mark  
> MySQL, if that’s what you are using, let’s you set the character  
> encoding at various different levels (server, database, table, column).  
> If you are using MySQL you could try something like an ALTER TABLE to  
> change the encoding to UTF-8 (which I’m guessing is what the original  
> YAML data is in). You might have to export the data and import it into a  
> table that’s already set to UTF-8, though, in which case if you still  
> have all the YAML data around it might be easier just to reload that  
> with the table set to the proper encoding.

[http://dev.mysql.com/doc/refman/5.0/en/charset.html](http://dev.mysql.com/doc/refman/5.0/en/charset.html)

–  
Michael W.

---

<div class="post-metadata">

**Author:** ![mdaoust](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/dbc845/32.png) [@mdaoust](https://www.ruby-forum.com/u/mdaoust)\
**Post date:** [September 8, 2007, 8:03pm UTC](https://www.ruby-forum.com/t/multibyte-character-references/114119/5 "2007-09-08T20:03:50Z")

</div>

Thanks for the help Michael.

Eventually, I managed to sort this without having to reimport. Thought,  
I’d post how in case somebody else got stuck in a similar way.

```
Model.find_all.each do |m|
  content = m.content
  content = Iconv.iconv('ISO-8859-1//TRANSLIT', 'UTF-8', 

```

content).to\_s  
m.update\_attribute(:content, content)  
end

This translated the UTF-8 encoded chars into ISO-8859-1 encoded  
equicalents. They are not the exact characters obviously, but close  
approximations.

~ Mark