# Converting data to utf8 to load into an ActiveRecord?

**URL:** <https://www.ruby-forum.com/t/converting-data-to-utf8-to-load-into-an-activerecord/132500>\
**Category:** Rails\
**Created:** [March 20, 2008, 7:37pm UTC](https://www.ruby-forum.com/t/converting-data-to-utf8-to-load-into-an-activerecord/132500 "2008-03-20T19:37:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Henry\_W](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/c77e96/32.png) [@Henry\_W](https://www.ruby-forum.com/u/Henry_W)\
**Post date:** [March 20, 2008, 7:37pm UTC](https://www.ruby-forum.com/t/converting-data-to-utf8-to-load-into-an-activerecord/132500/1 "2008-03-20T19:37:39Z")

</div>

Hi,  
I’ve got a text file of data that I need to parse and loaded into my  
Rails  
app. I’ve created a migration script the loads and parses the file, but  
some  
of the data contains non-utf8 strings, causing an exception.

I’ve found two Windows only ways to convert the strings to utf8, but I  
would  
like to deploy my app on heroku. Any suggestions for a non OS specific  
way  
to convert?

This is my current implementation using chilkat:

require ‘db/chilkat’

class AddDataSources \< ActiveRecord::Migration  
def self.encode(str)  
strObj = Chilkat::CkString.new()  
strObj.appendAnsi(str)  
strObj.urlDecode(“utf-8”)  
end

def self.up  
DataSource.delete\_all  
data = IO.readlines(‘db/data/data.txt’)  
data.each { |line|  
cells = line.split(‘,’)  
DataSource.create(:code =\> cells[0],  
:authors =\> encode(cells[1]))  
}  
end

def self.down  
DataSource.delete\_all  
end  
end

–  
Henry  
[http://www.henrywagner.org/](http://www.henrywagner.org/)

---

<div class="post-metadata">

**Author:** ![Henry\_W](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/c77e96/32.png) [@Henry\_W](https://www.ruby-forum.com/u/Henry_W)\
**Post date:** [March 20, 2008, 8:13pm UTC](https://www.ruby-forum.com/t/converting-data-to-utf8-to-load-into-an-activerecord/132500/2 "2008-03-20T20:13:03Z")

</div>

2008/3/20, Henry W. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> Hi,
> 
> I’ve got a text file of data that I need to parse and loaded into my Rails  
> app. I’ve created a migration script the loads and parses the file, but some  
> of the data contains non-utf8 strings, causing an exception.
> 
> I’ve found two Windows only ways to convert the strings to utf8, but I would  
> like to deploy my app on heroku. Any suggestions for a non OS specific way  
> to convert?

Iconv?

```
irb(main):001:0> require "iconv"
=> true
irb(main):002:0> Iconv.conv("utf-8", "windows-1252", "\200")
=> "€"

```

Stefan

---

<div class="post-metadata">

**Author:** ![Henry\_W](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/h/c77e96/32.png) [@Henry\_W](https://www.ruby-forum.com/u/Henry_W)\
**Post date:** [March 20, 2008, 8:27pm UTC](https://www.ruby-forum.com/t/converting-data-to-utf8-to-load-into-an-activerecord/132500/3 "2008-03-20T20:27:44Z")

</div>

Thanks, that works.

On Thu, Mar 20, 2008 at 12:12 PM, Stefan L. \<  
removed\_email\_address@domain.invalid\> wrote:

> > I’ve found two Windows only ways to convert the strings to utf8, but I  
> > =\> “€”
> 
> Stefan
> 
> >

–  
Henry  
[http://www.henrywagner.org/](http://www.henrywagner.org/)