# Legacy ISO-8859-1 MSSQL db and UTF8 rails app **URL:** \ **Category:** Rails I18n\ **Created:** [August 28, 2007, 10:55am UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168 "2007-08-28T10:55:04Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [August 28, 2007, 10:55am UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/1 "2007-08-28T10:55:04Z")
Hi! I have problem with legacy database in MSSQL encoding. The new app in Rails and most of the database is Utf8, while tables from legacy MSSQL are encoded in ISO-8859-1. I am not sure how to approach this problem. I would imagine creation of some kind of filter with Iconv.conv(“UTF-8”, “ISO-8859-1”, input) for reading sql statements and the reverse for write methods. Any suggestions how this could be done? Cheers, Tomasz Bak ---
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [August 28, 2007, 12:15pm UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/2 "2007-08-28T12:15:36Z")
Hi, So far I have come come with poor solution: class String require ‘iconv’ def iso1\_to\_utf8 Iconv.conv(“UTF-8”, “ISO-8859-1”, self) rescue self end def utf8\_to\_iso1 Iconv.conv(“ISO-8859-1”, “UTF-8”, self) rescue self end end class NCustomer \< ActiveRecord::Base … def after\_initialize self.attributes.each do |k,v| if v.class.to\_s == “String” and !v.empty? self[k] = v.iso1\_to\_utf8 end end end def before\_save self.attributes.each do |k,v| if v.class.to\_s == “String” and !v.empty? self[k] = v.utf8\_to\_iso1 end end end … end Cheers, Tomasz Bak ---
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [August 28, 2007, 9:31pm UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/3 "2007-08-28T21:31:01Z")
Tom Bak wrote: > I have problem with legacy database in MSSQL encoding. > > The new app in Rails and most of the database is Utf8, while > tables from legacy MSSQL are encoded in ISO-8859-1. > > I am not sure how to approach this problem. I would imagine > creation of some kind of filter with Iconv.conv(“UTF-8”, “ISO-8859-1”, > input) > for reading sql statements and the reverse for write methods. > > Any suggestions how this could be done? I think that everyone who’s ever been cornered by this can attest that it can be a royal pain and that the options are all choosing between lesser evils. Last time I found myself there I exported the database to a plain text SQL file and then ran recode [1] on it. It worked well enough but I still found myself scanning over the file to correct tiny bits here and there. I’ll admit that your solution has a ring to it – being able to do it on the fly improves the time to deploy. I like it. If however speed or strict migration paths are an issue for you then recode may be an option to consider. [1] [Recode](http://www.gnu.org/software/recode/) – Roderick van Domburg > **[De domeinnaam Nedforce.nl is te koop.](http://www.nedforce.nl)** > > Deze mooie domeinnaam kan al over een paar minuten van jou zijn. Betalen gaat eenvoudig en vertrouwd via iDEAL. Direct na de betaling ontvang je de verhuiscode per e-mail en kun je aan de slag. ---
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [September 3, 2007, 11:33am UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/4 "2007-09-03T11:33:24Z")
I solved my problem with own plugin: recode\_legacy\_database :from =\> “ISO-8859-1”, :to =\> “UTF-8” Which translates fields in models on read and before write to database. Cheers, Tomasz ---
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [September 3, 2007, 11:02am UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/5 "2007-09-03T11:02:15Z")
i’m having a quite similar issue atm and i’m unsure what to do. same thing, Legacy MSSQL Database with windows-1252 charset (so iso compatible) on the legacy tables, and the Rails app purely using UTF-8. i had no problems with encodings while using the sqlserver driver in ado mode and setting: WIN32OLE.codepage = WIN32OLE::CP\_UTF8 but unfortunately the ado driver is leaking memory so badly that the mongrels need to be restarted every 2 hours. when i switch to odbc mode, the memory leak is gone but with ODBC the codepage setting doesn’t work anymore, so i get messed up texts from those old tables. so i have to choose between the 2 evils: 1. memory leaking app with proper encoding or 2. stable app with garbled text ---
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [September 3, 2007, 12:09pm UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/6 "2007-09-03T12:09:08Z")
Tom Bak wrote: > I solved my problem with own plugin: > > recode\_legacy\_database :from =\> “ISO-8859-1”, :to =\> “UTF-8” > > Which translates fields in models on read and before write to database. > > Cheers, > Tomasz so this is for a one time conversion of the whole db? i don’t have the luxury to do that because the legacy tables still get accessed by the legacy app ☹ ---
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [September 3, 2007, 12:15pm UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/7 "2007-09-03T12:15:40Z")
neongrau \_\_ wrote: > Tom Bak wrote: > > > I solved my problem with own plugin: > > > > recode\_legacy\_database :from =\> “ISO-8859-1”, :to =\> “UTF-8” > > > > Which translates fields in models on read and before write to database. > > > > Cheers, > > Tomasz > > so this is for a one time conversion of the whole db? Exacly the oposit. I probably should change naming. I recodes db “on fly”. Check the code. Cheers, Tomasz Bak ---
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [November 5, 2007, 4:39pm UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/8 "2007-11-05T16:39:27Z")
Tom Bak wrote: > I solved my problem with own plugin: > > recode\_legacy\_database :from =\> “ISO-8859-1”, :to =\> “UTF-8” > > Which translates fields in models on read and before write to database. > > Cheers, > Tomasz Tom, How is the usage of this plugin? I have to run this command manual? When I have to put this code? Thanks, Julio ---
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [December 18, 2007, 2:46pm UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/9 "2007-12-18T14:46:45Z")
Julio Damasceno wrote: > Tom Bak wrote: > > > I solved my problem with own plugin: > > > > recode\_legacy\_database :from =\> “ISO-8859-1”, :to =\> “UTF-8” > > > > Which translates fields in models on read and before write to database. > > > > Cheers, > > Tomasz > > Tom, > > How is the usage of this plugin? I have to run this command manual? When > I have to put this code? > > Thanks, > > Julio I did it this way : - Unzip the zip-file into you rails vendor/plugins directory - Add the following line to environments.rb ActiveRecord::Base.recode\_legacy\_database :from=\>“ISO-8859-1”,:to=\>“UTF-8” ---
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [November 17, 2008, 2:42pm UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/10 "2008-11-17T14:42:41Z")
I had problems with blob fields beeing treated as Strings as well. I made a little change, so it is posible to exclude some fields from the conversion. use in model like this: recode\_legacy\_database :from =\> “ISO-8859-1”, :to =\> “UTF-8”, :exclude =\> [“myblob1”,“myblob2”] ---
**Author:** ![linus](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/f4b2a3/32.png) [@linus](https://www.ruby-forum.com/u/linus)\ **Post date:** [May 6, 2010, 12:44pm UTC](https://www.ruby-forum.com/t/legacy-iso-8859-1-mssql-db-and-utf8-rails-app/113168/11 "2010-05-06T12:44:45Z")
Tomasz Bak wrote: > I solved my problem with own plugin: > > recode\_legacy\_database :from =\> “ISO-8859-1”, :to =\> “UTF-8” > > Which translates fields in models on read and before write to database. > > Cheers, > Tomasz Dear Tomasz I’ll never be able to thank you enough for this recode plugin! You are my saviour Tommaso