Database Interaction Encoding Problem

This is cross-posted to railsforum.com, so if you frequent both you may
see it in both places.

So, my co-worker and I are working on this project, a Rails app using an
MS SQL database with lots of legacy stuff built in. He’s stumbled
across a strange error, and we don’t really know where to start. Here’s
the gist of things:

This is an online place for buying things, and descriptions for each
item are stored in
the database as encoded strings. On a particular product, pulling a
database object from the database with Product.find formats this string
incorrectly, replacing encoded characters with ‘?’, in all environments
on both servers we run this project from, EXCEPT from the CONSOLE of the
production environment (in which case, everything comes out exactly as
it should). In development environments, both the console and the app
return mis-encoded strings. In the actual production application
(diagnosed with logs, and shown on the actual page), the encoding is
messed up. SO, the question is:

What could be giving different behavior between the console and the app
of the same environment? Why isn’t this difference consistent among
environments (i.e., why isn’t the output sane in the development
environment console)? If we could nail that difference, we could
compensate for whatever is giving this wonky output. As it is, we’re
pretty stumped.

Thanks in advance for any response.

On 11 Sep 2008, at 16:37, Aa Wilson wrote:

the gist of things:
production environment (in which case, everything comes out exactly as
compensate for whatever is giving this wonky output. As it is, we’re
pretty stumped.

Could be what your terminal is set to dispay, eg if the terminal on
the production machine is set ti displaying stuff in ISO-Latin-1 and
the underlying data is too but everyone else is expecting the data to
be utf-8 then you might get what you observe. I’d check if the
database encoding setting (in database.yml) matches what the database
is doing and that any headers on the actual page saying what the
content-type/charset are also match that.

Fred

Frederick C. wrote:

On 11 Sep 2008, at 16:37, Aa Wilson wrote:

the gist of things:
production environment (in which case, everything comes out exactly as
compensate for whatever is giving this wonky output. As it is, we’re
pretty stumped.

Could be what your terminal is set to dispay, eg if the terminal on
the production machine is set ti displaying stuff in ISO-Latin-1 and
the underlying data is too but everyone else is expecting the data to
be utf-8 then you might get what you observe. I’d check if the
database encoding setting (in database.yml) matches what the database
is doing and that any headers on the actual page saying what the
content-type/charset are also match that.

Fred

It turned out to be a setting in our freetds.conf (client charset=utf8)
that was missing. It took us forever to find out that was the case,
because the damn production console behavior was throwing us off, so we
worked our way through the entire chain before we eliminated everything
but tds.

Thanks for your response!