Problems with UTF-8 and AJAX

Hey :slight_smile:

I have an AJAX tree to manage pages in a CMS-like manner. The leaves
can be clicked to expand an editor which allows the backend admin to
alter settings, set page titles, add content, etc. The site’s primary
language is Danish, so we have a lot of æ, ø, and å characters.

If I got into the backend pages administration module and render the
tree, these special characters are rendered correctly, so that all
visible leaves with æ, ø, or å in them show these special characters
correctly. The page information tab in the Web D. plugin for
Firefox show content encoding to be UTF-8 for both the response header
and the document encoding.

If i then click a leaf to expand it, an AJAX call is made to render a
partial in which I get a form with fields that will allow me to edit
titles and texts, among other things. Still, æ, ø, and å are displayed
correctly.

If I then submit the form, again via AJAX, the data is saved correctly
(I can confirm that æ, ø, and å characters exist in the database, and
that they’re rendered correctly in the frontend, and also in the
backend if I refresh the entire page), but the form that is refreshed
and rerended inside the tree leaf and which now contains the edited
and saved data, now shows special characters like “?” for text fields,
and for text areas it’s the same character except it displays like a
diamond with a question mark inside of it - this is also how it looks
if I do a View Source on the page.

If I choose to change something else inside the form and save again,
all the “?” characters are now stripped away effectively killing all
æ, ø, and å characters.

Has anyone experienced something similar or have any good ideas about
where I might go and look to find clues?

Below are the headers from my request as shown by Firebug.

Thanks in advance for any help,
Daniel :slight_smile:

Headers:

Expanding the leaf (æøå all OK):

Response Headers
Connection close
Date Tue, 01 May 2007 08:05:16 GMT
Set-Cookie polyteknisk_boghandel=b9ddf690012462818663652131508fcd;
path=/; expires=Sun, 30 Apr 2017 19:52:59 GMT
Status 200 OK
Cache-Control no-cache
Server Mongrel 1.0.1
Content-Type text/javascript; charset=utf-8
Content-Length 24423

Request Headers
Host localhost:3000
User-Agent Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.1) Gecko/
20061115 Mandriva/2.0.0.1-4mdv2007.0 (2007.0) Firefox/2.0.0.1
Accept text/javascript, text/html, application/xml, text/xml, /
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 300
Connection keep-alive
X-Requested-With XMLHttpRequest
X-Prototype-Version 1.5.0
Content-Type application/x-www-form-urlencoded; charset=UTF-8
Referer http://localhost:3000/manage/pages
Content-Length 0
Cookie polyteknisk_boghandel=b9ddf690012462818663652131508fcd; dbx-
postmeta=grabit=0-,1-,2-,3-,4-,5-,6-&advancedstuff=0+,1-,2+,3-,4+,5-
Pragma no-cache
Cache-Control no-cache

Saving inside the leaf and getting a refreshed version of the form
(æøå garbled):

Response Headers
Connection close
Date Tue, 01 May 2007 08:05:49 GMT
Set-Cookie polyteknisk_boghandel=b9ddf690012462818663652131508fcd;
path=/; expires=Sun, 30 Apr 2017 19:52:59 GMT
Status 200 OK
Cache-Control no-cache
Server Mongrel 1.0.1
Content-Type text/javascript; charset=utf-8
Content-Length 27637

Request Headers
Host localhost:3000
User-Agent Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.1) Gecko/
20061115 Mandriva/2.0.0.1-4mdv2007.0 (2007.0) Firefox/2.0.0.1
Accept text/javascript, text/html, application/xml, text/xml, /
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 300
Connection keep-alive
X-Requested-With XMLHttpRequest
X-Prototype-Version 1.5.0
Content-Type application/x-www-form-urlencoded; charset=UTF-8
Referer http://localhost:3000/manage/pages
Content-Length 252
Cookie polyteknisk_boghandel=b9ddf690012462818663652131508fcd; dbx-
postmeta=grabit=0-,1-,2-,3-,4-,5-,6-&advancedstuff=0+,1-,2+,3-,4+,5-
Pragma no-cache
Cache-Control no-cache

FYI, I have another administration module which does not use the tree
structure, but still uses a remote_form_for just like the other
administration module, however this one has no problems like described
above. Its headers on a save are here:


Response Headers
Connection close
Date Tue, 01 May 2007 08:38:02 GMT
Set-Cookie polyteknisk_boghandel=b9ddf690012462818663652131508fcd;
path=/; expires=Sun, 30 Apr 2017 19:52:59 GMT
Status 200 OK
Cache-Control no-cache
Server Mongrel 1.0.1
Content-Type text/html; charset=utf-8
Content-Length 6435

Request Headers
Host localhost:3000
User-Agent Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.1) Gecko/
20061115 Mandriva/2.0.0.1-4mdv2007.0 (2007.0) Firefox/2.0.0.1
Accept text/javascript, text/html, application/xml, text/xml, /
Accept-Language en-us,en;q=0.5
Accept-Encoding gzip,deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 300
Connection keep-alive
X-Requested-With XMLHttpRequest
X-Prototype-Version 1.5.0
Content-Type application/x-www-form-urlencoded; charset=UTF-8
Referer http://localhost:3000/manage/system_settings
Content-Length 811
Cookie polyteknisk_boghandel=b9ddf690012462818663652131508fcd; dbx-
postmeta=grabit=0-,1-,2-,3-,4-,5-,6-&advancedstuff=0+,1-,2+,3-,4+,5-
Pragma no-cache
Cache-Control no-cache

Ok, I just spotted the difference in content-type on the different
headers, text/html vs. text/javascript. The latter breaks. Still
stumped, though. Any ideas?

More info:

The one that breaks uses render :update, while the ‘nice’ one uses
render :partial…

Not sure if that means that the first one is RJS, and the other is
not, but either way… I’m a bit 'fused about what to do about this…
I can’t very well alter the content-type to text/html as it really IS
javascript and needs to be executed, not inserted…

Argh!

Apologies, it had nothing to do with AJAX, it was the fact that I was
converting from UTF-8 to ISO-8859-1 before_validation because the
backend server is MS SQL, and forgetting to convert back after save.
I’ve done that now, and life is sweet :slight_smile: