# Microsoft "stupid quotes" in params

**URL:** <https://www.ruby-forum.com/t/microsoft-stupid-quotes-in-params/68470>\
**Category:** Rails\
**Created:** [September 12, 2006, 11:10pm UTC](https://www.ruby-forum.com/t/microsoft-stupid-quotes-in-params/68470 "2006-09-12T23:10:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mcampbell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8e7dd6/32.png) [@mcampbell](https://www.ruby-forum.com/u/mcampbell)\
**Post date:** [September 12, 2006, 11:10pm UTC](https://www.ruby-forum.com/t/microsoft-stupid-quotes-in-params/68470/1 "2006-09-12T23:10:03Z")

</div>

I’m looking for feedback on this, read on, please…

I’m just fed up with Microsoft’s “stupid quotes” feature (and for sake  
of later searchers, I’ll add that it’s often known as “smart quotes”,  
although as with anything Microsoft you’re safe to substitute the word  
“stupid” anywhere they use the word “smart”).

I just completed a nice application, and suddenly an external piece  
failed. It first uses xmlrpc to grab some data from the database and  
stick it in a yaml file. A couple of other programs read the yaml file  
and create various other files. Those programs were crapping because it  
couldn’t read the entire yaml file.

It turns out that the problem was with people using stupid quotes.  
Here’s the sledgehammer that I applied in  
app/controllers/application.rb:

before\_filter :fix\_stupid\_quotes\_in\_params

def fix\_stupid\_quotes\_in\_params  
dig\_deep(@params) { |s| fix\_stupid\_quotes!(s) }  
end

def dig\_deep(hash, &block)  
if hash.instance\_of? String  
yield(hash)  
elsif hash.kind\_of? Hash  
hash.each\_key { |h| dig\_deep(hash[h]) { |s| block.call(s) } }  
else  
nil  
end  
end

def fix\_stupid\_quotes!(s)  
s.gsub!(/\x82/,‘,’)  
s.gsub!(/\x84/,‘,’)  
s.gsub!(/\x85/,‘…’)  
s.gsub!(/\x88/,‘^’)  
s.gsub!(/\x89/,‘o/oo’)  
s.gsub!(/\x8b/,‘\<’)  
s.gsub!(/\x8c/,‘OE’)  
s.gsub!(/\x91|\x92/,“'”)  
s.gsub!(/\x93|\x94/,‘"’)  
s.gsub!(/\x95/,‘\*’)  
s.gsub!(/\x96/,‘-’)  
s.gsub!(/\x97/,‘–’)  
s.gsub!(/\x98/,‘~’)  
s.gsub!(/\x99/,‘TM’)  
s.gsub!(/\x9b/,‘\>’)  
s.gsub!(/\x9c/,‘oe’)  
end

If this is a bad idea, I’ll have to implement it on one particular page.  
The fact is, though, that these characters are always invalid (in  
Latin/UTF-8 type char sets) so I see no reason to allow them through  
ever. I hate modifying the params, but again, these are just not valid  
characters. I don’t want to have to think about it in each model or  
controller.

This is a sledgehammer approach, as it will always walk through params  
on every page and fix the stupid quotes characters. I’m looking for any  
thoughts, suggestions, comments, etc. on the above code.

## Thanks, Michael

Michael Darrin Chaney  
removed\_email\_address@domain.invalid  
[http://www.michaelchaney.com/](http://www.michaelchaney.com/)

---

<div class="post-metadata">

**Author:** ![mcampbell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8e7dd6/32.png) [@mcampbell](https://www.ruby-forum.com/u/mcampbell)\
**Post date:** [November 19, 2007, 6:06pm UTC](https://www.ruby-forum.com/t/microsoft-stupid-quotes-in-params/68470/2 "2007-11-19T18:06:18Z")

</div>

Michael C. wrote:

> I’m looking for feedback on this, read on, please…  
> these characters are always invalid (in Latin/UTF-8 type char sets)

It may be hard to go back and recreate all this. I bumped into this  
page and thought I’d comment.

My question is if Microsoft is reporting that the request is in UTF-8.  
It may say that the request is in some other code page. And, if you  
want UTF-8 and the request is not in UTF-8, then you need to convert it  
using something like iconv.

If the request says it is utf-8 and, as you point out, it has invalid  
utf-8 code points, then you have a great reason to bitch.

---

<div class="post-metadata">

**Author:** ![mcampbell](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/8e7dd6/32.png) [@mcampbell](https://www.ruby-forum.com/u/mcampbell)\
**Post date:** [November 20, 2007, 1:57am UTC](https://www.ruby-forum.com/t/microsoft-stupid-quotes-in-params/68470/3 "2007-11-20T01:57:51Z")

</div>

IIRC, these are double-byte characters. The problem is not so much in  
using them, but in interpreting them. For e.g., user types stuff into  
Word, copies, then pastes into a textarea. Hits send. Application  
obediently stores in database. Application displays data and s(mart|  
upid) quotes are in place correctly. Then programmer gets a wild idea  
– like restoring the database from a backup. Because the backup is a  
text file, the DBCs are misinterpreted as they are imported into the  
database. Result, improper display of these characters.

If you come up with a solution that works for these cute characters  
that (whatever you call them) everyone has in their word processing  
documents, let us all know. Here are some references that sort of work:

demoronizer (Perl script) www.fourmilab.ch/webtools/demoroniser

I can’t attribute this second one, but it’s a shell script. I tried  
it on a database dump and it left me with less cleanup work – maybe  
it will provide some clues for you:

#!/bin/sh  
this\_directory=`pwd`  
for x  
do  
echo -n "converting $x: "  
if test “$x” = runiconv.sh; then  
echo “not editing script itself!”  
elif [-d $x]; then  
(cp runiconv.sh $x; cd $x; sh runiconv.sh \*; rm -f runiconv.sh cd … )  
elif test -s $x; then  
iconv --from-code=euc-kr --to-code=UTF-8 \< $x \> $this\_directory/$x$$ ;  
if [$? == 0]  
then  
cp $this\_directory/$x$$ $x  
rm -f $this\_directory/$x$$  
else  
echo -n "ICONVE ERROr "  
rm -f $this\_directory/$x$$  
fi  
echo “done”;  
else  
echo “original file is empty”  
fi  
done  
echo “all done”