# Help with str.sub(pattern, replacement) and french characters

**URL:** <https://www.ruby-forum.com/t/help-with-str-sub-pattern-replacement-and-french-characters/202159>\
**Category:** Rails I18n\
**Created:** [January 23, 2011, 4:04pm UTC](https://www.ruby-forum.com/t/help-with-str-sub-pattern-replacement-and-french-characters/202159 "2011-01-23T16:04:58Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\
**Post date:** [January 23, 2011, 4:04pm UTC](https://www.ruby-forum.com/t/help-with-str-sub-pattern-replacement-and-french-characters/202159/1 "2011-01-23T16:04:58Z")

</div>

I am trying to search my database which has product names with french  
accents. They are encoded using the html entity codes such as &eactue;  
etc.

If a user enters a word with a french accent in the search box I must  
convert it to the html entity code so it can be found in the database.

So I thought to use str.sub(pattern, replacement) =\> new\_str

However when I try this using product.sub(‘é’,‘é’) for example it  
results in the following:

find(:all, :select =\> ‘product\_id, name’, :order =\> “name”, :conditions  
=\> [“name like ? and locale =?”, “%#{product.sub(‘é’,‘é’)}%”,  
I18n.locale])

When I enter ‘é’ in the seach box I get the following:

SELECT product\_id, name FROM `product_descriptions` WHERE (name like  
‘%é%’ and locale =‘en’)

so it does not replace the ‘é’ with ‘&eactue’

But if I change the letter from ‘é’ to ‘e’ and do a search for ‘e’ I  
get the following:

SELECT product\_id, name FROM `product_descriptions` WHERE (name like  
‘%é%’ and locale =‘en’)

so the replacement works.

Can anyone explain why it won’t work for the character with french  
accent?

Thank you in advance.  
Mitch

---

<div class="post-metadata">

**Author:** ![mitch\_newbie](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ea5d25/32.png) [@mitch\_newbie](https://www.ruby-forum.com/u/mitch_newbie)\
**Post date:** [January 23, 2011, 4:28pm UTC](https://www.ruby-forum.com/t/help-with-str-sub-pattern-replacement-and-french-characters/202159/2 "2011-01-23T16:28:47Z")

</div>

Ideally, you shouldn’t have HTML entities in the database. If you need  
them  
in your HTML (and you don’t, if you set an explicit encoding, except for  
things like &\<\>) then you should add them outside the database.

If you do have “” stored in the database, not as an entity, I believe  
MySQL’s “LIKE” will be accent-insensitive by default, unless you use  
“COLLATE utf8\_bin” (google for details).

Note that if you use “sub”, you will only replace the first occurrence  
in  
the string. You probably want “gsub”.

And if you do something like “blh”.sub("", “é”) and it doesn’t  
replace the “”, the issue could be how the “” is represented. In UTF-8,  
accented characters can be represented either composed as a single glyph  
(“latin small letter e with acute”) or decomposed as two glyphs: “latin  
small letter e” + “combining acute accent”. So if your string contains  
the  
first type of and your sub/gsub tries to replace the other type, it  
won’t  
work. You can normalize the string to ensure everything is composed or  
decomposed, but it would be better not to have entities in the database.

---

<div class="post-metadata">

**Author:** ![mitch\_newbie](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ea5d25/32.png) [@mitch\_newbie](https://www.ruby-forum.com/u/mitch_newbie)\
**Post date:** [January 24, 2011, 12:14am UTC](https://www.ruby-forum.com/t/help-with-str-sub-pattern-replacement-and-french-characters/202159/3 "2011-01-24T00:14:43Z")

</div>

Henrik — wrote in post #976928:

> Ideally, you shouldn’t have HTML entities in the database. If you need  
> them  
> in your HTML (and you don’t, if you set an explicit encoding, except for  
> things like &\<\>) then you should add them outside the database.
> 
> If you do have “” stored in the database, not as an entity, I believe  
> MySQL’s “LIKE” will be accent-insensitive by default, unless you use  
> “COLLATE utf8\_bin” (google for details).
> 
> Note that if you use “sub”, you will only replace the first occurrence  
> in  
> the string. You probably want “gsub”.
> 
> And if you do something like “blh”.sub(“”, “é”) and it doesn’t  
> replace the “”, the issue could be how the “” is represented. In UTF-8,  
> accented characters can be represented either composed as a single glyph  
> (“latin small letter e with acute”) or decomposed as two glyphs: “latin  
> small letter e” + “combining acute accent”. So if your string contains  
> the  
> first type of and your sub/gsub tries to replace the other type, it  
> won’t  
> work. You can normalize the string to ensure everything is composed or  
> decomposed, but it would be better not to have entities in the database.

Ok I will take your advise and remove the html entities from the  
database.  
The reason I put them in was because even with explicit encoding I was  
not getting the characters to show properly. I was getting a black  
triangle with a question mark.

Could you assist me on how to encode the web page so that it shows the  
accents. I thought you just use UTF-8?

Thanks for your help. I really appreciate it.

---

<div class="post-metadata">

**Author:** ![mitch\_newbie](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ea5d25/32.png) [@mitch\_newbie](https://www.ruby-forum.com/u/mitch_newbie)\
**Post date:** [January 24, 2011, 12:20am UTC](https://www.ruby-forum.com/t/help-with-str-sub-pattern-replacement-and-french-characters/202159/4 "2011-01-24T00:20:55Z")

</div>

Yes, encode the file in UTF-8 and add a tag like this on your head  
section:

---

<div class="post-metadata">

**Author:** ![mitch\_newbie](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ea5d25/32.png) [@mitch\_newbie](https://www.ruby-forum.com/u/mitch_newbie)\
**Post date:** [January 24, 2011, 12:22am UTC](https://www.ruby-forum.com/t/help-with-str-sub-pattern-replacement-and-french-characters/202159/5 "2011-01-24T00:22:12Z")

</div>

On Mon, Jan 24, 2011 at 00:14, Mitchell G. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Ok I will take your advise and remove the html entities from the  
> database.  
> The reason I put them in was because even with explicit encoding I was  
> not getting the characters to show properly. I was getting a black  
> triangle with a question mark.
> 
> Could you assist me on how to encode the web page so that it shows the  
> accents. I thought you just use UTF-8?

Check what your browser thinks the encoding is. Check that UTF-8 is  
declared  
in the HTTP headers or a meta element (and if they disagree, I’m not  
entirely sure what goes - research that).  
[UTF-8: The Secret of Character Encoding - HTML Purifier](http://htmlpurifier.org/docs/enduser-utf8.html) has some info.

Also ensure the font you’re using can handle that glyph. I would guess  
most  
fonts can display . But if everything else looks right, try some  
standard  
font like Times and see what happens.

---

<div class="post-metadata">

**Author:** ![mitch\_newbie](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ea5d25/32.png) [@mitch\_newbie](https://www.ruby-forum.com/u/mitch_newbie)\
**Post date:** [January 24, 2011, 8:39am UTC](https://www.ruby-forum.com/t/help-with-str-sub-pattern-replacement-and-french-characters/202159/6 "2011-01-24T08:39:12Z")

</div>

I removed some HTML entities from my database to test the effect. I made  
sure my web page is UTF-8 encoded.

Now instead of “électronique” I get name: “\xC9lectronic” where  
the"\xC9" displays like a black triangle with a “?” in it.

I also changed the font to times.

I read up and learned that MYSQL might be delivering the characters in a  
format other than UTF-8.

I changed my database, table, and field to be UTF-8.

I still get the same problem as stated above.

What gives?

Thanks in advance

MItch

---

<div class="post-metadata">

**Author:** ![mitch\_newbie](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ea5d25/32.png) [@mitch\_newbie](https://www.ruby-forum.com/u/mitch_newbie)\
**Post date:** [January 24, 2011, 8:55am UTC](https://www.ruby-forum.com/t/help-with-str-sub-pattern-replacement-and-french-characters/202159/7 "2011-01-24T08:55:03Z")

</div>

Hi,  
I figured it all out. I need to explicitly tell Rails that the database  
is

using utf8 encoding by putting the following in the database.yml file

encoding: utf8

now it displays perfectly.

I hope this is still in line with best practices as I don’t want to mess  
this up again.

Thanks

MItch