# Handling special characters

**URL:** <https://www.ruby-forum.com/t/handling-special-characters/144912>\
**Category:** Ruby\
**Created:** [August 4, 2008, 10:21am UTC](https://www.ruby-forum.com/t/handling-special-characters/144912 "2008-08-04T10:21:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sajal](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/96bed5/32.png) [@sajal](https://www.ruby-forum.com/u/sajal)\
**Post date:** [August 4, 2008, 10:21am UTC](https://www.ruby-forum.com/t/handling-special-characters/144912/1 "2008-08-04T10:21:58Z")

</div>

Hi all.

I am very new to Ruby (5 days old) so my question might sound very  
noobish. I am posting it only cause I couldn’t find a solution.

I am using ruby to scrape content of a site.

To be precise I am having problems with the ’ character.

Sample source page :  
[http://thainews.prd.go.th/newsenglish/previewnews.php?news\_id=255108040023](http://thainews.prd.go.th/newsenglish/previewnews.php?news_id=255108040023)

The encoding is in tis-620 and I use Iconv to convert it to utf8,  
however the special quote character gives the following error on iconv

/home/…/main.rb:37:in `iconv’: "\222s announcement "…  
(Iconv::IllegalSequence)

the affected code area

body = story.search(“//font[@color="#333333"]”).inner\_html  
body = body.gsub(/\<(.|\n)+?\>/, “”)  
body = body.gsub(/�/, “'”)  
puts body  
body = Iconv.iconv(“utf8”, “tis-620”, body) #\<-- this is line 37  
puts body

Or try the following on irb

require ‘rubygems’  
require ‘net/http’  
require ‘open-uri’  
require ‘iconv’  
story =  
Hpricot(open(‘[http://thainews.prd.go.th/newsenglish/previewnews.php?news\_id=255108040023](http://thainews.prd.go.th/newsenglish/previewnews.php?news_id=255108040023)’  
))  
body = story.search(“//font[@color="#333333"]”).inner\_html  
body = body.gsub(/\<(.|\n)+?\>/, “”)  
body = body.gsub(/’/, “'”)  
puts body

no matter whatever i put in the “’” it doesn’t replace anything and the  
iconv still gives errors.

I am looking for pointers on one of the following.

1. how do i replace “’” to “'” ?  
or 2) How can I make iconv ignore the “’” ?

At first I thought this to be a I18n issue, but i guess getting rid of  
the special character would be a simple string manipulation which i dont  
get.

---

<div class="post-metadata">

**Author:** ![sajal](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/96bed5/32.png) [@sajal](https://www.ruby-forum.com/u/sajal)\
**Post date:** [August 4, 2008, 10:24am UTC](https://www.ruby-forum.com/t/handling-special-characters/144912/2 "2008-08-04T10:24:07Z")

</div>

and oh. you would also need to  
require ‘mechanize’

in the irb to emulate the issue

require ‘rubygems’  
require ‘net/http’  
require ‘open-uri’  
require ‘mechanize’  
require ‘iconv’  
story =  
Hpricot(open(‘[http://thainews.prd.go.th/newsenglish/previewnews.php?news\_id=255108040023](http://thainews.prd.go.th/newsenglish/previewnews.php?news_id=255108040023)’  
))  
body = story.search(“//font[@color="#333333"]”).inner\_html  
body = body.gsub(/\<(.|\n)+?\>/, “”)  
body = body.gsub(/’/, “'”)  
puts body

---

<div class="post-metadata">

**Author:** ![sajal](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/96bed5/32.png) [@sajal](https://www.ruby-forum.com/u/sajal)\
**Post date:** [August 4, 2008, 11:17am UTC](https://www.ruby-forum.com/t/handling-special-characters/144912/3 "2008-08-04T11:17:30Z")

</div>

Hi,

2008/8/4 Sajal K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> [http://thainews.prd.go.th/newsenglish/previewnews.php?news\_id=255108040023](http://thainews.prd.go.th/newsenglish/previewnews.php?news_id=255108040023)  
> body = body.gsub(/\<(.|\n)+?\>/, “”)  
> require ‘iconv’
> 
> I am looking for pointers on one of the following.
> 
> 1. how do i replace “'” to “'” ?  
> body = body.gsub(/\222/,“'”)

> or 2) How can I make iconv ignore the “'” ?

The ’ character (0x92) is not in tis-620 but in windows-874 character  
set.

Refer to

> **[TIS 620 Charset](http://www.langbox.com/codeset/tis620.html)**
>
> LangBox International is a company specialized in
> Internationalization and Localization of UNIX applications. Both Character (TTY) and Graphical
> (X11/Motif) interfaces and several languages (Arabic, Farsi, Hebrew, Greek, Cyrillic, Turkish,...

[http://www.microsoft.com/globaldev/reference/sbcs/874.mspx](http://www.microsoft.com/globaldev/reference/sbcs/874.mspx)

Try  
body = Iconv.iconv(“utf-8”, “windows-874”, body).join

Regards,

Park H.

---

<div class="post-metadata">

**Author:** ![sajal](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/96bed5/32.png) [@sajal](https://www.ruby-forum.com/u/sajal)\
**Post date:** [August 4, 2008, 11:57am UTC](https://www.ruby-forum.com/t/handling-special-characters/144912/4 "2008-08-04T11:57:25Z")

</div>

Heesob P. wrote:

> The ’ character (0x92) is not in tis-620 but in windows-874 character  
> set.
> 
> Refer to  
> [TIS 620 Charset](http://www.langbox.com/codeset/tis620.html)  
> [http://www.microsoft.com/globaldev/reference/sbcs/874.mspx](http://www.microsoft.com/globaldev/reference/sbcs/874.mspx)
> 
> Try  
> body = Iconv.iconv(“utf-8”, “windows-874”, body).join
> 
> Regards,
> 
> Park H.

Awesome works like a charm now. Thanks for the prompt response.

Seems like the source site was putting in the wrong html headers.

You saved me from going bald 😃