# How to remove strange characters

**URL:** <https://www.ruby-forum.com/t/how-to-remove-strange-characters/150356>\
**Category:** Ruby\
**Created:** [October 7, 2008, 6:30pm UTC](https://www.ruby-forum.com/t/how-to-remove-strange-characters/150356 "2008-10-07T18:30:42Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex-osu3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/13edae/32.png) [@alex-osu3](https://www.ruby-forum.com/u/alex-osu3)\
**Post date:** [October 7, 2008, 6:30pm UTC](https://www.ruby-forum.com/t/how-to-remove-strange-characters/150356/1 "2008-10-07T18:30:42Z")

</div>

Hi all,

I grap some info from a webpage. Sometimes I get some stranges  
characters as follows (by p):  
To depart in a hurry; abscond: \342\200\234Your horse  
has\nabsquatulated!\342\200\235 (Robert M. Bird) To die.

or (by print):  
To depart in a hurry; abscond: “Your horse has absquatulated!”  
(Robert M. Bird) To die.

Any idea to to get rid of them?

Thanks,

Li

---

<div class="post-metadata">

**Author:** ![alex-osu3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/13edae/32.png) [@alex-osu3](https://www.ruby-forum.com/u/alex-osu3)\
**Post date:** [October 8, 2008, 4:26pm UTC](https://www.ruby-forum.com/t/how-to-remove-strange-characters/150356/2 "2008-10-08T16:26:16Z")

</div>

Stephen C. wrote:

> Those are multi-byte characters (curly quotes, in this case). You  
> probably don’t want to get rid of them, but you can use the iconv  
> library to transliterate them back to their ASCII almost-equivalents:
> 
> > > string = “To depart in a hurry; abscond: \342\200\234Your horse has\nabsquatulated!\342\200\235 (Robert M. Bird) To die.”  
> > > =\> “To depart in a hurry; abscond: \342\200\234Your horse  
> > > has\nabsquatulated!\342\200\235 (Robert M. Bird) To die.”
> > 
> > > require ‘iconv’  
> > > =\> true
> > 
> > > puts Iconv.iconv(‘ascii//translit’, ‘utf-8’, string).to\_s  
> > > To depart in a hurry; abscond: “Your horse has  
> > > absquatulated!” (Robert M. Bird) To die.  
> > > =\> nil
> 
> Stephen

Thank you,

Li

---

<div class="post-metadata">

**Author:** ![alex-osu3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/13edae/32.png) [@alex-osu3](https://www.ruby-forum.com/u/alex-osu3)\
**Post date:** [October 8, 2008, 6:16pm UTC](https://www.ruby-forum.com/t/how-to-remove-strange-characters/150356/3 "2008-10-08T18:16:20Z")

</div>

Hi Stephen and others,

Iconv only works for some characters. It doesn’t work for the following  
scripts.

Any idea?

Thanks,

Li

C:\Users\Alex\>irb  
irb(main):001:0\> require ‘iconv’  
=\> true  
irb(main):002:0\> string1=“Fatal injury or ruin:\223Hath some fond lover  
tic’d thee to thy bane?\224  
\342\200\246”  
=\> “Fatal injury or ruin:\223Hath some fond lover tic’d thee to thy  
bane?\224\342\200\246”  
irb(main):003:0\> puts  
Iconv.iconv(‘ASCII//TRANSLIT’,‘utf-8’,string1).to\_s  
Iconv::IllegalSequence: "\223Hath some fond "…  
from (irb):3:in `iconv’  
from (irb):3  
irb(main):004:0\>

---

<div class="post-metadata">

**Author:** ![alex-osu3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/13edae/32.png) [@alex-osu3](https://www.ruby-forum.com/u/alex-osu3)\
**Post date:** [October 7, 2008, 8:15pm UTC](https://www.ruby-forum.com/t/how-to-remove-strange-characters/150356/4 "2008-10-07T20:15:27Z")

</div>

Hi,

On Tue, Oct 7, 2008 at 11:28 AM, Li Chen [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> I grap some info from a webpage. Sometimes I get some stranges  
> characters as follows (by p):  
> To depart in a hurry; abscond: \342\200\234Your horse  
> has\nabsquatulated!\342\200\235 (Robert M. Bird) To die.
> 
> or (by print):  
> To depart in a hurry; abscond: “Your horse has absquatulated!† 
> (Robert M. Bird) To die.
> 
> Any idea to to get rid of them?

Those are multi-byte characters (curly quotes, in this case). You  
probably don’t want to get rid of them, but you can use the iconv  
library to transliterate them back to their ASCII almost-equivalents:

> > string = “To depart in a hurry; abscond: \342\200\234Your horse has\nabsquatulated!\342\200\235 (Robert M. Bird) To die.”  
> > =\> “To depart in a hurry; abscond: \342\200\234Your horse  
> > has\nabsquatulated!\342\200\235 (Robert M. Bird) To die.”  
> > require ‘iconv’  
> > =\> true  
> > puts Iconv.iconv(‘ascii//translit’, ‘utf-8’, string).to\_s  
> > To depart in a hurry; abscond: “Your horse has  
> > absquatulated!” (Robert M. Bird) To die.  
> > =\> nil

Stephen

---

<div class="post-metadata">

**Author:** ![alex-osu3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/13edae/32.png) [@alex-osu3](https://www.ruby-forum.com/u/alex-osu3)\
**Post date:** [October 9, 2008, 5:47am UTC](https://www.ruby-forum.com/t/how-to-remove-strange-characters/150356/5 "2008-10-09T05:47:03Z")

</div>

Li Chen wrote:

> Hi all,
> 
> I grap some info from a webpage. Sometimes I get some stranges  
> characters as follows (by p):  
> To depart in a hurry; abscond: \342\200\234Your horse  
> has\nabsquatulated!\342\200\235 (Robert M. Bird) To die.

Here’s a quick hack I used recently. It was messing my display on  
ncurses, and I did not need the characters.

dataitem.gsub!(/[^[:space:][:print:]]/,’’)

I got this while googling, iirc, its used somewhere in ROR.

---

<div class="post-metadata">

**Author:** ![alex-osu3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/13edae/32.png) [@alex-osu3](https://www.ruby-forum.com/u/alex-osu3)\
**Post date:** [October 8, 2008, 6:37pm UTC](https://www.ruby-forum.com/t/how-to-remove-strange-characters/150356/6 "2008-10-08T18:37:40Z")

</div>

what do you think doing something like this?

class String  
def remove\_nonascii(replacement)  
n=self.split(“”)  
self.slice!(0…self.size)  
n.each{|b|  
if (b[0].to\_i\< 32 || b[0].to\_i\>124) then  
self.concat(replacement)  
elsif  
[34,35,37,42,43,44,45,47,60,61,62,63,91,92,93,94,96,123].include?(b[0].to\_i)  
self.concat(replacement)  
else  
self.concat(b)  
end  
}  
self.to\_s  
end  
end

“Fatal injury or ruin:\223Hath some fond lover tic’d thee to  
thybane?\224\342\200\246”.remove\_nonascii(‘+’)

=\> “Fatal injury or ruin:+Hath some fond lover tic’d thee to  
thybane+++++”

how you can see, it made the replacement with char ‘+’.

2008/10/8 Li Chen [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)

---

<div class="post-metadata">

**Author:** ![alex-osu3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/13edae/32.png) [@alex-osu3](https://www.ruby-forum.com/u/alex-osu3)\
**Post date:** [October 10, 2008, 2:57am UTC](https://www.ruby-forum.com/t/how-to-remove-strange-characters/150356/7 "2008-10-10T02:57:41Z")

</div>

There is no one-click installer for 1.9 on Windows as far as I can tell.  
Downloading and unpacking the ziped binaries didn’t get me very far as  
both ruby and irb complain that something is missing. Does binary  
distribution require me to install anything else? Like libraries? If  
this is the case what additional stuff do I need to make 1.9 to work and  
where can I get it?

Thanks,  
Alex

---

<div class="post-metadata">

**Author:** ![alex-osu3](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/13edae/32.png) [@alex-osu3](https://www.ruby-forum.com/u/alex-osu3)\
**Post date:** [October 9, 2008, 9:53pm UTC](https://www.ruby-forum.com/t/how-to-remove-strange-characters/150356/8 "2008-10-09T21:53:28Z")

</div>

Nit K. wrote:

> Here’s a quick hack I used recently. It was messing my display on  
> ncurses, and I did not need the characters.
> 
> dataitem.gsub!(/[^[:space:][:print:]]/,’’)
> 
> I got this while googling, iirc, its used somewhere in ROR.

It works on scenario where iconv doesn’t work. Good job!!!

Li