# How can I get malformed UTF-8 characters to display properly

**URL:** <https://www.ruby-forum.com/t/how-can-i-get-malformed-utf-8-characters-to-display-properly/113095>\
**Category:** Rails\
**Created:** [August 27, 2007, 4:39pm UTC](https://www.ruby-forum.com/t/how-can-i-get-malformed-utf-8-characters-to-display-properly/113095 "2007-08-27T16:39:44Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [August 27, 2007, 4:39pm UTC](https://www.ruby-forum.com/t/how-can-i-get-malformed-utf-8-characters-to-display-properly/113095/1 "2007-08-27T16:39:44Z")

</div>

Hello everyone,

I’m scraping a lot of sites for a project, and occasionally the  
scraped content will have “malformed UTF-8” characters. When the  
scraped content is processed (basically a database record is created),  
these characters often don’t appear as they’re supposed to.

Normally, the following code works great:

str.unpack(“U\*”).collect {|s| (s \> 127 ? “&##{s};” : s.chr) }.join(“”)

But it won’t work with these “malformed UTF-8” characters. So I’ve  
written the following to handle these characters, but it still isn’t  
perfect. For example, I scraped this page:  
[http://web.mac.com/j3mbeck/iWeb/JohnBeckPaper\_Steel/Fireplace%20Surrounds.html](http://web.mac.com/j3mbeck/iWeb/JohnBeckPaper_Steel/Fireplace%20Surrounds.html)

The alt attribute of the first thumbnail, steel surround, contains the  
text “Steel has that effect where you’d least expect it”. The ’  
character shows up as Õ when I use the method below, and the “d” is  
just swallowed.

```
data.gsub!(/\323/, '&quot;')

require 'oniguruma'

o = Oniguruma::ORegexp.new('[^[:ascii:]]')
# o = Oniguruma::ORegexp.new('[^[:ascii:]]', {:encoding =>

```

Oniguruma::ENCODING\_UTF8})  
chars = []  
data.each\_char{|c|chars \<\< c}  
chars.collect do |c|  
if o.match c  
begin  
“&##{c.unpack(‘U\*’).first};”  
rescue ArgumentError  
add\_log\_message(“Has malformed UTF-8 characters”)  
#handling malformed UTF-8 : a huge pain and possibly future  
cause of problems  
bytes = []  
c.each\_byte{|b| bytes \<\< b}  
# assumes we’re handling at most, 2-byte strings. We have  
no way if the malformed character is  
# supposed to be one byte or two, but we’re assuming it’s 1.  
[“&##{bytes[0]}”] + bytes[1…-1].collect{|b|b.chr}  
end  
else  
c  
end  
end.flatten.join(‘’)

Any suggestions?

Thanks!  
Daniel