How can I get malformed UTF-8 characters to display properly

Hello everyone,

I’m scraping a lot of sites for a project, and occasionally the
scraped content will have “malformed UTF-8” characters. When the
scraped content is processed (basically a database record is created),
these characters often don’t appear as they’re supposed to.

Normally, the following code works great:

str.unpack(“U*”).collect {|s| (s > 127 ? “&##{s};” : s.chr) }.join("")

But it won’t work with these “malformed UTF-8” characters. So I’ve
written the following to handle these characters, but it still isn’t
perfect. For example, I scraped this page:
http://web.mac.com/j3mbeck/iWeb/JohnBeckPaper_Steel/Fireplace%20Surrounds.html

The alt attribute of the first thumbnail, steel surround, contains the
text “Steel has that effect where you’d least expect it”. The ’
character shows up as Õ when I use the method below, and the “d” is
just swallowed.

data.gsub!(/\323/, '"')

require 'oniguruma'

o = Oniguruma::ORegexp.new('[^[:ascii:]]')
# o = Oniguruma::ORegexp.new('[^[:ascii:]]', {:encoding =>

Oniguruma::ENCODING_UTF8})
chars = []
data.each_char{|c|chars << c}
chars.collect do |c|
if o.match c
begin
“&##{c.unpack(‘U*’).first};”
rescue ArgumentError
add_log_message(“Has malformed UTF-8 characters”)
#handling malformed UTF-8 : a huge pain and possibly future
cause of problems
bytes = []
c.each_byte{|b| bytes << b}
# assumes we’re handling at most, 2-byte strings. We have
no way if the malformed character is
# supposed to be one byte or two, but we’re assuming it’s 1.
["&##{bytes[0]}"] + bytes[1…-1].collect{|b|b.chr}
end
else
c
end
end.flatten.join(’’)

Any suggestions?

Thanks!
Daniel

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs