# Forcing a string to valid UTF-8

**URL:** <https://www.ruby-forum.com/t/forcing-a-string-to-valid-utf-8/187629>\
**Category:** Ruby\
**Created:** [April 27, 2010, 12:45am UTC](https://www.ruby-forum.com/t/forcing-a-string-to-valid-utf-8/187629 "2010-04-27T00:45:48Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![phrogz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/2acd7d/32.png) [@phrogz](https://www.ruby-forum.com/u/phrogz)\
**Post date:** [April 27, 2010, 12:45am UTC](https://www.ruby-forum.com/t/forcing-a-string-to-valid-utf-8/187629/1 "2010-04-27T00:45:48Z")

</div>

I have some legacy text data that’s gone through several databases and  
web services in its life, playing promiscuously with dirty web  
servers, browsers, and encodings.

It’s coming out of the source database as ASCII-8bit. I’m trying to  
bring it all into UTF-8. I’ve found ways to coerce many of the bad  
entries into compliance, but now I’ve hit one that is simply bad. I  
want to just delete the minimum necessary to make it valid UTF-8. What  
I’m trying isn’t working. Here’s my code:

if new\_value.is\_a? String  
begin  
utf8 = new\_value.force\_encoding(‘UTF-8’)  
if utf8.valid\_encoding?  
new\_value = utf8  
else  
new\_value.encode!( ‘UTF-8’, ‘Windows-1252’ )  
end  
rescue EncodingError =\> e  
puts “Bad encoding: #{old\_table}.#{pk}:#{old\_row[pk]} -  
#{new\_value.inspect}”  
new\_value.encode!( ‘UTF-8’, invalid: :replace, undef: :replace,  
replace: ‘’ )  
p new\_value.encoding unless new\_value.valid\_encoding?  
end  
end

When I fall into the rescue clause, I’m getting out:  
Bad encoding: bugs.id:2469 - “Indexing C:\\コピ\xE3\x81E \x81E  
\x81EZCa\_zu5.264”  
#Encoding:UTF-8  
The conversion resulted in an invalid UTF-8 string (that happens to be  
the same as the original, as far as I can tell.) I’m surprised,  
because I thought the purpose of invalid/undef replace was to clean  
things up.

How do I force it into a valid UTF-8 encoding, losing as little data  
as possible but happily throwing out the senseless bits?

---

<div class="post-metadata">

**Author:** ![phrogz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/2acd7d/32.png) [@phrogz](https://www.ruby-forum.com/u/phrogz)\
**Post date:** [April 27, 2010, 12:19pm UTC](https://www.ruby-forum.com/t/forcing-a-string-to-valid-utf-8/187629/2 "2010-04-27T12:19:13Z")

</div>

Gavin K. wrote:

> How do I force it into a valid UTF-8 encoding, losing as little data  
> as possible but happily throwing out the senseless bits?

AFAICS, the trouble with your rescue clause is that the string failed to  
be encoded into Windows-1252, so it remains with its existing UTF-8 tag,  
and so an attempt to “re-encode” as UTF-8 is silently ignored because  
it’s already UTF-8, even though it contains invalid characters.

For example, this doesn’t do anything:

> > a = “abc\xffdef”.force\_encoding(“UTF-8”)  
> > =\> “abc\xFFdef”
> 
> > b = a.encode(“UTF-8”, :invalid=\>:replace, :replace=\>"?")  
> > =\> “abc\xFFdef”

but this does:

> > b = a.encode(“UTF-16BE”, :invalid=\>:replace, :replace=\>"?").encode(“UTF-8”)  
> > =\> “abc?def”

Proviso: ruby 1.9 string handling is undocumented and subject to  
continuous change. I tested the above with

> > RUBY\_DESCRIPTION  
> > =\> “ruby 1.9.2dev (2009-07-18 trunk 24186) [i686-linux]”

so it may or may not work with your version, or with future versions of  
Ruby.

---

<div class="post-metadata">

**Author:** ![phrogz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/2acd7d/32.png) [@phrogz](https://www.ruby-forum.com/u/phrogz)\
**Post date:** [April 27, 2010, 4:45pm UTC](https://www.ruby-forum.com/t/forcing-a-string-to-valid-utf-8/187629/3 "2010-04-27T16:45:51Z")

</div>

On Apr 27, 4:19 am, Brian C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Gavin K. wrote:
> 
> > How do I force it into a valid UTF-8 encoding, losing as little data  
> > as possible but happily throwing out the senseless bits?
> 
> AFAICS, the trouble with your rescue clause is that the string failed to  
> be encoded into Windows-1252, so it remains with its existing UTF-8 tag,  
> and so an attempt to “re-encode” as UTF-8 is silently ignored because  
> it’s already UTF-8, even though it contains invalid characters.

Excellent point. Fixing that led me to a similar error earlier: I had  
assumed that  
s2 = s1.force\_encoding(…)  
left s1 intact. In fact, it modifies and returns s1. Thank you very  
much, Brian.

For those that care or stumble upon this via Google, here’s a modified  
version that works:

# Converting ASCII-8BIT to UTF-8 based domain-specific guesses

if new\_value.is\_a? String  
begin  
# Try it as UTF-8 directly  
cleaned = new\_value.dup.force\_encoding(‘UTF-8’)  
unless cleaned.valid\_encoding?  
# Some of it might be old Windows code page  
cleaned = new\_value.encode( ‘UTF-8’, ‘Windows-1252’ )  
end  
new\_value = cleaned  
rescue EncodingError  
# Force it to UTF-8, throwing out invalid bits  
new\_value.encode!( ‘UTF-8’, invalid: :replace, undef: :replace )  
end  
end

> Proviso: ruby 1.9 string handling is undocumented and subject to  
> continuous change. I tested the above with

FWIW my new code works on ruby 1.9.1p243 (2009-07-16 revision 24175)  
[i386-mingw32]

Thanks again!