Has anyone seen a non-english characters library for Ruby walking
around? For now, I need to remove letter decorators (in other words,
‘Ã±’ becomes n and ‘Ã¢’ becomes a) and drop non-alphanumeric
characters (’!etter’ becomes ‘etter’).
I thought iconv transliteration might do this, but it doesn’t:
i = Iconv.new('ascii//TRANSLIT//IGNORE, ‘iso-8859-1’)
A bit of googling turns up the following:
Both work roughly the same way; they use iconv to convert the source
string to UTF-16BE, followed by a mapping table to map accented
characters to their non-accented equivalents.
The unac link above has a bit more information about how these mapping
tables are generated; basically they have a script that parses a unicode
data file at build time and generates the mapping table.
The mapping table is available here:
So anyway, the answers to your question appears to be:
- If you’re just converting a couple of characters one time, just
use a regular expression.
- If you’re looking to convert an arbitrary number of characters one
time and have access to a machine with GNU tools, just use unac.
- If you’re not particular about the language, use the Perl library.
- If you don’t mind installing the unac library, I wrote a quick wrapper
for it. See below for more.
This has been discussed on ruby-talk before:
I wrote a quick binding for the unac library, you can grab it from here:
http://pablotron.org/files/unac-ruby-0.1.0.tar.gz.asc (PGP Signature)
http://hg.pablotron.org/unac-ruby (Mercurial Repository)
If you’re interested, I can probably write a pure-Ruby version
relatively quickly too.