# Unicode in Regex

**URL:** <https://www.ruby-forum.com/t/unicode-in-regex/121786>\
**Category:** Ruby\
**Created:** [November 30, 2007, 9:18pm UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786 "2007-11-30T21:18:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [November 30, 2007, 9:18pm UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/1 "2007-11-30T21:18:05Z")

</div>

This is mostly a Ruby thing, and partly a Rails thing.

I’m expecting a validate\_format\_of with a regex like this

/^[a-zA-Z\xC0-\xD6\xD9-\xF6\xF9-\xFF.’-\]\*?$/

to allow many of the normal characters like Ã¶ Ã© Ã¥ to be submitted via  
web form.

However, the extended characters are being rejected.

This works just fine though (which is just a-zA-Z)

/^[\x41-\x5A\x61-\x7A.’-\]\*?$/

It also seems to fail with full \x0000 numbers, is there limit at \xFF?

Some plain Ruby tests seem to suggest unicode characters don’t work at  
all??

p ‘abvHgtwHFuG’.scan(/[a-z]/)  
p ‘abvHgtwHFuG’.scan(/[A-Z]/)  
p ‘abvHgtwHFuG’.scan(/[\x41-\x5A]/)  
p ‘abvHgtwHFuG’.scan(/[\x61-\x7A]/)  
p ‘aÃ©bvHÃ¶gtÃ¥wHÃ…FuG’.scan(/[\xC0-\xD6\xD9-\xF6\xF9-\xFF]/)

[“a”, “b”, “v”, “g”, “t”, “w”, “u”]  
[“H”, “H”, “F”, “G”]  
[“H”, “H”, “F”, “G”]  
[“a”, “b”, “v”, “g”, “t”, “w”, “u”]  
["\303", “\303”, “\303”, “\303”]

So, what’s the secret to using unicode character ranges in Ruby regex  
(or Rails validations)?

–  
def gw  
acts\_as\_n00b  
writes\_at(www.railsdev.ws)  
end

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [November 30, 2007, 10:10pm UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/2 "2007-11-30T22:10:58Z")

</div>

On Nov 30, 2:18 pm, Greg W. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> So, what’s the secret to using unicode character ranges in Ruby regex  
> (or Rails validations)?

Tim B. gave a great talk about I18N, M17N and Unicode at the 2006  
Ruby Conference. His presentation can be found at:

> **[rubyconf2006.pdf](https://www.tbray.org/talks/rubyconf2006.pdf)**
>
> 3.96 MB

He described how many member functions have trouble dealing with these  
character sets. He made special reference to regular expressions.

–Dale

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [November 30, 2007, 11:01pm UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/3 "2007-11-30T23:01:02Z")

</div>

Dale M. wrote:

> On Nov 30, 2:18 pm, Greg W. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:
> 
> > So, what’s the secret to using unicode character ranges in Ruby regex  
> > (or Rails validations)?
> 
> Tim B. gave a great talk about I18N, M17N and Unicode at the 2006  
> Ruby Conference. His presentation can be found at:
> 
> [http://www.tbray.org/talks/rubyconf2006.pdf](http://www.tbray.org/talks/rubyconf2006.pdf)
> 
> He described how many member functions have trouble dealing with these  
> character sets. He made special reference to regular expressions.

That’s just beyond sad.

I’ve been using Lasso for several years now, and _2003_ it provided  
complete support for Unicode. I know there’s some esoterics it may not  
deal with, but for all practical purposes we can round-trip data in  
western and eastern languages with Lasso quite easily.

How can all these other languages be so far behind?

Pretty bad if I can’t even allow Mr. MuÃ±os or GÃ¶ran to enter their names  
in a web form with proper server side validations. Aargh.

– gw

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 1, 2007, 6:25am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/4 "2007-12-01T06:25:47Z")

</div>

On Nov 30, 4:00 pm, Greg W. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> How can all these other languages be so far behind?
> 
> Pretty bad if I can’t even allow Mr. Muños or Göran to enter their names  
> in a web form with proper server side validations. Aargh.
> 
> ## – gw
> 
> Posted viahttp://www.ruby-forum.com/.

Ruby 1.8 doesn’t have unicode support (1.9 is starting to get it).  
Everything in ruby is a bytestring.

irb(main):001:0\> ‘aébvHögtåwHÅFuG’.scan(/./)  
=\> [“a”, “\303”, “\251”, “b”, “v”, “H”, “\303”, “\266”, “g”, “t”,  
“\303”, “\245”, “w”, “H”, “\303”, “\205”, “F”, “u”, “G”]

So your character class is matching the first byte of the composite  
characters (which is \303 in octal), and skipping the next (since it’s  
below the range). You probably want something like…

reg = /[\xc0-\xd6\xd9-\xf6\xf9-\xff][\x80-\xbc]/  
‘aébvHögtåwHÅFuG’.scan(reg)

irb(main):006:0\* reg = /[\xc0-\xd6\xd9-\xf6\xf9-\xff][\x80-\xbc]/  
=\> /[\xc0-\xd6\xd9-\xf6\xf9-\xff][\x80-\xbc]/  
irb(main):007:0\> ‘aébvHögtåwHÅFuG’.scan(reg)  
=\> [“\303\251”, “\303\266”, “\303\245”, “\303\205”]  
irb(main):008:0\> “å” == “\303\245”  
=\> true

Ps. I’m not entirely sure the value of the second character class is  
right.

Regards,  
Jordan

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 1, 2007, 11:16am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/5 "2007-12-01T11:16:48Z")

</div>

> Unicode in Regex  
> Posted by Greg W. (-gw-) on 30.11.2007 21:18  
> This is mostly a Ruby thing, and partly a Rails thing.
> 
> I’m expecting a validate\_format\_of with a regex like this
> 
> /^[a-zA-Z\xC0-\xD6\xD9-\xF6\xF9-\xFF.'-\]\*?$/
> 
> to allow many of the normal characters like Ã¶ Ã© Ã¥ to be submitted via  
> web form.

How about the utf8 validation regex here:  
[http://snippets.dzone.com/posts/show/4527](http://snippets.dzone.com/posts/show/4527) ?

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 3, 2007, 2:19am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/6 "2007-12-03T02:19:11Z")

</div>

MonkeeSage wrote:

> Ruby 1.8 doesn’t have unicode support (1.9 is starting to get it).

I enrages me to see this kind of FUD. Through regular expressions, ruby  
1.8 has 80-90% complete utf8 support. And oniguruma makes utf8 support  
well-near 100% complete.

> > ‘aÃ©bvHÃ¶gtÃ¥wHÃ…FuG’.scan(/./u)  
> > =\> [“a”, “Ã©”, “b”, “v”, “H”, “Ã¶”, “g”, “t”, “Ã¥”, “w”, “H”, “Ã…”, “F”,  
> > “u”, “G”]

> > ‘aÃ©bvHÃ¶gtÃ¥wHÃ…FuG’.scan(/[Ã©Ã¶Ã¥Ã…]/u)  
> > =\> [“Ã©”, “Ã¶”, “Ã¥”, “Ã…”]

Ok, sometimes you have to take a weird approach because of the missing  
10-20%, but it’s still workable

> > ‘aÃ©bvHÃ¶gtÃ¥wHÃ…FuG’.scan(/(?:\303\251|\303\266|\303\245|\303\205)/u)  
> > =\> [“Ã©”, “Ã¶”, “Ã¥”, “Ã…”]

> Everything in ruby is a bytestring.

YES! And that’s exactyly how it should be. Who is it that spread the  
flawed idea that strings are fundamentally made of characters? I’d like  
to slap him around a little. Fundamentally, ever since the word “string”  
was applied to computing, strings were made of 8-BIT CHARS, not n-bit  
characters. If only the creators of C has called that datatype “byte”  
instead of “char” it would have saved us so many misunderstandings.

Usually the complaint about the support lack of unicode support is that  
something like “æ—¥æœ¬èªž”.length returns 9 instead of 3, or that "æ—¥æœ¬èªž  
".index(“èªž”) returns 6 instead of 2. It’s nice that people want to  
completely redefine the API to return character positions and all that,  
but please don’t complain that it’s broken just because you happen to be  
using it incorrectly. Use the right tool for the job. SQL for database  
queries, non-home-brewed crypto libraries for security, regular  
expressions for string manipulation.

I’m terribly sorry for the rant but I had to get it off my chest.

Dan

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 3, 2007, 2:41am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/7 "2007-12-03T02:41:26Z")

</div>

Greg W. wrote:

> characters. I have discovered that I can get a little further by doing
> 
> But… this fails /^[a-zA-Z\xE4-\xE6]\*?&/u
> 
> But… this works /^[a-zA-Z\xE4\xE5\xE6]\*?&/u
> 
> I’ve boiled the experiments down to realizing I can’t define a range  
> with \x
> 
> Is this just one of those things that just doesn’t work yet WRT Ruby/  
> Rails/UTF8, or is there another syntax? I’ve scoured all the regex  
> docs I can find, and they seem to indicate a range should work.

Let me try to explain that in order to redeem myself from my previous  
angry post.

Basically, \xE4 is counted as the byte value 0xE4, not the unicode  
character U+00E4. And in a range expression, each escaped value is taken  
as one character within the range. Which results in not-immediately  
obvious situations:

> > ‘aÃ©bvHÃ¶gtÃ¥wHÃ…FuG’.scan(/[\303\251]/u)  
> > =\> []
> 
> > ‘aÃ©bvHÃ¶gtÃ¥wHÃ…FuG’.scan(/[#{"\303\251"}]/u)  
> > =\> [“Ã©”]

What is happening in the first case is that the string does not contain  
characters \303 or \251 because those are invalid utf8 sequences. But  
when the value “\303\251” is _inlined_ into the regex, that is  
recognized as the utf8 character “Ã©” and a match is found.

So ranges _do_ work in utf8 but you have to be careful:

> > “Ã Ã¢Ã¤Ã§Ã¨Ã©ÃªÃ®Ã¯Ã´Ã¼”.scan(/[Ã¤-Ã®]/u)  
> > =\> [“Ã¤”, “Ã§”, “Ã¨”, “Ã©”, “Ãª”, “Ã®”]
> 
> > “Ã Ã¢Ã¤Ã§Ã¨Ã©ÃªÃ®Ã¯Ã´Ã¼”.scan(/[\303\244-\303\256]/u)  
> > =\> ["\303", “\303”, “\303”, “\244”, “\303”, “\247”, “\303”, “\250”,  
> > “\303”, “\251”, “\303”, “\252”, “\303”, “\256”, “\303”, “\257”, “\303”,  
> > “\264”, “\303”, “\274”]
> 
> > “Ã Ã¢Ã¤Ã§Ã¨Ã©ÃªÃ®Ã¯Ã´Ã¼”.scan(/[#{"\303\244-\303\256"}]/u)  
> > =\> [“Ã¤”, “Ã§”, “Ã¨”, “Ã©”, “Ãª”, “Ã®”]

Hope this helps.

Dan

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 2, 2007, 9:35pm UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/8 "2007-12-02T21:35:37Z")

</div>

Greg W. wrote:

> I’m expecting a validate\_format\_of with a regex like this  
> /^[a-zA-Z\xC0-\xD6\xD9-\xF6\xF9-\xFF.’-\]\*?$/  
> to allow many of the normal characters like Ã¶ Ã© Ã¥ to be submitted via  
> web form. However, the extended characters are being rejected.

So, I’ve been pounding the web for info on UTF8 in Ruby and Rails the  
past couple days to concoct some validations that allow UTF8  
characters. I have discovered that I can get a little further by doing  
the  
following:

- declaring $KCODE = ‘UTF8’
- adding /u to regex expressions.

The only thing not working now is the ability to define a range of \x  
characters in a regex.

So, this /^[a-zA-Z\xE4]\*?&/u will validate that a string is allowed  
to have an Ã¤ in it. Perfect.

But… this fails /^[a-zA-Z\xE4-\xE6]\*?&/u

But… this works /^[a-zA-Z\xE4\xE5\xE6]\*?&/u

I’ve boiled the experiments down to realizing I can’t define a range  
with \x

Is this just one of those things that just doesn’t work yet WRT Ruby/  
Rails/UTF8, or is there another syntax? I’ve scoured all the regex  
docs I can find, and they seem to indicate a range should work.

For now, I just have all the characters I want included \< \xFF listed  
individually.

utf\_accents = ‘\xC0\xC1\xC2…’

Is\_person\_name = /^[a-zA-Z#{utf\_accents}.’-\]\*?$/u

But I’d like to solve the range notation if I can.

–  
def gw  
acts\_as\_n00b  
writes\_at(www.railsdev.ws)  
end

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 3, 2007, 2:56am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/9 "2007-12-03T02:56:38Z")

</div>

Greg W. wrote:

> So, this /^[a-zA-Z\xE4]\*?&/u will validate that a string is allowed  
> to have an Ã¤ in it. Perfect.

If that actually works, it means you are really using ISO-8859-1  
strings, not UTF-8.

> utf\_accents = ‘\xC0\xC1\xC2…’

Nope, that’s not UTF-8. UTF-8 characters Ã€ÃÃ‚ would look like  
utf\_accents = “\xC3\x80\xC3\x81\xC3\x82…”

Dan

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 3, 2007, 2:50am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/10 "2007-12-03T02:50:47Z")

</div>

On Dec 2, 2:35 pm, Greg W. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> following:
> 
> ## individually. writes\_at(www.railsdev.ws) end
> 
> Posted viahttp://www.ruby-forum.com/.

This seems to work…

$KCODE = “UTF8”  
p /^[a-zA-Z\xC0-\xD6\xD9-\xF6\xF9-\xFF.'-\]\*?/u =~ “Jäsp…it  
works”

# =\> 0

However, it looks to me like it would be more robust to use a slightly  
modified version of UTF8REGEX (found in the link Jimmy posted  
above)…

UTF8REGEX = /\A(?:  
[a-zA-Z.-'\]  
| [\xC2-\xDF][\x80-\xBF]  
| \xE0[\xA0-\xBF][\x80-\xBF]  
| [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}  
| \xED[\x80-\x9F][\x80-\xBF]  
| \xF0[\x90-\xBF][\x80-\xBF]{2}  
| [\xF1-\xF3][\x80-\xBF]{3}  
| \xF4[\x80-\x8F][\x80-\xBF]{2}  
)\*\z/mnx

p UTF8REGEX =~ “Jäsp…it works here too”

# =\> 0

Look at the link to see the explanation of the alternations.

Regards,  
Jordan

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 3, 2007, 8:47pm UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/11 "2007-12-03T20:47:05Z")

</div>

Daniel DeLorme wrote:

> Greg W. wrote:

> > But… this fails /^[a-zA-Z\xE4-\xE6]_?&/u  
> > But… this works /^[a-zA-Z\xE4\xE5\xE6]_?&/u
> > 
> > I’ve boiled the experiments down to realizing I can’t define a range  
> > with \x

> Let me try to explain that in order to redeem myself from my previous  
> angry post.

🙂

> Basically, \xE4 is counted as the byte value 0xE4, not the unicode  
> character U+00E4. And in a range expression, each escaped value is taken  
> as one character within the range. Which results in not-immediately  
> obvious situations:
> 
> > > ‘aÃ©bvHÃ¶gtÃ¥wHÃ…FuG’.scan(/[\303\251]/u)  
> > > =\> []
> > 
> > > ‘aÃ©bvHÃ¶gtÃ¥wHÃ…FuG’.scan(/[#{"\303\251"}]/u)  
> > > =\> [“Ã©”]

OK, I see oniguruma docs refer to \x as encoded byte value and \x{} as a  
character code point – which with your explanation I can finally tie  
together what that means.

Took me a second to recognize the #{} as Ruby and not some new regex I’d  
never seen 😛

And I realize now too I wasn’t picking up on the use of octal vs  
decimal.

Seems like Ruby doesn’t like to use the hex \x{7HHHHHHH} variant?

> What is happening in the first case is that the string does not contain  
> characters \303 or \251 because those are invalid utf8 sequences. But  
> when the value “\303\251” is _inlined_ into the regex, that is  
> recognized as the utf8 character “Ã©” and a match is found.
> 
> So ranges _do_ work in utf8 but you have to be careful:
> 
> > > “Ã Ã¢Ã¤Ã§Ã¨Ã©ÃªÃ®Ã¯Ã´Ã¼”.scan(/[Ã¤-Ã®]/u)  
> > > =\> [“Ã¤”, “Ã§”, “Ã¨”, “Ã©”, “Ãª”, “Ã®”]
> > 
> > > “Ã Ã¢Ã¤Ã§Ã¨Ã©ÃªÃ®Ã¯Ã´Ã¼”.scan(/[\303\244-\303\256]/u)  
> > > =\> ["\303", “\303”, “\303”, “\244”, “\303”, “\247”, “\303”, “\250”,  
> > > “\303”, “\251”, “\303”, “\252”, “\303”, “\256”, “\303”, “\257”, “\303”,  
> > > “\264”, “\303”, “\274”]
> > 
> > > “Ã Ã¢Ã¤Ã§Ã¨Ã©ÃªÃ®Ã¯Ã´Ã¼”.scan(/[#{"\303\244-\303\256"}]/u)  
> > > =\> [“Ã¤”, “Ã§”, “Ã¨”, “Ã©”, “Ãª”, “Ã®”]
> 
> Hope this helps.

Yes!

– gw

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 3, 2007, 10:07pm UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/12 "2007-12-03T22:07:59Z")

</div>

> > Basically, \xE4 is counted as the byte value 0xE4, not the unicode  
> > character U+00E4. And in a range expression, each escaped value is taken  
> > as one character within the range. Which results in not-immediately  
> > obvious situations:
> > 
> > > > ‘aÃ©bvHÃ¶gtÃ¥wHÃ…FuG’.scan(/[\303\251]/u)  
> > > > =\> []
> > > 
> > > > ‘aÃ©bvHÃ¶gtÃ¥wHÃ…FuG’.scan(/[#{"\303\251"}]/u)  
> > > > =\> [“Ã©”]

OK, one thing I’m still confused about – when I look up Ã© in any table,  
it’s DEC is 233 which converted to OCT is 351, yet you’re using 251 (and  
indeed it seems like reducing the OCTs I come up with by 100 is what  
actually works).

Where is this 100 difference coming from?

– gw

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 3, 2007, 10:20pm UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/13 "2007-12-03T22:20:49Z")

</div>

On Dec 3, 2:07 pm, Greg W. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Where is this 100 difference coming from?  
> [Unicode Character 'LATIN SMALL LETTER E WITH ACUTE' (U+00E9)](http://www.fileformat.info/info/unicode/char/00e9/index.htm)

The UTF-16 value is 233 (decimal), but the UTF-8 value is 0xC3 0xA9,  
which is 195 169 in decimal, or 0303 0251 in octal.

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 4, 2007, 1:32am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/14 "2007-12-04T01:32:07Z")

</div>

Daniel DeLorme wrote:

> Usually the complaint about the support lack of unicode support is that  
> something like “æ—¥æœ¬èªž”.length returns 9 instead of 3, or that "æ—¥æœ¬èªž  
> ".index(“èªž”) returns 6 instead of 2. It’s nice that people want to  
> completely redefine the API to return character positions and all that,  
> but please don’t complain that it’s broken just because you happen to be  
> using it incorrectly. Use the right tool for the job. SQL for database  
> queries, non-home-brewed crypto libraries for security, regular  
> expressions for string manipulation.
> 
> I’m terribly sorry for the rant but I had to get it off my chest.

Regular expressions for all character work would be a _terribly_ slow  
way to get things done. If you want to get the nth character, should you  
do a match for n-1 characters and a group to grab the nth? Or would it  
be better if you could just index into the string and have it do the  
right thing? How about if you want to iterate over all characters in a  
string? Should the iterating code have to know about the encoding?  
Should you use a regex to peel off one character at a time? Absurd.

Regex for string access goes a long way, but’s just about the heaviest  
way to do it. Strings should be aware of their encoding and should be  
able to provide you access to characters as easily as bytes. That’s what  
1.9 (and upcoming changes in JRuby) fixes.

- Charlie

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 4, 2007, 1:45am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/15 "2007-12-04T01:45:34Z")

</div>

On Dec 2, 7:40 pm, Daniel DeLorme [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > characters. I have discovered that I can get a little further by doing
> 
> > > ‘aébvHögtåwHÅFuG’.scan(/[#{“\303\251”}]/u)  
> > > =\> [“ä”, “ç”, “è”, “é”, “ê”, “î”]  
> > > “àâäçèéêîïôü”.scan(/[\303\244-\303\256]/u)  
> > > =\> [“\303”, “\303”, “\303”, “\244”, “\303”, “\247”, “\303”, “\250”,  
> > > “\303”, “\251”, “\303”, “\252”, “\303”, “\256”, “\303”, “\257”, “\303”,  
> > > “\264”, “\303”, “\274”]  
> > > “àâäçèéêîïôü”.scan(/[#{“\303\244-\303\256”}]/u)  
> > > =\> [“ä”, “ç”, “è”, “é”, “ê”, “î”]
> 
> Hope this helps.
> 
> Dan

I missed your ranting.

Firstly, ruby doesn’t have unicode support in 1.8, since unicode _IS_  
a standard mapping of bytes to _characters_. That’s what unicode is.  
I’m sorry you don’t like that, but don’t lie and say ruby 1.8 supports  
unicode when it knows nothing about that standard mapping and treats  
everything as individual bytes (and any byte with a value greater than  
126 just prints an octal escape); and please don’t accuse others of  
spreading FUD when they state the facts.

Secondly, as I said in my first post to this thread, the characters  
trying to be matched are composite characters, which requires you to  
match both bytes. You can try to using a unicode regexp, but then you  
run into the problem you mention–the regexp engine expects the pre-  
composed, one-byte form…

“ò”.scan(/[\303\262]/u) # =\> []  
“ò”.scan(/[\xf2]/u) # =\> [“\303\262”]

…which is why I said it’s more robust to use something like the the  
regexp that Jimmy linked to and I reposted, instead of a unicode  
regexp.

Regards,  
Jordan

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 3, 2007, 11:21pm UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/16 "2007-12-03T23:21:08Z")

</div>

On Dec 3, 12:47 pm, Greg W. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 4, 2007, 1:50am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/17 "2007-12-04T01:50:50Z")

</div>

On Dec 3, 1:47 pm, Greg W. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> 🙂
> 
> Seems like Ruby doesn’t like to use the hex \x{7HHHHHHH} variant?  
> Oniguruma is not in ruby 1.8 (though you can install it as a gem). It  
> is in 1.9.

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 4, 2007, 9:48am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/18 "2007-12-04T09:48:02Z")

</div>

Charles Oliver N. wrote:

> Regular expressions for all character work would be a _terribly_ slow  
> way to get things done. If you want to get the nth character, should you  
> do a match for n-1 characters and a group to grab the nth? Or would it  
> be better if you could just index into the string and have it do the

Ok, I’m not very familiar with the internal working of strings in 1.9,  
but it seems to me that for character sets with variable byte size, it  
is logically _impossible_ to directly index into the string. Unless  
there’s some trick I’m unaware of, you _have_ to count from the  
beginning of the string for utf8 strings.

> right thing? How about if you want to iterate over all characters in a  
> string? Should the iterating code have to know about the encoding?  
> Should you use a regex to peel off one character at a time?

That is certainly one possible way of doing things…  
string.scan(/./){ |char| do\_someting\_with(char) }

> Regex for string access goes a long way, but’s just about the heaviest  
> way to do it.

Heavy compared to what? Once compiled, regex are orders of magnitude  
faster than jumping in and out of ruby interpreted code.

> Strings should be aware of their encoding and should be  
> able to provide you access to characters as easily as bytes. That’s what  
> 1.9 (and upcoming changes in JRuby) fixes.

Overall I agree that the encoding stuff in 1.9 is very nice.  
Encapsulating the encoding with the string is very OO. Very intuitive.  
No need to think about encoding anymore, now it “just works” for  
encoding-ignorant programmers (at least until the abstraction leaks). It  
allows to shut up one frequent complaint about ruby; a clear political  
victory. Overall it is more robust and less error-prone than the 1.8  
way.

But my point was that there _is_ a 1.8 way. The thing that riled me up  
and that I was responding to was the claim that 1.8 did not have unicode  
support AT ALL. Unequivocally, it does, and it works pretty well for me.  
IMHO there is a certain minimalist elegance in considering strings as  
encoding-agnostic and using regex to get encoding-specific views. I  
could do str[/./n] and str[/./u]; I can’t do that anymore.

1.9 makes encodings easier for the english-speaking masses not used to  
extended characters, but let’s remember that ruby _always_ had support  
for multibyte character sets; after all it _did_ originate from a  
country with two gazillion “characters”.

Daniel

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 4, 2007, 10:08am UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/19 "2007-12-04T10:08:03Z")

</div>

MonkeeSage wrote:

> Firstly, ruby doesn’t have unicode support in 1.8, since unicode _IS_  
> a standard mapping of bytes to _characters_. That’s what unicode is.  
> I’m sorry you don’t like that, but don’t lie and say ruby 1.8 supports  
> unicode when it knows nothing about that standard mapping and treats  
> everything as individual bytes (and any byte with a value greater than  
> 126 just prints an octal escape)

Ok, then how do you explain this:

> > $KCODE=‘u’  
> > =\> “u”
> 
> > “abc\303\244”.scan(/./)  
> > =\> [“a”, “b”, “c”, “ä”]

This doesn’t require any libraries, and it seems to my eyes that ruby is  
converting 5 bytes into 4 characters. It shows an awareness of utf8. If  
that’s not _some_ kind of unicode support then please tell me what it  
is. It seem were disagreeing on some basic definition of what “unicode  
support” means.

> Secondly, as I said in my first post to this thread, the characters  
> trying to be matched are composite characters, which requires you to  
> match both bytes. You can try to using a unicode regexp, but then you  
> run into the problem you mention–the regexp engine expects the pre-  
> composed, one-byte form…
> 
> “ò”.scan(/[\303\262]/u) # =\> []  
> “ò”.scan(/[\xf2]/u) # =\> ["\303\262"]

Wow, I never knew that second one could work. Unicode support is  
actually better than I thought! You learn something new every day.

> …which is why I said it’s more robust to use something like the the  
> regexp that Jimmy linked to and I reposted, instead of a unicode  
> regexp.

I’m not sure what makes that huge regexp more robust than a simple  
unicode regexp.

Daniel

---

<div class="post-metadata">

**Author:** ![S-gw](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@S-gw](https://www.ruby-forum.com/u/S-gw)\
**Post date:** [December 4, 2007, 1:32pm UTC](https://www.ruby-forum.com/t/unicode-in-regex/121786/20 "2007-12-04T13:32:46Z")

</div>

On Dec 4, 3:07 am, Daniel DeLorme [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> support" means.  
> I guess we were talking about different things then. I never meant to  
> imply that the regexp engine can’t match unicode characters (it’s  
> “dumb” implementation though; it basically only knows that bytes above  
> 127 can have more bytes following and should be grouped together as  
> candidates for a match; that’s slightly simplified, but basically  
> accurate).

I, like Charles (and I think most people), was referring to the  
ability to index into strings by characters, find their lengths in  
characters, to compose and decompose composite characters, to  
normalize characters, convert them to other encodings like shift-jis,  
and other such things. Ruby 1.9 has started adding such support, while  
ruby 1.8 lacks it. It can be hacked together with regular expressions  
(e.g., the link Jimmy posted), or even as a real, compiled extension  
[1], but merely saying that _you_ the programmer can implement it  
using ruby 1.8, is not the same thing as saying ruby 1.8 supports it  
(just like I could build a python VM in ruby, but that doesn’t mean  
that the ruby interpreter runs python bytecode). Anyhow, I guess it’s  
just a difference of opinion. I don’t mind being wrong (happens a  
lot! 😉 I just don’t like being accused of spreading FUD about ruby,  
which to my mind implies malice of forethought rather that simply  
mistake.

[1] [http://rubyforge.org/projects/char-encodings/](http://rubyforge.org/projects/char-encodings/)  
[http://git.bitwi.se/ruby-character-encodings.git/](http://git.bitwi.se/ruby-character-encodings.git/)

> actually better than I thought! You learn something new every day.
> 
> > …which is why I said it’s more robust to use something like the the  
> > regexp that Jimmy linked to and I reposted, instead of a unicode  
> > regexp.
> 
> I’m not sure what makes that huge regexp more robust than a simple  
> unicode regexp.
> 
> Daniel

Well, I won’t claim that you can’t get a unicode regexp to match the  
same. And I only saw that large regexp when it was posted here, so  
I’ve not tested it to any great length. Interestingly, 1.9 uses this  
regexp (originally from jcode.rb in stdlib) to classify a string as  
containing utf-8: ‘[\xc0-\xdf][\x80-\xbf]|[\xe0-\xef][\x80-\xbf][\x80-  
\xbf]’. My thought was that without knowing all of the minute  
intricacies of unicode and how ruby strings and regexps work with  
unicode values (which I don’t, and assume the OP doesn’t), I think the  
huge regexp is more likely to Just Work in more cases than a home-  
brewed unicode regexp. But like I said, that’s just an initial  
conclusion, I don’t claim it’s absolutely correct.

Regards,  
Jordan

[Next page](https://www.ruby-forum.com/t/unicode-in-regex/121786.md?page=2)
