Win32 ruby1.9 regexp and cyrillic string

Ruby
#1

#coding: utf-8
str2 = “asdfМикимаус”
p str2.encoding #Encoding:UTF-8
p str2.scan /\p{Cyrillic}/ #found all cyrillic charachters
str2.gsub!(/\w/u,’’) #removes only latin characters
puts str2

The question is why /\w/ ignore cyrillic characters?

I have installed latest ruby package from http://rubyinstaller.org/.
Here is my output of ruby -v
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-mingw32]

#2

str2.gsub!(/\w/u,’’) #removes only latin characters

The question is why /\w/ ignore cyrillic characters?

Are cyrillic characters supposed to count as “word characters”? (\w) ?
If so then looks like a bug to me. Ping core.
-rp

#3

Nikolay K. wrote:

#coding: utf-8
str2 = “asdfМикимаус”
p str2.encoding #Encoding:UTF-8
p str2.scan /\p{Cyrillic}/ #found all cyrillic charachters
str2.gsub!(/\w/u,’’) #removes only latin characters
puts str2

The question is why /\w/ ignore cyrillic characters?

I have installed latest ruby package from http://rubyinstaller.org/.
Here is my output of ruby -v
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-mingw32]

http://redmine.ruby-lang.org/issues/show/3181
http://redmine.ruby-lang.org/issues/show/3202

might be related. If you think it’s wrong then bring it up on core.
-rp

#4

Caleb C. wrote:

On 4/27/10, Nikolay K. [email protected] wrote:

#coding: utf-8
str2 = “asdfМикимаус”
p str2.encoding #Encoding:UTF-8
p str2.scan /\p{Cyrillic}/ #found all cyrillic charachters
str2.gsub!(/\w/u,’’) #removes only latin characters
puts str2

The question is why /\w/ ignore cyrillic characters?

I think that \w (and similar shortcuts) is supposed to match ascii
characters only… thus it’s equivalent to [a-zA-Z].

Isn’t there some kind of unicode character class you can use?

Actually “asdfМикимаус”.gsub!(/\w/u,’’) returns “” on linux so the
problem is from the windows package.

you can use “asdfМикимаус”.gsub!(/\p{L}/,’’) to remove letters thought

#5

Actually “asdfМикимаус”.gsub!(/\w/u,’’) returns “” on linux so the
problem is from the windows package.

you can use “asdfМикимаус”.gsub!(/\p{L}/,’’) to remove letters thought

If they’re the same version then it might be a window bug. Try it with
trunk and if it still fails then submit a bug report to the tracker…

#6

On 4/27/10, Nikolay K. [email protected] wrote:

#coding: utf-8
str2 = “asdfМикимаус”
p str2.encoding #Encoding:UTF-8
p str2.scan /\p{Cyrillic}/ #found all cyrillic charachters
str2.gsub!(/\w/u,’’) #removes only latin characters
puts str2

The question is why /\w/ ignore cyrillic characters?

I think that \w (and similar shortcuts) is supposed to match ascii
characters only… thus it’s equivalent to [a-zA-Z].

Isn’t there some kind of unicode character class you can use?

#7

Roger P. wrote:

Actually “asdfМикимаус”.gsub!(/\w/u,’’) returns “” on linux so the
problem is from the windows package.

Here’s a copy of trunk if that would be useful:

http://rubydoc.ruby-forum.com/ruby_distros/ruby_trunk_no_patches_installed.7z

GL.
-rp