# Re: Premature end of regular expression with non-ascii chara **URL:** \ **Category:** Ruby\ **Created:** [January 31, 2006, 12:27am UTC](https://www.ruby-forum.com/t/re-premature-end-of-regular-expression-with-non-ascii-chara/99145 "2006-01-31T00:27:41Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [January 31, 2006, 12:27am UTC](https://www.ruby-forum.com/t/re-premature-end-of-regular-expression-with-non-ascii-chara/99145/1 "2006-01-31T00:27:41Z")
When I read in a text with accents from a file under cygwin, these get converted to something like ‘\352’. You can then search for these using regexps: a=“un texte extrêmement énervant” p splitted\_text=a.split(/(?=)/) b=/extr\352mement/ d=a.match(b) p d[0] =\> extr\352mement When I write the result to a file, it appears correctly as “extrêmement”. f=File.new(“t.txt”,‘w’) f.puts d[0] f.close Hope that helps, Best regards, Axel ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [January 31, 2006, 12:07pm UTC](https://www.ruby-forum.com/t/re-premature-end-of-regular-expression-with-non-ascii-chara/99145/2 "2006-01-31T12:07:48Z")
Hi Axel, thanks for the reply. If I try your code, my characters with accents don’t get translated to numbers, unfortunately. Do you know where these numbers come from, I looked on the net but \352 is not the octal, hexadecimal or UTF-8 representation of ê . Could you split the following sentence for me and let me know what the result is: a=“Ils sont très énervé les regexps.” splitted\_text=a.split(/\s/) Not my best French. But if I try this, ‘très énervé les’ is still one part, eventhough I split it on the spaces. Maybe it is different with you and then I have to look deeper. Thanks for your help. If anybody is able to split is like ‘très’, ‘énervé’, ‘les’ please let me know!! Kind regards, Nick ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [January 31, 2006, 12:34pm UTC](https://www.ruby-forum.com/t/re-premature-end-of-regular-expression-with-non-ascii-chara/99145/3 "2006-01-31T12:34:57Z")
The odds are your text is in non-UTF8 encoding, but in CP1252 or similar. Then indeed, if $KCODE = ‘u’ split won’t work right. 2006/1/31, Nick S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [January 31, 2006, 1:11pm UTC](https://www.ruby-forum.com/t/re-premature-end-of-regular-expression-with-non-ascii-chara/99145/4 "2006-01-31T13:11:44Z")
Indeed, it isn’t in UTF-8. It’s in ISO-8859-1 (Latin1). The problem here is that I would like to work in UTF-8, but I have to read in files. And these files are often (almost always) in ISO-8859-1. And I haven’t found a way of converting these strings to Unicode in Ruby. é and è etc. form part of ISO-8859-1. Anyway I remove $KCODE altogether in config/environment.rb and now it works. And Axel I also get the numbers. In config/environment.rb I added: $KCODE = ‘u’ require ‘jcode’ to get Gettext to work. So it turns out that if you aren’t fully working in UTF-8, you have to be carefull adding this. Thanks for pointing me to $KCODE, twice! Kind regards, Nick ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [January 31, 2006, 1:23pm UTC](https://www.ruby-forum.com/t/re-premature-end-of-regular-expression-with-non-ascii-chara/99145/5 "2006-01-31T13:23:38Z")
2006/1/31, Nick S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > Indeed, it isn’t in UTF-8. It’s in ISO-8859-1 (Latin1). The problem here > is that I would like to work in UTF-8, but I have to read in files. And > these files are often (almost always) in ISO-8859-1. And I haven’t found > a way of converting these strings to Unicode in Ruby. é and è etc. form > part of ISO-8859-1. use Iconv library ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [January 31, 2006, 2:18pm UTC](https://www.ruby-forum.com/t/re-premature-end-of-regular-expression-with-non-ascii-chara/99145/6 "2006-01-31T14:18:21Z")
Hi Nikolai, thanks for the suggestion I will definitely give Iconv a try. Hope it doesn’t slow things down a lot. Kind regards, Nick ---
**Author:** ![unknown](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/u/4bbf92/32.png) [@unknown](https://www.ruby-forum.com/u/unknown)\ **Post date:** [February 1, 2006, 11:40pm UTC](https://www.ruby-forum.com/t/re-premature-end-of-regular-expression-with-non-ascii-chara/99145/7 "2006-02-01T23:40:26Z")
Nick S. wrote: > Indeed, it isn’t in UTF-8. It’s in ISO-8859-1 (Latin1). The problem here > is that I would like to work in UTF-8, but I have to read in files. And > these files are often (almost always) in ISO-8859-1. And I haven’t found > a way of converting these strings to Unicode in Ruby. é and è etc. form > part of ISO-8859-1. I have to deal with similar problems when processing the infamous german umlaute äöü. My solution has been to convert a string from latin1 or latin15 to utf8 via this utf8\_string=latin1\_string.unpack(“C\*”).pack(“U\*”) and the other way round with latin1\_string=utf8\_string.unpack(“U\*”).pack(“C\*”) Did work so far and does not include changes in the environment. HTH, Lars