What wrong?
irb
irb(main):001:0> xxx = “лошадь”
=> “\320\273\320\276\321\210\320\260\320\264\321\214”
irb(main):002:0> xxx.length
=> 12
Fresh M. wrote:
What wrong?
irb
irb(main):001:0> xxx = “лошадь”
=> “\320\273\320\276\321\210\320\260\320\264\321\214”
irb(main):002:0> xxx.length
=> 12
I assume you’re wondering why each character appears to be represented
by two bytes - and I believe it’s because the encoding is, of necessity,
UTF-8 or something very similar. If I recall correctly, this encoding is
designed to be able to represent the world’s alphabets, etc., rather
than merely the limited character set used in western European
languages, and so two bytes must be used to allow for all the
possibilities.
If I don’t have this quite right (or right at all), I’m sure I’ll be set
right by those who know more here (and they are legion!).
t.
–
Tom C., MS MA, LMHC - Private practice Psychotherapist
Bellingham, Washington, U.S.A: (360) 920-1226
<< [email protected] >> (email)
<< TomCloyd.com >> (website)
<< sleightmind.wordpress.com >> (mental health weblog)
On 03.05.2009 11:26, Tom C. wrote:
by two bytes - and I believe it’s because the encoding is, of necessity,
UTF-8 or something very similar. If I recall correctly, this encoding is
designed to be able to represent the world’s alphabets, etc., rather
than merely the limited character set used in western European
languages, and so two bytes must be used to allow for all the possibilities.If I don’t have this quite right (or right at all), I’m sure I’ll be set
right by those who know more here (and they are legion!).
Actually I do not call myself in when it comes to encodings in Ruby.
But I believe there is one important bit of information missing that’s
needed to properly answer the OP’s question: what Ruby version did you
use?
Kind regards
robert
Robert K. wrote:
what Ruby version did you
use?
$ ruby -v
ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]
7stud – wrote:
In 1.8.* versions, ruby doesn’t recognize unicode, where characters are
represented by multiple bytes. ruby thinks everything in an ascii
character where characters are represented by one byte.
Corrections:
In 1.8.* versions, ruby doesn’t recognize unicode, where characters [may
be]
represented by multiple bytes. ruby thinks everything [is] an ascii
character where characters are represented by one byte.
Fresh M. wrote:
irb
irb(main):001:0> xxx = “лошадь”
=> “\320\273\320\276\321\210\320\260\320\264\321\214”
irb(main):002:0> xxx.length
=> 12What wrong?
In 1.8.* versions, ruby doesn’t recognize unicode, where characters are
represented by multiple bytes. ruby thinks everything in an ascii
character where characters are represented by one byte.
Try this:
xxx = “лошадь”
puts xxx.length
–output:–
12
$KCODE = “u”
require ‘jcode’
puts xxx.jlength
–output:–
6
xxx.each_char do |u|
puts u
end
–output:–
л
о
ш
а
д
ь
Robert K. wrote:
But I believe there is one important bit of information missing that’s
needed to properly answer the OP’s question: what Ruby version did you
use?
Why is that relevant? Can unicode be switched off in ruby 1.9?
On 03.05.2009 13:27, 7stud – wrote:
Robert K. wrote:
But I believe there is one important bit of information missing that’s
needed to properly answer the OP’s question: what Ruby version did you
use?Why is that relevant? Can unicode be switched off in ruby 1.9?
It is relevant because handling of encodings has significantly changed
between 1.8 and 1.9, which I believe your other posting demonstrates.
Cheers
robert
Hi,
In message “Re: String problem”
on Sun, 3 May 2009 20:23:49 +0900, 7stud – [email protected]
writes:
|Corrections:
|
|In 1.8.* versions, ruby doesn’t recognize unicode, where characters [may
|be]
|represented by multiple bytes. ruby thinks everything [is] an ascii
|character where characters are represented by one byte.
More Corrections:
In 1.8.* versions, string methods of ruby doesn’t recognize multi-byte
characters. ruby thinks everything is a sequence of bytes. Regular
expressions in 1.8.* recognize UTF-8, EUC-JP, and Shift_JIS. So you
can handle Unicode strings by using regular expressions.
matz.
On May 3, 2009, at 5:59 PM, Yukihiro M. wrote:
|be]
|represented by multiple bytes. ruby thinks everything [is] an ascii
|character where characters are represented by one byte.More Corrections:
In 1.8.* versions, string methods of ruby doesn’t recognize multi-byte
characters. ruby thinks everything is a sequence of bytes. Regular
expressions in 1.8.* recognize UTF-8, EUC-JP, and Shift_JIS. So you
can handle Unicode strings by using regular expressions.
And those interested in how all that works may find this series on my
blog helpful:
http://blog.grayproductions.net/articles/understanding_m17n
James Edward G. II
On May 4, 2009, at 5:10 AM, 7stud – wrote:
I’d bet that 90% of the readers of your article stop reading at that
exact spot.
I think that’s dramatically overstating the problem.
I do really appreciate your feedback though. Obviously I want this
content to be as helpful to everyone as possible. I’ve adjusted the
sentence as you recommend. Thanks.
James Edward G. II
Yukihiro M. wrote:
Regular
expressions in 1.8.* recognize UTF-8, EUC-JP, and Shift_JIS. So you
can handle Unicode strings by using regular expressions.
Too vague.
James G. wrote:
And those interested in how all that works may find this series on my
blog helpful:
Excellent website.
Here is something that is unclear:
To use the jcode library, set $KCODE and then require the library.
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting it (as long as you took my advice and
turned *them on)…
http://blog.grayproductions.net/articles/the_kcode_variable_and_jcode_library
In the sentence:
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting it (as long as you took my advice and
turned them on)…
‘it’ and ‘them’ are pronouns, which should refer to nouns. The pronoun
‘it’ looks like it might refer to ‘jcode’ when ‘it’ actually refers to
‘$KCODE’. That is pretty easy to sort out.
However, what does ‘them’ refer to? ‘them’ should refer to a plural
noun, so if you actually stop and try to sort it out rather than just
dismissing the whole paragraph in confusion, ‘them’ looks like it must
refer to ‘$Kcode’ and ‘jcode’. However, that doesn’t make sense because
you don’t ‘set’ jcode–you require jcode.
Apparently, ‘them’ refers to ‘warning’, which is not only grammatically
incorrect but it is very hard to make that association. In any case, in
that sentence if you change ‘it’ and ‘them’ to $KCODE and ‘warnings’
respectively, you will change a confusing and unreadable sentence into a
sentence whose clarity will be unmatched in modern literature:
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting $KCODE (as long as you took my advice and
turned warnings on with -w)…
I’d bet that 90% of the readers of your article stop reading at that
exact spot.
James G. wrote:
I’ve adjusted the
sentence as you recommend. Thanks.
Ahh…pure poetry. I didn’t know about the ‘u’ flag for a regex. Thank
you.
James G. wrote:
On May 4, 2009, at 5:10 AM, 7stud – wrote:
I’d bet that 90% of the readers of your article stop reading at that
exact spot.I think that’s dramatically overstating the problem.
I do really appreciate your feedback though. Obviously I want this
content to be as helpful to everyone as possible. I’ve adjusted the
sentence as you recommend. Thanks.James Edward G. II
Aw, you think so? I kinda liked the idea that, since I understood it
immediately, I was in the top 10% of all Ruby programmers, themselves
already a smart bunch. 