Forum: Ruby String problem

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Fresh M. (Guest)
on 2009-05-03 12:34
What wrong?

# irb
irb(main):001:0> xxx = "лошадь"
=> "\320\273\320\276\321\210\320\260\320\264\321\214"
irb(main):002:0> xxx.length
=> 12
Tom C. (Guest)
on 2009-05-03 13:27
(Received via mailing list)
Fresh M. wrote:
> What wrong?
>
> # irb
> irb(main):001:0> xxx = "лошадь"
> => "\320\273\320\276\321\210\320\260\320\264\321\214"
> irb(main):002:0> xxx.length
> => 12
>
I assume you're wondering why each character appears to be represented
by two bytes - and I believe it's because the encoding is, of necessity,
UTF-8 or something very similar. If I recall correctly, this encoding is
designed to be able to represent the world's alphabets, etc., rather
than merely the limited character set used in western European
languages, and so two bytes must be used to allow for all the
possibilities.

If I don't have this quite right (or right at all), I'm sure I'll be set
right by those who know more here (and they are legion!).

t.


--

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tom C., MS MA, LMHC - Private practice Psychotherapist
Bellingham, Washington, U.S.A: (360) 920-1226
<< removed_email_address@domain.invalid >> (email)
<< TomCloyd.com >> (website)
<< sleightmind.wordpress.com >> (mental health weblog)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Robert K. (Guest)
on 2009-05-03 14:55
(Received via mailing list)
On 03.05.2009 11:26, Tom C. wrote:
> by two bytes - and I believe it's because the encoding is, of necessity,
> UTF-8 or something very similar. If I recall correctly, this encoding is
> designed to be able to represent the world's alphabets, etc., rather
> than merely the limited character set used in western European
> languages, and so two bytes must be used to allow for all the possibilities.
>
> If I don't have this quite right (or right at all), I'm sure I'll be set
> right by those who know more here (and they are legion!).

Actually I do not call myself in when it comes to encodings in Ruby.
But I believe there is one important bit of information missing that's
needed to properly answer the OP's question: what Ruby version did you
use?

Kind regards

  robert
Fresh M. (Guest)
on 2009-05-03 15:01
Robert K. wrote:
>  what Ruby version did you
> use?


$ ruby -v
ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]
7stud -. (Guest)
on 2009-05-03 15:21
Fresh M. wrote:
>
> # irb
> irb(main):001:0> xxx = "лошадь"
> => "\320\273\320\276\321\210\320\260\320\264\321\214"
> irb(main):002:0> xxx.length
> => 12
>
> What wrong?


In 1.8.* versions, ruby doesn't recognize unicode, where characters are
represented by multiple bytes.  ruby thinks everything in an ascii
character where characters are represented by one byte.

Try this:

xxx = "лошадь"
puts xxx.length

--output:--
12

$KCODE = "u"
require 'jcode'

puts xxx.jlength

--output:--
6

xxx.each_char do |u|
  puts u
end

--output:--
л
о
ш
а
д
ь
7stud -. (Guest)
on 2009-05-03 15:23
7stud -- wrote:
>
> In 1.8.* versions, ruby doesn't recognize unicode, where characters are
> represented by multiple bytes.  ruby thinks everything in an ascii
> character where characters are represented by one byte.
>

Corrections:

In 1.8.* versions, ruby doesn't recognize unicode, where characters [may
be]
represented by multiple bytes.  ruby thinks everything [is] an ascii
character where characters are represented by one byte.
7stud -. (Guest)
on 2009-05-03 15:27
Robert K. wrote:
>
> But I believe there is one important bit of information missing that's
> needed to properly answer the OP's question: what Ruby version did you
> use?
>

Why is that relevant?  Can unicode be switched off in ruby 1.9?
Robert K. (Guest)
on 2009-05-03 16:30
(Received via mailing list)
On 03.05.2009 13:27, 7stud -- wrote:
> Robert K. wrote:
>> But I believe there is one important bit of information missing that's
>> needed to properly answer the OP's question: what Ruby version did you
>> use?
>
> Why is that relevant?  Can unicode be switched off in ruby 1.9?

It is relevant because handling of encodings has significantly changed
between 1.8 and 1.9, which I believe your other posting demonstrates.

Cheers

  robert
Yukihiro M. (Guest)
on 2009-05-04 03:00
(Received via mailing list)
Hi,

In message "Re: String problem"
    on Sun, 3 May 2009 20:23:49 +0900, 7stud -- 
<removed_email_address@domain.invalid>
writes:

|Corrections:
|
|In 1.8.* versions, ruby doesn't recognize unicode, where characters [may
|be]
|represented by multiple bytes.  ruby thinks everything [is] an ascii
|character where characters are represented by one byte.

More Corrections:

In 1.8.* versions, string methods of ruby doesn't recognize multi-byte
characters.  ruby thinks everything is a sequence of bytes.  Regular
expressions in 1.8.* recognize UTF-8, EUC-JP, and Shift_JIS.  So you
can handle Unicode strings by using regular expressions.

              matz.
James G. (Guest)
on 2009-05-04 07:13
(Received via mailing list)
On May 3, 2009, at 5:59 PM, Yukihiro M. wrote:

> |be]
> |represented by multiple bytes.  ruby thinks everything [is] an ascii
> |character where characters are represented by one byte.
>
> More Corrections:
>
> In 1.8.* versions, string methods of ruby doesn't recognize multi-byte
> characters.  ruby thinks everything is a sequence of bytes.  Regular
> expressions in 1.8.* recognize UTF-8, EUC-JP, and Shift_JIS.  So you
> can handle Unicode strings by using regular expressions.

And those interested in how all that works may find this series on my
blog helpful:

http://blog.grayproductions.net/articles/understanding_m17n

James Edward G. II
7stud -. (Guest)
on 2009-05-04 14:10
Yukihiro M. wrote:
> Regular
> expressions in 1.8.* recognize UTF-8, EUC-JP, and Shift_JIS.  So you
> can handle Unicode strings by using regular expressions.

Too vague.


James G. wrote:
>
> And those interested in how all that works may find this series on my
> blog helpful:
>
> http://blog.grayproductions.net/articles/understanding_m17n
>

Excellent website.  <c-word here>

Here is something that is unclear:

----------
To use the jcode library, set $KCODE and then require the library.
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting it (as long as you took my advice and
turned ****them*** on)...

http://blog.grayproductions.net/articles/the_kcode...
---------

In the sentence:

-------
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting it (as long as you took my advice and
turned them on)...
--------

'it' and 'them' are pronouns, which should refer to nouns.  The pronoun
'it' looks like it might refer to 'jcode' when 'it' actually refers to
'$KCODE'.   That is pretty easy to sort out.

However, what does 'them' refer to?  'them' should refer to a plural
noun, so if you actually stop and try to sort it out rather than just
dismissing the whole paragraph in confusion, 'them' looks like it must
refer to '$Kcode' and 'jcode'.  However, that doesn't make sense because
you don't 'set' jcode--you require jcode.

Apparently, 'them' refers to 'warning', which is not only grammatically
incorrect but it is very hard to make that association.  In any case, in
that sentence if you change 'it' and 'them' to $KCODE and 'warnings'
respectively, you will change a confusing and unreadable sentence into a
sentence whose clarity will be unmatched in modern literature:

-----
Setting $KCODE first is important, and you will receive a warning if you
require jcode without setting $KCODE (as long as you took my advice and
turned warnings on with -w)...
______

I'd bet that 90% of the readers of your article stop reading at that
exact spot.
James G. (Guest)
on 2009-05-04 19:58
(Received via mailing list)
On May 4, 2009, at 5:10 AM, 7stud -- wrote:

> I'd bet that 90% of the readers of your article stop reading at that
> exact spot.

I think that's dramatically overstating the problem.

I do really appreciate your feedback though.  Obviously I want this
content to be as helpful to everyone as possible.  I've adjusted the
sentence as you recommend.  Thanks.

James Edward G. II
Adam G. (Guest)
on 2009-05-04 20:02
James G. wrote:
> On May 4, 2009, at 5:10 AM, 7stud -- wrote:
>
>> I'd bet that 90% of the readers of your article stop reading at that
>> exact spot.
>
> I think that's dramatically overstating the problem.
>
> I do really appreciate your feedback though.  Obviously I want this
> content to be as helpful to everyone as possible.  I've adjusted the
> sentence as you recommend.  Thanks.
>
> James Edward G. II

Aw, you think so? I kinda liked the idea that, since I understood it
immediately, I was in the top 10% of all Ruby programmers, themselves
already a smart bunch. ;)
7stud -. (Guest)
on 2009-05-04 20:50
James G. wrote:
> I've adjusted the
> sentence as you recommend.  Thanks.
>

Ahh..pure poetry.  I didn't know about the 'u' flag for a regex.  Thank
you.
This topic is locked and can not be replied to.