Forum: Ruby Parsing Japanese Language and Some Ruby Trivia

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Michael S. (Guest)
on 2006-01-11 17:12
(Received via mailing list)
All this talk about Unicode support and HTML parsing got me to
wondering about how to parse Japanese text.  There are no spaces to
separate words, and though there are some modifiers, or particles in
the Japanese language they are used sometime inconsistently.  I could
quote examples, but of you can't read Kanji, Hiragana, and Katakana
they would most likely be meaningless.

So, knowing what little I do of Japanese (been studying for a while
and living in Japan for close to four years), I was wondering how
search engines like Google and Yahoo parse Japanese text, much less
web pages.  There are numerous filters to extract text from web
pages, but parsing Japanese text is another matter altogether.

So, I have found one Open Source project which seems to be addressing
this, but I was wondering if there is a solution for Ruby?

Now for the trivia...  I've been reading some Japanese text,
"Hiragana Times" - a magazine which prints their articles in Japanese
and English as a learning tool and my newspaper "The Japan Times"
which has a weekly section devoted to bilingual education, as well as
my class textbooks.  I've also read some Manga as well.  They
generally present the Kanji with tiny Hiragana characters either
above them which are the phonetic equivalent to the Kanji.

Guess what these tiny Hiragana helpers are called... you guessed it
"Ruby Annotation".  Check out what I found on W3C, either click on
the link or: http://www.w3.org/TR/ruby/

Coincidence?

Mike

--
Mobile: +81-80-3202-2599
Office: +81-3-3395-6055

"Any sufficiently advanced technology is indistinguishable from
magic..."
                                   - A. C. Clarke
David V. (Guest)
on 2006-01-11 18:18
(Received via mailing list)
Michael S. wrote:

> [snip]
>
> Coincidence?

Absolutely. I suspect the term and concept of "Ruby annotation" to be a
lot older than the programming language, and AFAIK, the name of the Ruby
programming language is a reference to its roots in Perl. The fact the
gemstone used as the language is ruby might, but doesn't have to be,
intentionally, subconsciously, or coinkydinkally related to Ruby
annotation. If you want to know for certain, submit Matz to regression
hypnosis and take him back to the time he was deciding for a name of the
language.

David V.
John Fry (Guest)
on 2006-01-11 18:33
(Received via mailing list)
Michael S. <removed_email_address@domain.invalid> writes:

> I was wondering how search engines like Google and Yahoo parse
> Japanese text, much less web pages.  There are numerous filters to
> extract text from web pages, but parsing Japanese text is another
> matter altogether.

I'm not sure what you mean by "parsing", but if you mean segmentation
and morphological analysis of Japanese, then two popular packages for
doing this are ChaSen and MeCab.

Best,

John
Gene T. (Guest)
on 2006-01-11 18:45
(Received via mailing list)
David V. wrote:
> intentionally, subconsciously, or coinkydinkally related to Ruby
> annotation. If you want to know for certain, submit Matz to regression
> hypnosis and take him back to the time he was deciding for a name of the
> language.
>
> David V.

[going OT] i've noted that google has gotten much better at separating
hits of Sam Ruby's pages from pages that refer to ruby language.  Must
be all that python programming they're donig ;-p
Michael S. (Guest)
on 2006-01-12 03:24
(Received via mailing list)
Hi,

I posted this last night and probably didn't hit the correct
audience.  I got one relevant answer and need to go check the
packages recommended.  But to those in Asian Time zones, I'll ask
again about parsing Japanese text.

And to clarify, I am looking for a way to extract "words" from the
text for cataloging in a database.

Cheers,
Mike

--
Mobile: +81-80-3202-2599
Office: +81-3-3395-6055

"The two most common elements in the universe are hydrogen... and
stupidity."
                                -  Harlan Ellison


Begin forwarded message:
Mauricio F. (Guest)
on 2006-01-12 03:52
(Received via mailing list)
On Thu, Jan 12, 2006 at 10:23:11AM +0900, Michael S. wrote:
> Hi,
>
> I posted this last night and probably didn't hit the correct
> audience.  I got one relevant answer and need to go check the
> packages recommended.  But to those in Asian Time zones, I'll ask
> again about parsing Japanese text.
>
> And to clarify, I am looking for a way to extract "words" from the
> text for cataloging in a database.

http://raa.ruby-lang.org/cache/ruby-chasen/
-> It looks old and the source code is not very enticing, though.

This is probably better:
http://chasen.org/~taku/software/mecab/bindings.html


I wrote something similar to what you want long ago; for some reason I
ended
up parsing the output of mecab instead of using the bindings (can't
remember
why atm., nor if it still applies).

The following (old, ugly) code collects some words (names, verbs, "i
adjectives")
from a utf8 string held in 'text':

    text = Iconv.iconv("eucjp", "utf-8", text).first
    temp = Tempfile.new "jphints"
    temp.puts text
    temp.close
    analysis = `mecab #{temp.path}`
    output = Iconv.iconv("utf-8", "eucjp", analysis).first
    temp.close!
    hints = []
    output.each_line do |line|
        break if /\AEOS\s$/u.match line
        hint, nature, canonical =
/\A(.+)\s+([^,]+),[^,]+,[^,]+,[^,]+,[^,]+,[^,]+,([^,]+),/u.match(line).captures
# UGH
        case nature
        when %w[e5 90 8d e8 a9 9e].map{|x| x.to_i(16)}.pack("c*"), #
name
            %w[e5 8b 95 e8 a9 9e].map{|x| x.to_i(16)}.pack("c*"), # verb
            %w[ e5 bd a2 e5 ae b9 e8 a9 9e].map{|x|
x.to_i(16)}.pack("c*") # adj i
            puts "REG HINT #{hint} -> #{canonical}\t #{nature}"
            hints << canonical
        else
            # puts "IGNORED #{hint}"
        end
    end

    # now the words are in hints, as utf8 strings



Hope this helps.
Michael S. (Guest)
on 2006-01-12 08:50
(Received via mailing list)
Thanks, this looks like what I'm looking for.

Cheers,

Mike

--
Mobile: +81-80-3202-2599
Office: +81-3-3395-6055

"Haggis... uh, I was briefed on haggis.... No!"
         G W Bush (dubya) - Japan Times, 12 July 2005
Horacio S. (Guest)
on 2006-01-12 09:36
(Received via mailing list)
To delimit words in a japanese text you can use MeCab and/or Kakasi
(google
them).

Kakasi has ruby bindings
http://raa.ruby-lang.org/project/ruby-kakasi/

MeCab also has bindings for several scripting languages (Ruby included):
http://chasen.org/~taku/software/mecab/bindings.html

If you want database text search support for japanese you can use
Tsearch2
with Teramoto SQL function that uses Kakasi to index japanese words.

http://www.sai.msu.su/~megera/postgres/gist/tsearc...

Finally MSSQL text search function works with Japanese (with MSSQL
Japanese
version of course).


Hope this helps....

Horacio

Thursday 12 January 2006 15:48ã?Michael S. ã?ã??はæ?¸ãã¾ã?ã?:
Brylie O. (Guest)
on 2009-07-18 12:38
Attachment: tinyJPsamplecorpus2.txt (0 Bytes)
Hi,
I have a similar task. I have attached a text file with Japanese
characters. Some of the words are fake and made to look similar to
actually occurring words, as an experiment in linguistics.

Essentially what we are trying to accomplish is to get a count of every
time one of the hiragana characters occurs adjacent to another hiragana
character (including itself) within the context of a word.  The majority
of the data in the file is frequency counts for the word over the course
of several years, the numbers can be ignored for the purposes of this
count.  At this point we are only concerned with the co-occurance of
hiragana characters, but katakana and kanji may eventually be useful.

I currently have Ruby 1.8.7 installed and when I paste the characters
into irb they print out as bytes and I am not sure where to begin
figuring out how to write an effective regexp.
--Brylie
Heesob P. (Guest)
on 2009-07-18 15:04
(Received via mailing list)
Hi,

2009/7/18 Brylie O. <removed_email_address@domain.invalid>:
> count.  At this point we are only concerned with the co-occurance of
> hiragana characters, but katakana and kanji may eventually be useful.
>
> I currently have Ruby 1.8.7 installed and when I paste the characters
> into irb they print out as bytes and I am not sure where to begin
> figuring out how to write an effective regexp.


According to your attachment, I guess you want to handle Shift_JIS
encoded text.

Here are some clues.

str.scan(/\w+/s)                    # => scan for word including
Japanese
str.scan(/./s)                      # => scan for character including
Japanese
str.scan(/\x82[\x9f-\xf1]/)         # => scan for one hiragana character
str.scan(/\x83[\x40-\x96]/)         # => scan for one katakana character
str.scan(/[\x88-\xee][\x40-\xfc]/)  # => scan for one kanji character
(roughly)
str.scan(/(?:\x82[\x9f-\xf1])+/)    # => scan for two or more hiragana
characters

Refer to
http://blog.grayproductions.net/articles/bytes_and...
http://www.rikai.com/library/kanjitables/kanji_cod...

Regards,

Park H.
Harry K. (Guest)
on 2009-07-18 16:32
(Received via mailing list)
> Essentially what we are trying to accomplish is to get a count of every
> time one of the hiragana characters occurs adjacent to another hiragana
> character (including itself) within the context of a word.
>

I'm not sure what you want if there are more than 2 hiragana.
For example, this "$B$i$7$D$E$1$k(B"
or this
["$B$i(B", "$B$7(B"]
["$B$7(B", "$B$D(B"]
["$B$D(B", "$B$E(B"]
["$B$E(B", "$B$1(B"]
["$B$1(B", "$B$k(B"]

Don't trust this too much. I'm just playing around a bit. Maybe it
will give you some ideas.

$KCODE = 'u'
p str.scan(/[$B$"(B-$B$s(B]{2,}/)

OR

require 'enumerator'
$KCODE = 'u'
str.scan(/[$B$"(B-$B$s(B]{2,}/).each {|x|
x.split(//).each_cons(2){|a| p a}}


Harry
Brylie O. (Guest)
on 2009-07-18 20:52
Harry K. wrote:

> I'm not sure what you want if there are more than 2 hiragana.
I think that a string, or array, of each duple from the string of 2 or
more hiragana would be desired. We want to count each occurrence of the
hiragana duples as a running tally through a large document.

There will be duplicates so perhaps each hiragana duple could be a
variable? e.g. "暮ら+= 1" whenever a new instance of 暮ら is found. Would
there be a more efficient way of counting the co-occurrences?


> $KCODE = 'u'
> p str.scan(/[$B$"(B-$B$s(B]{2,}/)
>
> OR
>
> require 'enumerator'
> $KCODE = 'u'
> str.scan(/[$B$"(B-$B$s(B]{2,}/).each {|x|
> x.split(//).each_cons(2){|a| p a}}
> Harry

Also, I don't think that the encoding is Unicode. I have opened the
document in OO.o and JEdit using the Shift-JIS (as well as 'Apple
Macintosh' and 'Windows-932') encoding(s) and the characters seem to
render correctly. I am unsure how to convert this text to Unicode for
proper analysis in Ruby.
--Brylie
Brylie O. (Guest)
on 2009-07-18 21:01
Heesob P. wrote:
> According to your attachment, I guess you want to handle Shift_JIS
> encoded text.
>
> Refer to
> http://blog.grayproductions.net/articles/bytes_and...
> http://www.rikai.com/library/kanjitables/kanji_cod...
>
> Regards,
>
> Park H.

Thank you Park! You're right, it is Shift-JIS. I'll review those links
right away.
--Brylie
Harry K. (Guest)
on 2009-07-19 08:46
(Received via mailing list)
>
> There will be duplicates so perhaps each hiragana duple could be a
> variable? e.g. "$BJk$i(B+= 1" whenever a new instance of $BJk$i(B is found. Would
> there be a more efficient way of counting the co-occurrences?
>
>

I guess I'm missing something.
It seems to me that you are trying to find groups of 2 or more
hiragana and calling each group a word.
But, many words contain no hiragana at all.
And a group of consecutive hiragana could be 1 word, 2 words, or
several words, or no word.
Some words are just 1 hiragana.

I don't know about the statistics you are working on. I guess you do.
Are you just interested in finding patterns without concern for the word
status?

If I just stated the obvious, sorry for the noise.

Harry
James G. (Guest)
on 2009-07-21 15:38
(Received via mailing list)
On Jul 18, 2009, at 11:52 AM, Brylie O. wrote:

> Also, I don't think that the encoding is Unicode. I have opened the
> document in OO.o and JEdit using the Shift-JIS (as well as 'Apple
> Macintosh' and 'Windows-932') encoding(s) and the characters seem to
> render correctly. I am unsure how to convert this text to Unicode for
> proper analysis in Ruby.

Perhaps this would help:

http://blog.grayproductions.net/articles/encoding_...

James Edward G. II
This topic is locked and can not be replied to.