Deadlock detection for 1.9

[Yusuke_ENDOH]

é è—¤ã§ã™ã€‚

1.9 のスレッドにデッドロック検出を実装してみました。

$ ./ruby -e ‘Thread.new { Thread.stop }; Thread.stop’
-e:1:in stop': deadlock detected (fatal) from -e:1:in’

ちょっとでかいパッチ (添付) ですが、要旨は以下の通りです。

• スレッドの状態種別に THREAD_STOPPED_FOREVER ã‚’è¿½åŠ ã€‚
  無期限の sleep 状態と mutex 解放待ち状態を表す。

• rb_vm_t に変数 sleeper ã‚’è¿½åŠ ã€‚
  THREAD_STOPPED_FOREVER 状態のスレッドの数を表す。

• rb_thread_t に変数 locking_mutex ã‚’è¿½åŠ ã€‚
  このスレッドが待っている mutex を表す。

• rb_thread_t に変数 keeping_mutexes ã‚’è¿½åŠ ã€‚mutex_t に
  変数 next_mutex ã‚’è¿½åŠ ã€‚
  スレッドがロックしている mutex のリストを表す。

• mutex_t に変数 cond_notified ã‚’è¿½åŠ ã€‚
  cond_signal されてまだ起動していないスレッドの数を表す。

• rb_mutex_lock ã‚„ rb_mutex_unlock で上記の変数たちを
  適宜更新するようにした。

• 唯一動いていそうなスレッドが native_cond_wait しそうな
  時は、lock_func を抜けてデッドロック検査するようにした。

• rb_check_deadlock では vm->living_threads を列挙して、

  • 例外状態のスレッド、または
  • locking_mutex のロックに成功したスレッド、または
  • locking_mutex が誰にもロックされていないスレッド、
    のいずれもなければ main_thread に fatal な例外を投げる
    ようにした。

• スレッドの終了時にロックしていた mutex をすべて解放
  するようにした。

このパッチは元々 redmine の Feature #17 に流したものですが、
ä»¥ä¸‹ã®ä¿®æ­£ã‚’è¿½åŠ ã—ã¦ã„ã¾ã™ã€‚

• thread_win32.c ã®ä¿®æ­£ã‚’è¿½åŠ ã—ãŸ (ただし未テスト)

• Kernel#sleep ã§ãƒ‡ãƒƒãƒ‰ãƒ­ãƒƒã‚¯çŠ¶æ…‹ã«ãªã£ãŸå ´åˆã¯ä¾‹å¤–ã«ã—ãªã„
  ようにした

• バグフィクスとちょっと最適化

制限は以下の通りです。

• スレッドやロックの処理が少し重くなる。
  計測のたびに変わるけれど、だいたいこのくらいです。
  vm3_thread_create_join 6.565 6.586
  vm3_thread_mutex 3.023 3.155

• スレッドがロック中の mutex が GC されなくなる。
  loop { Mutex.new.lock } # どんどんメモリを消費する

• Thread#join 、#stop 、Mutex#lock 以外で固まっている
  スレッドがあるとデッドロック検出しない。
  r, w = IO.pipe; r.read # 固まり続ける

どんなもんでしょうか。

[Yusuke_ENDOH]

e$B$^$D$b$He(B e$B$f$-$R$m$G$9e(B

In message “Re: [ruby-dev:35044] deadlock detection for 1.9”
on Wed, 11 Jun 2008 00:25:26 +0900, “Yusuke ENDOH” [email protected]
writes:

|1.9 e$B$N%9%l%C%I$K%G%C%I%m%C%/8!=P$r<BAu$7$F$_$^$7$?!#e(B

e$B:G=E$K$3$l$r<h$j9~$`$+$I$&$+$NH=CG$O:{ED$/$s$K$*$^$+$;$7$^e(B
e$B$9$,!“%G%C%I%m%C%/$J$I%9%l%C%I4X78$N%H%i%V%k$OLLE]$J$N$G!”;de(B
e$B<+?H$O$3$N%Q%C%A$NF3F~$KBP$7$FA08~$-$G$9!#e(B

e$B$?$@$7!"e(B

| - e$B%9%l%C%I$,%m%C%/Cf$Ne(B mutex e$B$,e(B GC e$B$5$l$J$/$J$k!#e(B
| loop { Mutex.new.lock } # e$B$I$s$I$s%a%b%j$r>CHq$9$ke(B

e$B$3$l$@$1$O$$$?$@$1$^$;$s!#e(Bweak refe$B$J$I$r;H$C$F$J$s$H$+$G$-$Je(B
e$B$$$b$N$G$7$g$&$+$M$'!#e(B

[Yusuke_ENDOH]

é è—¤ã§ã™ã€‚

2008/06/11 9:18 Yukihiro M. [email protected]:

| - スレッドがロック中の mutex が GC されなくなる。
| loop { Mutex.new.lock } # どんどんメモリを消費する

これだけはいただけません。weak refなどを使ってなんとかできな
いものでしょうかねぇ。

なんとかできました。変更点:

• rb_thread_mark で keeping_mutexes ã‚’ mark しないようにした。

• mutex_mark で next_mutex ã‚’ mark しないようにした。

• mutex_free で unlock するようにした。

あと、mutex が lock 状態で放置されるのはおそらくプログラムのバグ
ãªã®ã§è­¦å‘Šã—ãŸæ–¹ãŒã„ã„ã®ã§ã¯ã€ã¨ã•ã•ã ã•ã‚“ã«ã‚³ãƒ¡ãƒ³ãƒˆã‚’é ‚ãã¾ã—ãŸã€‚
これもやってみました。

$ ./ruby -e ‘Thread.new { Mutex.new.lock }.join’
warning: mutex #Mutex:0x818e6f8 remains to be locked by terminated
thread

$ ./ruby -e ‘loop { Mutex.new.lock; GC.start; sleep 1 }’
-e:1: warning: free locked mutex
-e:1: warning: free locked mutex
-e:1: warning: free locked mutex
^C
:16: warning: mutex #Mutex:0x820dc3c remains to be
locked by terminated thread
-e:1:in sleep': Interrupt from -e:1:in block in ’
from -e:1:in loop' from -e:1:in ’

[Yusuke_ENDOH]

e$B!!$5$5$@$G$9!%e(B

Yukihiro M. wrote:

e$B$3$l$G;d$+$i$NCmJ8$O$b$&$"$j$^$;$s!#:{ED$5$s$,e(BOKe$B$J$i%3%_%C%He(B
e$B$7$F$/$@$5$$!#e(B

e$B!!3NG’$9$kM>M5$O$"$j$^$;$s$,!$$-$C$H$($s$I$&$5$s$J$iBg>fIW$@$m$&$H;We(B
e$B$&$N$G!$0[O@$"$j$^$;$s!%e(B

[Yusuke_ENDOH]

e$B$^$D$b$He(B e$B$f$-$R$m$G$9e(B

In message “Re: [ruby-dev:35050] Re: deadlock detection for 1.9”
on Wed, 11 Jun 2008 22:48:46 +0900, “Yusuke ENDOH” [email protected]
writes:

|2008/06/11 9:18 Yukihiro M. [email protected]:
|> | - e$B%9%l%C%I$,%m%C%/Cf$Ne(B mutex e$B$,e(B GC e$B$5$l$J$/$J$k!#e(B
|> | loop { Mutex.new.lock } # e$B$I$s$I$s%a%b%j$r>CHq$9$ke(B
|>
|> e$B$3$l$@$1$O$$$?$@$1$^$;$s!#e(Bweak refe$B$J$I$r;H$C$F$J$s$H$+$G$-$Je(B
|> e$B$$$b$N$G$7$g$&$+$M$'!#e(B
|
|e$B$J$s$H$+$G$-$^$7$?!#JQ99E@e(B:
|
| - rb_thread_mark e$B$Ge(B keeping_mutexes e$B$re(B mark e$B$7$J$$$h$&$K$7$?!#e(B
|
| - mutex_mark e$B$Ge(B next_mutex e$B$re(B mark e$B$7$J$$$h$&$K$7$?!#e(B
|
| - mutex_free e$B$Ge(B unlock e$B$9$k$h$&$K$7$?!#e(B
|
|
|e$B$"$H!“e(Bmutex e$B$,e(B lock e$B>uBV$GJ|CV$5$l$k$N$O$*$=$i$/%W%m%0%i%`$N%P%0e(B
|e$B$J$N$G7Y9p$7$?J}$,$$$$$N$G$O!”$H$5$5$@$5$s$K%3%a%s%H$rD:$-$^$7$?!#e(B
|e$B$3$l$b$d$C$F$_$^$7$?!#e(B
|
|$ ./ruby -e ‘Thread.new { Mutex.new.lock }.join’
|warning: mutex #Mutex:0x818e6f8 remains to be locked by terminated thread

e$B$3$l$G;d$+$i$NCmJ8$O$b$&$"$j$^$;$s!#:{ED$5$s$,e(BOKe$B$J$i%3%_%C%He(B
e$B$7$F$/$@$5$$!#e(B

[Yusuke_ENDOH]

e$B$3$s$P$s$Oe(B sheepman e$B$G$9!#e(B

e$B:Y$+$$OC$G$9$,!“0J2<$N$h$&$J%3!<%I$Ge(B 1.8
e$B$H5sF0$,0[$J$j$^$9!#e(B
1.8 e$B$@$He(B deadlock e$B07$$$G$O$J$/e(B sleep forever
e$B$K$J$j$^$9!#e(B
[ruby-dev:31041] e$B4XO”$NOCBj$G$9!#e(B

Thread.new do
sleep 1
end
Thread.stop

1.8
e$B$H$$$C$7$g$@$H%I%-%e%a%s%H$r=q$/$N$,3Z$G$$$$$J$H$$$&$@$1$J$s$G$9$,!#e(B

[Yusuke_ENDOH]

e$B1sF#$G$9!#e(B

2008/06/12 10:07 SASADA Koichi [email protected]:

Yukihiro M. wrote:

e$B$3$l$G;d$+$i$NCmJ8$O$b$&$"$j$^$;$s!#:{ED$5$s$,e(BOKe$B$J$i%3%_%C%He(B
e$B$7$F$/$@$5$$!#e(B

e$B3NG’$9$kM>M5$O$“$j$^$;$s$,!$$-$C$H$($s$I$&$5$s$J$iBg>fIW$@$m$&$H;We(B
e$B$&$N$G!$0[O@$”$j$^$;$s!%e(B

e$B%3%%C%H$7$^$7$?!#%3%%C%HD>8e$Oe(B win32 e$B4X78$J$I$G$A$g$C$He(B
e$B$4$?$4$?$7$^$7$?$,!":#$OF0$$$F$$$k$H;W$$$^$9!#e(B

e$B$J$+$`$ie(B(e$B$&e(B)e$B$5$s$"$j$,$H$&$4$6$$$^$7$?!#e(B

e$B$“$He(B mutex e$BJ|CV$N7Y9p$J$s$G$9$,!“e(BGC e$BCf$Ke(B rb_warn
e$B$rC!$/$N$Oe(B
e$B$^$:$$$+$b$H8@$&$3$H$G!”:#$O%3%a%s%H%”%&%H$7$F$$$^$9!#8e$Ge(B
e$B9M$($^$9!#e(B

[Yusuke_ENDOH]

e$B$^$D$b$He(B e$B$f$-$R$m$G$9e(B

In message “Re: [ruby-dev:35322] Re: deadlock detection for 1.9”
on Wed, 2 Jul 2008 23:14:39 +0900, sheepman
[email protected] writes:

|e$B:Y$+$$OC$G$9$,!“0J2<$N$h$&$J%3!<%I$Ge(B 1.8 e$B$H5sF0$,0[$J$j$^$9!#e(B
|1.8 e$B$@$He(B deadlock e$B07$$$G$O$J$/e(B sleep forever e$B$K$J$j$^$9!#e(B
|[ruby-dev:31041] e$B4XO”$NOCBj$G$9!#e(B
|
|Thread.new do
| sleep 1
|end
|Thread.stop
|
|1.8 e$B$H$$$C$7$g$@$H%I%-%e%a%s%H$r=q$/$N$,3Z$G$$$$$J$H$$$&$@$1$J$s$G$9$,!#e(B

e$B1sF#$5$s$KJ9$$$F$_$J$$$H!#e(B

[Yusuke_ENDOH]

e$B1sF#$G$9!#e(B

e$B1sF#$5$s$KJ9$$$F$_$J$$$H!#e(B

e$B%7%s%0%k%9%l%C%I$Ge(B Thread.stop
e$B$9$k$HNc30$K$J$k$N$G!"e(BThread.stop e$B$Ge(B
e$B0U?^E*$Ke(B sleep forever
e$B$7$h$&$H$$$&?M$O$$$J$$$h$&$J5$$,$7$^$9!#e(B
e$B$`$7$mNc30$J$I$G0U?^$;$:C10l%9%l%C%I$K$J$C$F$7$^$C$?>l9g$r9M$($k$H!"e(B
e$B>e$NNc$Ge(B deadlock
e$B07$$$7$F$/$l$J$$$N$O$&$l$7$/$J$$$H;W$$$^$9!#e(B

e$B$J$N$G!VC10l%9%l%C%I$G$Oe(B deadlock
e$B8!=P$7$J$$!W$H$$$&4p=`$K$O$"$^$je(B
e$BG<F@$G$-$J$$$N$G$9$,!"0l1~%Q%C%A$G$9!#e(B

Index: thread.c

— thread.c (revision 17818)
+++ thread.c (working copy)
@@ -3515,6 +3515,7 @@
int found = 0;

 if (vm_living_thread_num(vm) > vm->sleeper) return;

• if (vm_living_thread_num(vm) == 1) return;
  if (vm_living_thread_num(vm) < vm->sleeper) rb_bug(“sleeper must
  not be more than vm_living_thread_num(vm)”);

  st_foreach(vm->living_threads, check_deadlock_i,
  (st_data_t)&found);

e$B$"$He(B 1.8 e$B$H5sF0$,0c$&$H$$$($P!#e(B
[ruby-dev:34856] e$B$KJV;v$,$J$+$C$?$N$G$9$,!"e(B1.8
e$B$O0J2<$,=*N;$9$k$N$Ke(B

m = Mutex.new
Thread.new { m.lock }.join
m.lock

e$B0J2<$,e(B sleep forever e$B$K$J$k$N$O$*$+$7$/$J$$$G$7$g$&$+!#e(B

m = Mutex.new
Thread.new { m.lock; sleep 2 }
sleep 1; m.lock

1.9 e$B$G$O2<$NNc$G$b=*N;$9$k$h$&$K$7$F$$$^$9!#e(B
e$B$3$l$re(B 1.8 e$B$K$"$o$;$k$N$OFq$7$=$&$G$9!#e(B

[Yusuke_ENDOH]

e$B$J$+$@$G$9!#e(B

At Thu, 3 Jul 2008 01:45:54 +0900,
Yukihiro M. wrote in [ruby-dev:35331]:

e$B$^$9!#e(B
1.8e$B$@$H%9%l%C%I$,$I$Ne(BMutexe$B$r%m%C%/$7$F$k$+$OCN$j$^$;$s$+$i!#e(B
Mutex#locke$B$GC1$Ke(Bstope$B$9$k$N$G$O$J$/!"e(Blocke$B$7$F$$$ke(Bthreade$B$re(Bjoine$B$9$ke(B
e$B$3$H$GBT$D$h$&$K$9$l$P=*N;$9$k$h$&$K$G$-$^$9!#e(B

Index: eval.c

— eval.c (revision 17841)
+++ eval.c (working copy)
@@ -11361,8 +11361,8 @@ rb_thread_select(max, read, write, excep
}

-static int rb_thread_join _((rb_thread_t, double));
+static int rb_thread_join0 _((rb_thread_t, double));

static int
-rb_thread_join(th, limit)
+rb_thread_join0(th, limit)
rb_thread_t th;
double limit;
@@ -11406,4 +11406,13 @@ rb_thread_join(th, limit)
}

+int
+rb_thread_join(thread, limit)

• VALUE thread;
• double limit;
  +{
• if (limit < 0) limit = DELAY_INFTY;
• return rb_thread_join0(rb_thread_check(thread), limit);
  +}

/*
@@ -11455,9 +11464,8 @@ rb_thread_join_m(argc, argv, thread)
VALUE limit;
double delay = DELAY_INFTY;

• rb_thread_t th = rb_thread_check(thread);

  rb_scan_args(argc, argv, “01”, &limit);
  if (!NIL_P(limit)) delay = rb_num2dbl(limit);

• if (!rb_thread_join(th, delay))

• if (!rb_thread_join0(rb_thread_check(thread), delay))
  return Qnil;
  return thread;
  Index: ext/thread/thread.c
  ===================================================================
  — ext/thread/thread.c (revision 17841)
  +++ ext/thread/thread.c (working copy)
  @@ -243,8 +243,11 @@ wake_all(List *list)
  }

+extern int rb_thread_join _((VALUE thread, double limit));
+#define DELAY_INFTY 1E30
+
static VALUE
-wait_list_inner(List *list)
+wait_list_inner(VALUE arg)
{

• push_list(list, rb_thread_current());

• push_list((List *)arg, rb_thread_current());
  rb_thread_stop();
  return Qnil;
  @@ -252,8 +255,8 @@ wait_list_inner(List *list)

static VALUE
-wait_list_cleanup(List list)
+wait_list_cleanup(VALUE arg)
{
/ cleanup in case of spurious wakeups */

• remove_one(list, rb_thread_current());

• remove_one((List *)arg, rb_thread_current());
  return Qnil;
  }
  @@ -391,4 +394,23 @@ rb_mutex_try_lock(VALUE self)
  }

+static VALUE
+wait_mutex(VALUE arg)
+{

• Mutex *mutex = (Mutex *)arg;
• VALUE current = rb_thread_current();
• push_list(&mutex->waiting, current);
• do {
• rb_thread_critical = 0;
• rb_thread_join(mutex->owner, DELAY_INFTY);
• rb_thread_critical = 1;
• if (!MUTEX_LOCKED_P(mutex)) {

•  mutex->owner = current;
  

•  break;
  

• }
• } while (mutex->owner != current);
• return Qnil;
  +}

/*

• Document-method: lock
  @@ -411,12 +433,5 @@ lock_mutex(Mutex *mutex)
  }
  else {

• do {

•  wait_list(&mutex->waiting);
  

•  rb_thread_critical = 1;
  

•  if (!MUTEX_LOCKED_P(mutex)) {
  

• mutex->owner = current;
• break;

•  }
  

• } while (mutex->owner != current);

• rb_ensure(wait_mutex, (VALUE)mutex, wait_list_cleanup,
  (VALUE)&mutex->waiting);
  }

[Yusuke_ENDOH]

e$B$^$D$b$He(B e$B$f$-$R$m$G$9e(B

In message “Re: [ruby-dev:35330] Re: deadlock detection for 1.9”
on Thu, 3 Jul 2008 00:16:52 +0900, “Yusuke ENDOH” [email protected]
writes:

|e$B%7%s%0%k%9%l%C%I$Ge(B Thread.stop e$B$9$k$HNc30$K$J$k$N$G!“e(BThread.stop e$B$Ge(B
|e$B0U?^E*$Ke(B sleep forever e$B$7$h$&$H$$$&?M$O$$$J$$$h$&$J5$$,$7$^$9!#e(B
|e$B$$7$mNc30$J$I$G0U?^$;$:C10l%9%l%C%I$K$J$C$F$7$^$C$?>l9g$r9M$($k$H!"e(B |e$B>e$NNc$Ge(B deadlock e$B07$$$7$F$/$l$J$$$N$O$&$l$7$/$J$$$H;W$$$^$9!#e(B | |e$B$J$N$G!VC10l%9%l%C%I$G$Oe(B deadlock e$B8!=P$7$J$$!W$H$$$&4p=$K$O$”$^$je(B
|e$BG<F@$G$-$J$$$N$G$9$,!"0l1~%Q%C%A$G$9!#e(B

e$B$3$l$O1sF#$5$s$NJ}$,@bF@NO$,$"$j$^$9$M!#%Q%C%A$OEv$F$J$$$3$He(B
e$B$K$7$^$7$g$&!#e(B1.8e$B$re(B1.9e$B$K9g$o$;$k$Y$-$+$I$&$+$OJ,$+$j$^$;$s!#e(B

|e$B$"$He(B 1.8 e$B$H5sF0$,0c$&$H$$$($P!#e(B
|[ruby-dev:34856] e$B$KJV;v$,$J$+$C$?$N$G$9$,!“e(B1.8 e$B$O0J2<$,=N;$9$k$N$Ke(B
|
|m = Mutex.new
|Thread.new { m.lock }.join
|m.lock
|
|e$B0J2<$,e(B sleep forever e$B$K$J$k$N$O$$+$7$/$J$$$G$7$g$&$+!#e(B
|
|m = Mutex.new
|Thread.new { m.lock; sleep 2 }
|sleep 1; m.lock
|
|1.9 e$B$G$O2<$NNc$G$b=*N;$9$k$h$&$K$7$F$$$^$9!#e(B
|e$B$3$l$re(B 1.8 e$B$K$”$o$;$k$N$OFq$7$=$&$G$9!#e(B

e$B$J$s$Ge(Bsleep forevere$B$K$J$k$s$G$7$g$&$M!#3N$+$K$*$+$7$$5$$,$7e(B
e$B$^$9!#e(B

[Yusuke_ENDOH]

e$BKNIt$G$9!#e(B

Yukihiro M. e$B$5$s$O=q$-$^$7$?e(B:

e$B$&!<$s!"$3$N%Q%C%A$K4s$kJQ99$,5Z$$91F6A$O;d$K$O$h$/$o$+$i$Je(B
e$B$$$N$G$9$,!"1F6A$,>/$J$$$h$&$G$7$?$i%3%%C%H$7$F$[$7$$$G$9!#e(B
e$B%j%j!<%9D>A0$N$h$&$J$N$G:#$O%?%$%%s%0$,$^$:$$$+$J!#e(B

ruby_1_8e$B$O%j%j!<%9$K$O4X78$J$$$N$G!"5$$K$7$J$/$F$$$$$H;W$$$^$9!#e(B

[Yusuke_ENDOH]

e$B$^$D$b$He(B e$B$f$-$R$m$G$9e(B

In message “Re: [ruby-dev:35362] Re: deadlock detection for 1.9”
on Fri, 4 Jul 2008 19:34:49 +0900, Urabe S.
[email protected] writes:

|Yukihiro M. e$B$5$s$O=q$-$^$7$?e(B:
|> e$B$&!<$s!"$3$N%Q%C%A$K4s$kJQ99$,5Z$$91F6A$O;d$K$O$h$/$o$+$i$Je(B
|> e$B$$$N$G$9$,!"1F6A$,>/$J$$$h$&$G$7$?$i%3%%C%H$7$F$[$7$$$G$9!#e(B
|> e$B%j%j!<%9D>A0$N$h$&$J$N$G:#$O%?%$%%s%0$,$^$:$$$+$J!#e(B
|>
|ruby_1_8e$B$O%j%j!<%9$K$O4X78$J$$$N$G!"5$$K$7$J$/$F$$$$$H;W$$$^$9!#e(B

e$B$“!”$=$&$J$k$N$+!#$4$a$s$J$5$$4pK\E*$J$3$H$rM}2r$7$F$J$/$F!#e(B
e$B$N$G!"$J$+$@$5$s$Oe(B(e$BNI$$$H;W$C$?$ie(B)e$B%3%_%C%H$7$F$/$@$5$$!#e(B

[Yusuke_ENDOH]

e$B$^$D$b$He(B e$B$f$-$R$m$G$9e(B

In message “Re: [ruby-dev:35349] Re: deadlock detection for 1.9”
on Fri, 4 Jul 2008 12:15:38 +0900, Nobuyoshi N.
[email protected] writes:

|> |m = Mutex.new
|> |Thread.new { m.lock; sleep 2 }
|> |sleep 1; m.lock

|> e$B$J$s$Ge(Bsleep forevere$B$K$J$k$s$G$7$g$&$M!#3N$+$K$*$+$7$$5$$,$7e(B
|> e$B$^$9!#e(B
|
|1.8e$B$@$H%9%l%C%I$,$I$Ne(BMutexe$B$r%m%C%/$7$F$k$+$OCN$j$^$;$s$+$i!#e(B
|Mutex#locke$B$GC1$Ke(Bstope$B$9$k$N$G$O$J$/!"e(Blocke$B$7$F$$$ke(Bthreade$B$re(Bjoine$B$9$ke(B
|e$B$3$H$GBT$D$h$&$K$9$l$P=*N;$9$k$h$&$K$G$-$^$9!#e(B

e$B$&!<$s!"$3$N%Q%C%A$K4s$kJQ99$,5Z$$91F6A$O;d$K$O$h$/$o$+$i$Je(B
e$B$$$N$G$9$,!"1F6A$,>/$J$$$h$&$G$7$?$i%3%%C%H$7$F$[$7$$$G$9!#e(B
e$B%j%j!<%9D>A0$N$h$&$J$N$G:#$O%?%$%%s%0$,$^$:$$$+$J!#e(B