Forum: Ruby Thread#raise, Thread#kill, and timeout.rb are unsafe

Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-02-25 08:18
(Received via mailing list)
I wrote up an article on Thread#raise, Thread#kill, and timeout.rb that
I hope can start making the rounds. Long story short, neither
Thread#raise nor Thread#kill is safe to use, and as a result all
libraries that call them are also unsafe (including timeout.rb, net/*,
and many other libraries in the wider world).

Have a look, add comments, discuss. Hopefully we can find a way to fix
this, because it's going to hold Ruby back until we do.

http://headius.blogspot.com/2008/02/rubys-threadra...

- Charlie
0ec4920185b657a03edf01fff96b4e9b?d=identicon&s=25 Yukihiro Matsumoto (Guest)
on 2008-03-13 17:15
(Received via mailing list)
Hi,

In message "Re: Thread#raise, Thread#kill, and timeout.rb are unsafe"
    on Mon, 25 Feb 2008 16:18:01 +0900, Charles Oliver Nutter
<charles.nutter@sun.com> writes:

|I wrote up an article on Thread#raise, Thread#kill, and timeout.rb that
|I hope can start making the rounds. Long story short, neither
|Thread#raise nor Thread#kill is safe to use, and as a result all
|libraries that call them are also unsafe (including timeout.rb, net/*,
|and many other libraries in the wider world).

Unlike Java, Thread#raise etc. should be treated similar to
KeyboardInterrupt in Ruby.  No realtime exception posting is expected.
If you handle KeyboardInterrupt safely, do same thing for Thread#raise
etc., e.g. just turning on flags to reserve exception, then raise
exception at the safe place, as MRI does.  Nothing more is required.

I admit that timeout.rb is inefficient, so that it should not be used
when performance or completeness matters.  I agree with removing use
of timeout from net/* libraries.  It's just for casual use.  Maybe
it's good to add a note in CAPITAL LETTERS in the document of
timeout.rb.

              matz.
E0d864d9677f3c1482a20152b7cac0e2?d=identicon&s=25 Robert Klemme (Guest)
on 2008-03-13 18:00
(Received via mailing list)
2008/2/25, Charles Oliver Nutter <charles.nutter@sun.com>:
> I wrote up an article on Thread#raise, Thread#kill, and timeout.rb that
>  I hope can start making the rounds. Long story short, neither
>  Thread#raise nor Thread#kill is safe to use, and as a result all
>  libraries that call them are also unsafe (including timeout.rb, net/*,
>  and many other libraries in the wider world).
>
>  Have a look, add comments, discuss. Hopefully we can find a way to fix
>  this, because it's going to hold Ruby back until we do.
>
>  http://headius.blogspot.com/2008/02/rubys-threadra...

There is a typo: your second example misses the "main." before the
raise. :-)

I have to think a bit more about this, but one remark: IMHO the
locking should be outside the begin end block.  Reason is, that if
locking fails for a reason you would not want / need to unlock.

At the moment I'm pondering implications of these options:

main = Thread.current
timer = Thread.new { sleep 5; main.raise }
lock(some_resource)
begin
  do_some_work
ensure
  unlock_some_resource
  timer.kill  # moved downwards
end


main = Thread.current
timer = Thread.new { sleep 5; main.raise }
begin
  lock(some_resource)
  begin
    do_some_work
  ensure
    unlock_some_resource
  end
ensure
  timer.kill  # separate ensure
end

The last one has the added benefit that you can implement it in a
method with a block because timer code is not interleaved with work
code.

But both have the disadvantage that you risk getting a timeout
exception during unlock - which would be unfortunate.  OTOH, killing
the timer before the lock release can violate the timing constraint if
the unlocking takes longer - not very likely.

Nasty issues; we had some similar issues with changed timeout handling
of a TX manager in JBoss.  Basically the old TX manager was able to
bust an otherwise perfect transaction by timing out in the 2PC
process. :-))

Kind regards

robert
Fe6a008c1e3065327d1f1b007d8f1362?d=identicon&s=25 Paul Brannan (cout)
on 2008-03-13 22:01
(Received via mailing list)
On Fri, Mar 14, 2008 at 01:02:52AM +0900, Yukihiro Matsumoto wrote:
> Unlike Java, Thread#raise etc. should be treated similar to
> KeyboardInterrupt in Ruby.  No realtime exception posting is expected.
> If you handle KeyboardInterrupt safely, do same thing for Thread#raise
> etc., e.g. just turning on flags to reserve exception, then raise
> exception at the safe place, as MRI does.  Nothing more is required.

I'm not convinced KeyboardInterrupt can be handled safely, either, in a
script.  What happens if a KeyboardInterrupt exception is raised inside
an ensure block?  It could easily result in resource leaks.

I think the right way to handle signals and timeouts is either in an
event loop or a dedicated thread.  Asynchronous exceptions are
convenient but error-prone.

Paul
0ec4920185b657a03edf01fff96b4e9b?d=identicon&s=25 Yukihiro Matsumoto (Guest)
on 2008-03-13 23:55
(Received via mailing list)
Hi,

In message "Re: Thread#raise, Thread#kill, and timeout.rb are unsafe"
    on Fri, 14 Mar 2008 06:00:54 +0900, Paul Brannan
<pbrannan@atdesk.com> writes:

|I'm not convinced KeyboardInterrupt can be handled safely, either, in a
|script.  What happens if a KeyboardInterrupt exception is raised inside
|an ensure block?  It could easily result in resource leaks.

You are right, but keyboard interrupt has been there so long in the
history, and we hardly see such leak problems, so I assume it's safer
than you might think.  At least, we cannot not removed the interrupts
from the language.

              matz.
Fe6a008c1e3065327d1f1b007d8f1362?d=identicon&s=25 Paul Brannan (cout)
on 2008-03-14 14:33
(Received via mailing list)
On Fri, Mar 14, 2008 at 07:43:28AM +0900, Yukihiro Matsumoto wrote:
> history, and we hardly see such leak problems, so I assume it's safer
> than you might think.  At least, we cannot not removed the interrupts
> from the language.

For simple scripts, KeyboardInterrupt is probably the right behavior.

For applications that use an event loop, we can use trap('INT') to
disable KeyboardInterrupt.

I wonder if there should be an equivalent trap mechanism for other
asynchronous exceptions?

Paul
0ec4920185b657a03edf01fff96b4e9b?d=identicon&s=25 Yukihiro Matsumoto (Guest)
on 2008-03-14 15:44
(Received via mailing list)
Hi,

In message "Re: Thread#raise, Thread#kill, and timeout.rb are unsafe"
    on Fri, 14 Mar 2008 22:32:33 +0900, Paul Brannan
<pbrannan@atdesk.com> writes:

|I wonder if there should be an equivalent trap mechanism for other
|asynchronous exceptions?

Probably.  Does anyone have any idea?

              matz.
Bec38d63650c8912b6ba9b557fb953b9?d=identicon&s=25 Roger Pack (rogerdpack)
on 2008-03-14 17:00
Charles Oliver Nutter wrote:
> I wrote up an article on Thread#raise, Thread#kill, and timeout.rb that
> I hope can start making the rounds. Long story short, neither
> Thread#raise nor Thread#kill is safe to use, and as a result all
> libraries that call them are also unsafe (including timeout.rb, net/*,
> and many other libraries in the wider world).
>
> Have a look, add comments, discuss. Hopefully we can find a way to fix
> this, because it's going to hold Ruby back until we do.
>
> http://headius.blogspot.com/2008/02/rubys-threadra...
>
> - Charlie

It was the problems with timeout that make me wish we had an
EnsureCritical (an Ensure that gets Thread.critical=true set to it
before entering it), so that timeouts can end properly.
Just my $.02
B11f10c4cd9d53970e7be20caa43f940?d=identicon&s=25 Tanaka Akira (Guest)
on 2008-03-14 18:49
(Received via mailing list)
In article <E1JaB7P-0000Zc-QP@x61.netlab.jp>,
  Yukihiro Matsumoto <matz@ruby-lang.org> writes:

> Probably.  Does anyone have any idea?

The problem is, where is safe point.

You said "just turning on flags to reserve exception, then
raise exception at the safe place, as MRI does."

Your "safe" is for interpreter.  Ruby shouldn't SEGV, etc.

But Charles's "safe" is for application.  acquired lock
should be released later, etc.

Your "safe" is not enough for application.

We need a mechanism to delay asynchronous exceptions until a
safe point.

trap is bad way to do it.  trap is similar to Unix signal
handler.  Unix signal handler makes it difficult to avoid
race conditions around blocking operations such as I/O.
With trap, applications need to re-implement the race
condition handling in Ruby level.  It is very difficult if
it is possible.

It is preferable to have a way to declare safe points
directly.

The safe points vary according to applications.  Even in a
application, different kinds of exceptions may have
different safe points.

For example, there are applications KeyboardInterrupt is
safe as you said.  I think a filter, such as grep, is a kind
of the applications.  It means that any points in the
applications are safe points.

Another example: movemail is a program to move mbox in a
mail spool.  It needs to lock a mbox.  Assume we implement
movemail in Ruby and the lock scheme is file lock.  The lock
file is removed in ensure clause.  But asynchronous
exception in the ensure block may cause to fail removing the
lock file.  So the ensure block is not safe points.

Yet another example: Apache can be stopped with apachectl
"stop" and "graceful-stop".  "stop" aborts open connections
but "graceful-stop" doesn't.  Assume we implement a
threading http server with keep-alive and similar stopping
feature in Ruby.  "stop" can be implemented by killing
threads for each connections by (dangerous) Thread.kill.  It
means that any points in the threads are safe points.
"graceful-stop" need to wait the threads until they waits
requests.  Since keep-alive is assumed, the threads
doesn't terminate after the first request on a connection.
This means the safe points are the request waiting points of
the threads.

I think the safe points declaration mechanism should able to
define safe points for each exception and easy to handle
blocking operations without race conditions.
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-14 19:50
(Received via mailing list)
On Sat, 15 Mar 2008 01:00:49 +0900, Roger Pack <rogerpack2005@gmail.com>
wrote:
> It was the problems with timeout that make me wish we had an
> EnsureCritical (an Ensure that gets Thread.critical=true set to it
> before entering it), so that timeouts can end properly.

Anything using Thread.critical has its own problems, particularly
when native threads are involved.  It can easily result in deadlocks
if the running thread inadvertently tries to use something which another
stopped thread is using (imagine a thread stopped halfway through
releasing a lock that the ensure block needs to acquire, for example
-- something which would have worked fine with normal use of locks, but
becomes a problem once threads can "stop the world").

-mental
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-14 19:54
(Received via mailing list)
On Fri, 14 Mar 2008 23:42:37 +0900, Yukihiro Matsumoto
<matz@ruby-lang.org> wrote:
> Hi,
>
> In message "Re: Thread#raise, Thread#kill, and timeout.rb are unsafe"
>     on Fri, 14 Mar 2008 22:32:33 +0900, Paul Brannan <pbrannan@atdesk.com>
> writes:
>
> |I wonder if there should be an equivalent trap mechanism for other
> |asynchronous exceptions?
>
> Probably.  Does anyone have any idea?

The simplest way to "tame" asynchronous exceptions would be to enqueue
such
exceptions until the recieving thread makes a blocking call, or else it
calls
a special method like Thread.check_interrupts.

The main difficulty with the idea is defining clearly what a "blocking
call"
is.  There probably also needs to be some mechanism (which does not
"stop
the world" like Thread.critical) to mask asynchronous exceptions even
during
blocking calls, for particularly critical code.

-mental
0ec4920185b657a03edf01fff96b4e9b?d=identicon&s=25 Yukihiro Matsumoto (Guest)
on 2008-03-15 03:32
(Received via mailing list)
Hi,

In message "Re: Thread#raise, Thread#kill, and timeout.rb are unsafe"
    on Sat, 15 Mar 2008 02:48:33 +0900, Tanaka Akira <akr@fsij.org>
writes:

|But Charles's "safe" is for application.  acquired lock
|should be released later, etc.

I know.

|Your "safe" is not enough for application.
|
|We need a mechanism to delay asynchronous exceptions until a
|safe point.

Hmm.

|I think the safe points declaration mechanism should able to
|define safe points for each exception and easy to handle
|blocking operations without race conditions.

What should safe point declaration mechanism look like?

              matz.
Bec38d63650c8912b6ba9b557fb953b9?d=identicon&s=25 Roger Pack (rogerdpack)
on 2008-03-17 16:06
> What should safe point declaration mechanism look like?
>
>               matz.

Tongue in cheek I would say something like

uninterruptible do

end

or

ensureUninterruptible

end

or

uninterruptible do
  # some stuff
handle_exceptions_that_have_been_thrown
end
# where handle_exceptions is where they can all get handled.
Kind of scary still, though.

:)

Anyway my latest thought on the whole timeout difficulty is that there
are a few difficulties with it that might be overcomeable:

Unfortunately, as a previous poster noted, using raise seems itself
almost always inherently dangerous, so these would probably all be
band-aids to a bigger problem.

1) if you have two nested timeouts then "you don't know where the
timeout came from"
timeout 5 do
timeout 5 do
begin
 sleep
 rescue Exception
  # which one is it? What did I just rescue?
 end
end
end


You could overcome this by just ensuring that it throws distinct classes
per timeout (i.e. if different timeouts).
timeout 5 do # will by default throw class Timeout::ErrorDepth1 <
Timeout::Error
timeout 5 do # will by default throw class Timeout::ErrorDepth2 <
Timeout::Error

I suppose this would help people decipher what is happening when
timeouts are thrown, though not be helpful otherwise.

1.5) It creates a new thread once per call to 'timeout' which is less
than efficient.

While I typically don't care about efficiency when using timeout, it
might be possible to just have a "single timer thread" that handles all
the raise'ings on the others, to save on thread creation
(theoretically).


So something like this

(sorry if the syntax is wrong--this is just pseudo-code for now)

$timer_thread = nil # just so I remember the name--you wouldn't have to
use a global, of course

class TimerThread < Thread
 def handle_any_requests
   # if any thread requests a timeout, add it to some list
   if some_thread_is_waiting
     sleep_until_should_wake_next_thread
     if waiting_thread_is_still_running_its_timeout_block
        waiting_thread.raise Timeout::Error
     end
   else
    sleep -1
  end
 end

 def TimerThread.add_me_to_the_queue_to_interrupt_me_in seconds, &block
    $timer_thread.add_it_to_queue Thread.current, seconds
    $timer_thread.wakeup # might need to synchronize this call so that
we don't get confusion
    block.call # if we get interrupted between this line and the next we
are DEAD.  Exactly the same problem that we had before--an exception
could be raised "far in the future" to interrupt it, when it has already
left and it shouldn't be raised.
    $timer_thread.take_me_off_the_queue_I_am_done Thread.current # might
need to synchronize this call, too
  end

end



# begin program.  Note that you could just fire this up the first time
the script calls Timeout::timeout
$timer_thread = Thread.new {
  loop do
     handle_any_requests
  end
}


Now when you want timeouts you do something like
TimerThread.add_me_to_the_queue_to_interrupt_me_in 5 do # seconds

  # do some stuff for a long time

end

That would help with efficiency, since you wouldn't have to spawn a new
thread for each call (though it would, sigh, leave one running always in
the background).  It would alleviate some concurrency problems, and
still leave at least one in there.  Just throwing it out there.

2) nested timeouts can interrupt each other and leave and 'errant
thread' running which raises a

timeout 5 do
timeout 50 do
begin
 sleep
end
end

This causes two threads to be generated, each with a "time out bomb"
ticking, waiting to happen.  The outer loops' thread could (as Charles
Nutter noted) interrupt the inner thread during its ensure block, so it
ends up not killing its ticking "time out bomb" thread.  Which means
that 45 seconds later it will go off and randomly raise on the parent
thread.

To avoid this I "think" you could wrap the things with mutexes.
However, this also doesn't work because the mutex "synchronize" stuff
itself uses an ensure block, which could be interrupted by the same
confusion (and does, in my experience).

<<Sigh>>

Never mind.  I give up.  The only safe way to generate asynchronous
calls seems to be to use EventMachine or rev (which have asynchronous
timers) and have your own "exception handling" blocks (i.e. blocks where
it is designated to handle those exceptions).

Thanks.

-R
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-17 17:31
(Received via mailing list)
On Tue, 18 Mar 2008 00:06:47 +0900, Roger Pack <rogerpack2005@gmail.com>
wrote:
> end
>
> or
>
> uninterruptible do
>   # some stuff
> handle_exceptions_that_have_been_thrown
> end
> # where handle_exceptions is where they can all get handled.
> Kind of scary still, though.

In a language with side-effects, I've become convinced that the only
safe
way to handle asynchronous exceptions is for code to be uninterruptible
by
default, and only explicitly allow interrupts at specific points (not
broad
regions of code).

This gets back to the fundamental principle that, in order to write a
correct multithreaded program, any interaction between threads needs to
be
by explicit mutual agreement at specific places.  One thread needs to
explicitly initiate the interaction (in this case, by calling
Thread#raise),
and the other thread needs to explicitly accept it (e.g. by calling a
method
which is known to permit the delivery of any pending interrupts when it
is
called).  Otherwise, the resulting nondeterminism is impossible to
control.

-mental
B11f10c4cd9d53970e7be20caa43f940?d=identicon&s=25 Tanaka Akira (Guest)
on 2008-03-18 06:05
(Received via mailing list)
In article <E1JaM9H-0000jj-AB@x61.netlab.jp>,
  Yukihiro Matsumoto <matz@ruby-lang.org> writes:

> What should safe point declaration mechanism look like?

Basically, asynchronous events should be queued.

* make a queue for each thread.
* Thread#raise(exc) enqueue exc to the queue.
* Thread.check_interrupt(klass) checks an exception which is
  a kind of klass in the queue of the current thread.  If it
  exists, the exception is raised.
* other queue examining mechanism, such as sigpending, etc,
  may be useful.

Thread.check_interrupt is a safe point.

However safe points is not only Thread.check_interrupt.
Blocking operations, such as I/O, may or may not be safe
points.  It is because Thread.check_interrupt with blocking
operations causes race conditions.  So application must
choose that make a blocking operation interruption safe or
uninterruptible.

* Thread.blocking_interruptible(klass) { ... }
* Thread.blocking_uninterruptible(klass) { ... }

Blocking operations in Thread.blocking_interruptible are
safe points.

If a thread has no resources to release, it is safe to kill
asynchronously.  Other than that, safe points should be
declared explicitly.  When a resource is acquired,
application should declare it.

* Thread.delay_interrupt(klass) { ... }

It is safe points that out of outermost
Thread.delay_interrupt.

Another idea is about ensure clause.  Since an ensure
clause is used to release a resource, it may delay
asynchronous events as in Thread.delay_interrupt.
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 ara howard (Guest)
on 2008-03-18 06:20
(Received via mailing list)
On Mar 17, 2008, at 10:30 AM, MenTaLguY wrote:

> In a language with side-effects, I've become convinced that the only
> safe
> way to handle asynchronous exceptions is for code to be
> uninterruptible by
> default, and only explicitly allow interrupts at specific points
> (not broad
> regions of code).

no doubt you've read it, but for posterity i'll post it here:

   http://209.85.207.104/search?q=cache:2T61vSNQ4_EJ:...

a @ http://codeforpeople.com/
E7559e558ececa67c40f452483b9ac8c?d=identicon&s=25 Gary Wright (Guest)
on 2008-03-18 14:46
(Received via mailing list)
On Mar 18, 2008, at 1:19 AM, ara howard wrote:

>
> no doubt you've read it, but for posterity i'll post it here:
>
>   http://209.85.207.104/search?q=cache:
> 2T61vSNQ4_EJ:home.pacbell.net/ouster/threads.pdf+%22why+threads+are
> +a+bad+idea%22&hl=en&ct=clnk&cd=3&gl=us&client=firefox-a
>

Here is a related talk/paper that offers a slightly different
perspective than Ousterhout: <http://swtch.com/~rsc/talks/threads07/>

Gary Wright
4299e35bacef054df40583da2d51edea?d=identicon&s=25 James Gray (bbazzarrakk)
on 2008-03-18 14:55
(Received via mailing list)
On Mar 18, 2008, at 12:04 AM, Tanaka Akira wrote:

>  a kind of klass in the queue of the current thread.  If it
>  exists, the exception is raised.
> * other queue examining mechanism, such as sigpending, etc,
>  may be useful.

Perl went kind-of along these lines around version 5.8, if I recall
correctly.  They added "safe signals" which were signals that only
interrupted at times when it was safe for them to do so.

This did have a few weird side-effects though.  For example, running a
complex regex would shutoff signal handling for the duration.  That
meant you could no longer timeout a regex.

Perhaps they've fixes these issues now, but I remember the change was
not as smooth as they had hoped it would be.

James Edward Gray II
B11f10c4cd9d53970e7be20caa43f940?d=identicon&s=25 Tanaka Akira (Guest)
on 2008-03-18 15:03
(Received via mailing list)
In article <41DCFB5E-7EF8-4A1E-81B7-95A6A7691F04@grayproductions.net>,
  James Gray <james@grayproductions.net> writes:

> Perl went kind-of along these lines around version 5.8, if I recall
> correctly.  They added "safe signals" which were signals that only
> interrupted at times when it was safe for them to do so.

It's from 5.7.3.
http://search.cpan.org/dist/perl/pod/perlipc.pod#D...)

> This did have a few weird side-effects though.  For example, running a
> complex regex would shutoff signal handling for the duration.  That
> meant you could no longer timeout a regex.

Ruby already defer signal handling same as Perl.
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 ara howard (Guest)
on 2008-03-18 16:21
(Received via mailing list)
On Mar 18, 2008, at 7:44 AM, Gary Wright wrote:

>
> Here is a related talk/paper that offers a slightly different
> perspective than Ousterhout: <http://swtch.com/~rsc/talks/threads07/>

with 42 slides - it must be good!

;-)

a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 ara howard (Guest)
on 2008-03-18 16:23
(Received via mailing list)
On Mar 14, 2008, at 8:42 AM, Yukihiro Matsumoto wrote:

> Probably.  Does anyone have any idea?


Thread.current.raises do |exception|
   queue.push exception
end

....

even_loop do
   stuff
   if exception = queue.pop
     ...
   end
end

seems like it would address a ton of issued without overly
complicating ruby's internals - just provide an 'on raise' handler.

cheers.

a @ http://codeforpeople.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 ara howard (Guest)
on 2008-03-18 16:24
(Received via mailing list)
On Mar 18, 2008, at 7:44 AM, Gary Wright wrote:

>
> Here is a related talk/paper that offers a slightly different
> perspective than Ousterhout: <http://swtch.com/~rsc/talks/threads07/>

with 42 slides - it must be good!

;-)

a @ http://codeforpeople.com/
47b1910084592eb77a032bc7d8d1a84e?d=identicon&s=25 Joel VanderWerf (Guest)
on 2008-03-18 18:42
(Received via mailing list)
ara howard wrote:
> ....
>
> even_loop do
>   stuff
>   if exception = queue.pop
>     ...
>   end
> end
>
> seems like it would address a ton of issued without overly complicating
> ruby's internals - just provide an 'on raise' handler.

Is it possible to integrate this with IO#select, so a thread can handle
an exception sent to it while waiting on io?
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-18 18:53
(Received via mailing list)
On Wed, 19 Mar 2008 00:20:54 +0900, ara howard <ara.t.howard@gmail.com>
wrote:
>    stuff
>    if exception = queue.pop
>      ...
>    end
> end
>
> seems like it would address a ton of issued without overly
> complicating ruby's internals - just provide an 'on raise' handler.

How do you plan on protecting against non-determinism introduced
by the "on raise" handler?  For example, if queue were a
user-implemented data structure, and the handler fired in the middle
of queue.pop?  If queue.pop is not protected by a lock, there is
potential to corrupt the queue object.  If it is protected by a lock,
there is potential for deadlock.

(This is basically the same issue as asynchronous signal safety.)

I think the only way around this is for the receiving thread to
explicitly choose moments when it wants to receive any pending
asynchronous exceptions.  Can you think of another way?

-mental
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-18 18:58
(Received via mailing list)
On Tue, 18 Mar 2008 22:44:42 +0900, Gary Wright <gwtmp01@mac.com> wrote:
> Here is a related talk/paper that offers a slightly different
> perspective than Ousterhout: <http://swtch.com/~rsc/talks/threads07/>

+1; those slides are quite good.

-mental
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-18 19:10
(Received via mailing list)
On Tue, 18 Mar 2008 14:04:40 +0900, Tanaka Akira <akr@fsij.org> wrote:
> Thread.check_interrupt is a safe point.
>
> Blocking operations in Thread.blocking_interruptible are
> safe points.

This sounds very good!  I hadn't considered anything like
blocking_interruptible, but it seems useful.

-mental
Bec38d63650c8912b6ba9b557fb953b9?d=identicon&s=25 Roger Pack (rogerdpack)
on 2008-03-18 19:16
> Thread.current.raises do |exception|
>    queue.push exception
> end
>
> ....
>
> event_loop do
>    long_running_stuff
>    if exception = queue.pop
>      ...
>    end
> end
>
> seems like it would address a ton of issued without overly
> complicating ruby's internals - just provide an 'on raise' handler.

This looks nice!

I wish that a handler like this could be "unhandled," as well, as you
have the flexibility to handle exceptions only in parts of the code.
Say we call your first block of code as queue_future_exceptions

Then you could have code like

Thread.current.queue_future_exceptions
  # code in here is uninterruptible
Thread.current.no_longer_queue_future_exceptions # so I don't have to
worry about them EVERYWHERE in my code--only at the end of certain
blocks.
Thread.current.raises do |exception| ; end

So basically it would allow for uninterruptible blocks.  That's what I
wish we had.


In response to Mental's questions of concurrency problems for raises
injected whilst you are in the middle of handling raises, some options
that might help:

1) protect the queue with mutexes--add them to the end of the queue
atomically.
2) dare I say it, require the exception handling to be either within or
without of an "uninterruptible block" --that is, allow the exception
handling to be interrupted (itself), or prevent it from being
interrupted, always, to provide for sanity.

The first option (require handling outside of an uninterruptible block)
would allow for us to "not have to worry" about concurrency of adding to
the exception queue, since we no longer add to the queue but just raise
on the thread.  It would allow it to be interrupted, which might be
annoying, however, and isn't our purpose to "avoid" those pesky
asynchronous interrupts?

The second option (requiring handling within an uninterruptible block)
avoids handling's being interrupted, however, it could "miss" an
interrupt or two, should they be raised between when we handle them and
when an "uninterruptible block" ends.

So I guess the best way to deal with it would be to surround queue
addition with mutexes, and handle them both right before an
uninterruptible block ends, and again right after it ends.

Ok so not such a great idea.
Thoughts?
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 ara howard (Guest)
on 2008-03-18 20:14
(Received via mailing list)
On Mar 18, 2008, at 11:52 AM, MenTaLguY wrote:

> I think the only way around this is for the receiving thread to
> explicitly choose moments when it wants to receive any pending
> asynchronous exceptions.  Can you think of another way?



perhaps i wasn't being clear.  that's exactly what i was showing - a
handler that gets called (critical section of course) which simply
enques the exception.  then the thread itself must check for these at
the 'right' moment.


a @ http://codeforpeople.com/
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-18 22:23
(Received via mailing list)
On Wed, 19 Mar 2008 04:13:19 +0900, ara howard <ara.t.howard@gmail.com>
wrote:
> On Mar 18, 2008, at 11:52 AM, MenTaLguY wrote:
>
>> I think the only way around this is for the receiving thread to
>> explicitly choose moments when it wants to receive any pending
>> asynchronous exceptions.  Can you think of another way?
>
> perhaps i wasn't being clear.  that's exactly what i was showing - a
> handler that gets called (critical section of course) which simply
> enques the exception.  then the thread itself must check for these at
> the 'right' moment.

Okay.  I'd thought you were suggesting an API rather than illustrating
an idea.

-mental
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-18 22:38
(Received via mailing list)
On Wed, 19 Mar 2008 03:16:05 +0900, Roger Pack <rogerpack2005@gmail.com>
wrote:
> In response to Mental's questions of concurrency problems for raises
> injected whilst you are in the middle of handling raises, some options
> that might help:
>
> 1) protect the queue with mutexes--add them to the end of the queue
> atomically.

This would deadlock if the handler happened to fire while the queue's
lock was held.  Generally, locking doesn't mix with anything that can
interrupt or suspend the execution of a thread at an arbitrary time.

> 2) dare I say it, require the exception handling to be either within or
> without of an "uninterruptible block" --that is, allow the exception
> handling to be interrupted (itself), or prevent it from being
> interrupted, always, to provide for sanity.

That's a good point -- besides conflicts between the handler and
whatever
the recipient thread is running at the time, you do also need to be
concerned with conflicts between multiple handler instances (if that
is allowed).

> So I guess the best way to deal with it would be to surround queue
> addition with mutexes, and handle them both right before an
> uninterruptible block ends, and again right after it ends.
>
> Ok so not such a great idea.

In terms of an API for signal queuing, I think Tanaka Akira's idea
is elegant and reliable.

-mental
0ec4920185b657a03edf01fff96b4e9b?d=identicon&s=25 Yukihiro Matsumoto (Guest)
on 2008-03-19 02:05
(Received via mailing list)
Hi,

In message "Re: Thread#raise, Thread#kill, and timeout.rb are unsafe"
    on Tue, 18 Mar 2008 23:02:24 +0900, Tanaka Akira <akr@fsij.org>
writes:

|> This did have a few weird side-effects though.  For example, running a
|> complex regex would shutoff signal handling for the duration.  That
|> meant you could no longer timeout a regex.
|
|Ruby already defer signal handling same as Perl.

1.8 regexp does check interrupts during match, 1.9 doesn't check (yet).

              matz.
Fe6a008c1e3065327d1f1b007d8f1362?d=identicon&s=25 Paul Brannan (cout)
on 2008-03-19 18:29
(Received via mailing list)
On Tue, Mar 18, 2008 at 02:04:40PM +0900, Tanaka Akira wrote:
> It is because Thread.check_interrupt with blocking operations causes
> race conditions.

Raising an exception during a blocking write operation makes it
impossible to know how much data was written.

Similarly, raising an exception during a blocking read operation makes
it impossible to access the data that was read.

The same is true for any operation for which partial success is
possible.

> So application must choose that make a blocking operation interruption
> safe or uninterruptible.

> * Thread.blocking_interruptible(klass) { ... }
> * Thread.blocking_uninterruptible(klass) { ... }

Interesting.  Which of these two is the default?

> Another idea is about ensure clause.  Since an ensure clause is used
> to release a resource, it may delay asynchronous events as in
> Thread.delay_interrupt.

Reminds me of an old C++ interview question we used to joke about:

"While digging through source code, you discover an application which
executes entirely in the body of the destructor of a global object.
Give one or more reasons why the author may have chosen to implement the
application this way."

Paul
F889bf17449ffbf62345d2b2d316a937?d=identicon&s=25 Michal Suchanek (Guest)
on 2008-03-19 18:57
(Received via mailing list)
On 18/03/2008, ara howard <ara.t.howard@gmail.com> wrote:
>
>
> no doubt you've read it, but for posterity i'll post it here:
>
> 
http://209.85.207.104/search?q=cache:2T61vSNQ4_EJ:...

Whatever it's supposed to be the link does not work for me.

Thanks

Michal
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-19 19:51
(Received via mailing list)
On Thu, 20 Mar 2008 02:29:09 +0900, Paul Brannan <pbrannan@atdesk.com>
wrote:
> The same is true for any operation for which partial success is
> possible.

That's a good point.  To some extent it is an API design and
implementation problem; an API which can partially "succeed" in that
fashion is arguably flawed.

There are really two alternatives:

 1. fix the operation so that it succeeds or fails atomically

 2. otherwise, make the operation always non-interruptable

(Well, there is also a third solution, used by Java's NIO for blocking
reads and writes: on interrupt, destroy the object in question so that
it doesn't matter anymore.  But I don't think it's a particularly good
solution.)

-mental
Fe6a008c1e3065327d1f1b007d8f1362?d=identicon&s=25 Paul Brannan (cout)
on 2008-03-19 21:58
(Received via mailing list)
On Thu, Mar 20, 2008 at 03:50:13AM +0900, MenTaLguY wrote:
> There are really two alternatives:
>
>  1. fix the operation so that it succeeds or fails atomically
>
>  2. otherwise, make the operation always non-interruptable

For I/O operations, it is typically desirable for the operation to be
interruptible, but to get the result of the partial operation after it
is interrupted.  At least, that's how it works at the C level with
respect to signals.

Paul
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-19 23:24
(Received via mailing list)
On Thu, 20 Mar 2008 05:57:02 +0900, Paul Brannan <pbrannan@atdesk.com>
wrote:
> respect to signals.
Yes.  For my purposes, partial success is still success as long
as the caller can tell how far it got.  Even in the absence of
signals, a successful write() isn't guaranteed to write everything
that was requested.  The important thing is that the operation
doesn't report total failure when it really succeeded, or vice-versa.

Historical example:  under SVR4, a write() interrupted by a signal
would always fail with EINTR, even if some bytes had already
been written.  POSIX later remedied this, so that write() would
always report success if any bytes were written, even if it was
interrupted by a signal.  In that instance, the API was modified
to conform to alternative #1.

Now, the question with IO and asynchronous exceptions in Ruby is
whether there is a good way to report a partial result?

-mental
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-19 23:32
(Received via mailing list)
On Thu, 20 Mar 2008 07:21:06 +0900, MenTaLguY <mental@rydia.net> wrote:
> Yes.  For my purposes, partial success is still success as long
> as the caller can tell how far it got.

(For the sake of those following along who might not have a lot of
C or Unix experience, I forgot to mention that a successful write()
always returns the number of bytes actually written.  The issue with
SVR4 Unix was that interrupted writes were always treated like
failures, so no count was returned in those cases and you couldn't
tell how much you had left over in case you needed to try again.)

-mental
Bec38d63650c8912b6ba9b557fb953b9?d=identicon&s=25 Roger Pack (rogerdpack)
on 2008-03-21 08:08
I still like Ara's idea :)

However--is something like this already possible if we redefine the
Thread#raise method?  For example define it to be something like this.

class Thread
 def raise_safe *args
       # sorry, couldn't think of a better way than a mutex to prohibit
weirdness from occurring.  So that's where a problem would occur--in the
ensure block of the synchronize function.  Ahh well.  If you re-aliased
raise to be this method, maybe avoided.

  @raise_mutex.synchronize {
    unless @currently_non_interruptible_section
       raise *args
    else
          self.queued_interrupts << *args
    end
  }
 end

 def uninterruptible_block
    return if @currently_non_interruptible_section # already within one.
Maybe should keep a count of how deep you are. Not sure
    @currently_non_interruptible_section = true
    begin
      yield # might want to handle queued exceptions at the end of this
block
    ensure
      @raise_mutex.synchronize {@currently_non_interruptible_section =
false}
    end
    # after this point the thread is subject to interruptions again
 end

end

Then you could have a thread do something like

uninterruptible_block do
  # some stuff
  # handle the queued interrupts if desired.  Note that you could loop
and 'check if you've been interupted yet' once per loop or what not, if
desired

end

# handle queued interrupts, if desired



timeout would possibly become

def timeout(sec, exception=Error)
  return yield if sec == nil or sec.zero?
  raise ThreadError, "timeout within critical session"\
                                      if Thread.critical
  raise ThreadError, "called nested functions that want to use
uninterruptible and raise on it" if
Thread.current.currently_non_interruptible_section
  # this is necessary since we do use 'real' raise this time
  # to interrupt our block's yielding
  # so if you had nested--ever--the raise of an uppermost one
  # could interrupt a lower one in it ensure. Same problem would occur.
  # we can only have one at a time.
  # this is a hack that assumes we 'only' use uninterruptible_block for
calls to timeout.

  death_mutex = Mutex.new

  uninterruptible_block do
   begin
     x = Thread.current
     y = Thread.start {
       sleep sec
       death_mutex.synchronize {
         x.raise exception, "execution expired" if x.alive? # this is
not raise_safe, but real raise.  Still unsafe as per some previous
comments...but is it safer than it used to be now?
       }
     }
     yield sec # would this code be dampered somehow?
   ensure
     death_mutex.synchronize {
       y.kill if y and y.alive?
     }
   end
  end
  for interrupt in Thread.current.queued_interrupts do; raise interrupt;
end # or something like that
end

now you could "replace" the use of raise with raise_safe.  Would this
work?
This would prohibit 'other threads' from interrupting a thread that is
within a timeout block (obviously bad) until it timed out, but would it
be thread safe?

Thoughts?
Thank you for reading :)

Overall I'd say I'm in agreeance with the OP's call to deprecate
Thread#kill, Thread#raise, Thread#critical= as they can too easily hurt,
though something like this might help.  This suggestion also seems far
less complete than Tanaka's somehow.

Still wishing for an ensure block that was somehow 'guaranteed' to run
all the way through, and delay interruptions or ignore them till it
ended.
Thanks.

-R


Tanaka Akira wrote:
> In article <E1JaM9H-0000jj-AB@x61.netlab.jp>,
>   Yukihiro Matsumoto <matz@ruby-lang.org> writes:
>
>> What should safe point declaration mechanism look like?
>
> Basically, asynchronous events should be queued.
>
> * make a queue for each thread.
> * Thread#raise(exc) enqueue exc to the queue.
> * Thread.check_interrupt(klass) checks an exception which is
>   a kind of klass in the queue of the current thread.  If it
>   exists, the exception is raised.
> * other queue examining mechanism, such as sigpending, etc,
>   may be useful.
>
> Thread.check_interrupt is a safe point.
>
> However safe points is not only Thread.check_interrupt.
> Blocking operations, such as I/O, may or may not be safe
> points.  It is because Thread.check_interrupt with blocking
> operations causes race conditions.  So application must
> choose that make a blocking operation interruption safe or
> uninterruptible.
>
> * Thread.blocking_interruptible(klass) { ... }
> * Thread.blocking_uninterruptible(klass) { ... }
>
> Blocking operations in Thread.blocking_interruptible are
> safe points.
>
> If a thread has no resources to release, it is safe to kill
> asynchronously.  Other than that, safe points should be
> declared explicitly.  When a resource is acquired,
> application should declare it.
>
> * Thread.delay_interrupt(klass) { ... }
>
> It is safe points that out of outermost
> Thread.delay_interrupt.
>
> Another idea is about ensure clause.  Since an ensure
> clause is used to release a resource, it may delay
> asynchronous events as in Thread.delay_interrupt.
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-03-22 06:56
(Received via mailing list)
ara howard wrote:
> no doubt you've read it, but for posterity i'll post it here:
>
>
> 
http://209.85.207.104/search?q=cache:2T61vSNQ4_EJ:...

I don't think we'll get anywhere solving the problem by saying "let's
just remove threads" or "don't use threads".

Threads are hard, yes. Threads are too hard for most programmers, yes.
But since they're in Ruby, they're going to get used...and providing
provably unsafe operations on threads is obviously not making matters
better.

Besides, even if *you* avoid using threads, the libraries you call might
not. So we need a global answer.

- Charlie
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-03-22 07:03
(Received via mailing list)
Tanaka Akira wrote:
> Thread.check_interrupt is a safe point.
Agreed. This is essentially how these unsafe operations are implemented
in JRuby currently. The "raise queue" is one-element long, and
appropriate locks are acquired to modify it from outside the target
thread. All threads periodically check to see if they've been "raised"
or "killed" or whether they should sleep because another thread has
"gone critical". The checkpoints are currently at the same points during
execution that Ruby 1.8 checks whether it should run the thread
scheduler.

> However safe points is not only Thread.check_interrupt.
> Blocking operations, such as I/O, may or may not be safe
> points.  It is because Thread.check_interrupt with blocking
> operations causes race conditions.  So application must
> choose that make a blocking operation interruption safe or
> uninterruptible.
>
> * Thread.blocking_interruptible(klass) { ... }
> * Thread.blocking_uninterruptible(klass) { ... }

Also good. Awful names though :)

> It is safe points that out of outermost
> Thread.delay_interrupt.
>
> Another idea is about ensure clause.  Since an ensure
> clause is used to release a resource, it may delay
> asynchronous events as in Thread.delay_interrupt.

As mental has said, if threads are uninterruptible by default, this
would make ensures safe. I think that's the only reasonable option.

- Charlie
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-03-22 07:05
(Received via mailing list)
ara howard wrote:
> handler that gets called (critical section of course) which simply
> enques the exception.  then the thread itself must check for these at
> the 'right' moment.

Presumably by "critical section" you mean "locking on as narrow a lock
as possible so as not to stop the whole freaking world because one
thread wants to send an event to another". critical= is the devil.

- Charlie
Ede2aa10c6462f1d825143879be59e38?d=identicon&s=25 Charles Oliver Nutter (Guest)
on 2008-03-22 07:06
(Received via mailing list)
MenTaLguY wrote:
> -- something which would have worked fine with normal use of locks, but
> becomes a problem once threads can "stop the world").

And to clarify, this problem isn't restricted to native threads...even a
green-threaded model could deadlock if one thread goes critical while
other threads are holding unrelated locks.

- Charlie
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 ara howard (Guest)
on 2008-03-22 08:04
(Received via mailing list)
On Mar 22, 2008, at 12:04 AM, Charles Oliver Nutter wrote:

> Presumably by "critical section" you mean "locking on as narrow a
> lock as possible so as not to stop the whole freaking world because
> one thread wants to send an event to another". critical= is the devil

yes - lock only long enough to push the exception onto a stack/queue
where it can be checked at an opportune moment - as opposed to async
exceptions just bubbling in whenever.

a @ http://drawohara.com/
1bac2e65d64faf472cf2ebc94f0f5ee0?d=identicon&s=25 ara howard (Guest)
on 2008-03-22 08:06
(Received via mailing list)
On Mar 21, 2008, at 11:55 PM, Charles Oliver Nutter wrote:

> I don't think we'll get anywhere solving the problem by saying
> "let's just remove threads" or "don't use threads".

indeed.  the only sane approach, i think, is the middle one whereby
threads are there, exceptions are there, but cross thread exceptions
operated via channels and events.

a @ http://codeforpeople.com/
B11f10c4cd9d53970e7be20caa43f940?d=identicon&s=25 Tanaka Akira (Guest)
on 2008-03-24 12:15
(Received via mailing list)
In article <3bc15bbea3f2a64de52c462c8e32d6c3@localhost>,
  MenTaLguY <mental@rydia.net> writes:

> This sounds very good!  I hadn't considered anything like
> blocking_interruptible, but it seems useful.

The idea is come from the cancelation point of pthread.
B11f10c4cd9d53970e7be20caa43f940?d=identicon&s=25 Tanaka Akira (Guest)
on 2008-03-24 13:23
(Received via mailing list)
In article <20080319172919.GL26883@atdesk.com>,
  Paul Brannan <pbrannan@atdesk.com> writes:

> Raising an exception during a blocking write operation makes it
> impossible to know how much data was written.
>
> Similarly, raising an exception during a blocking read operation makes
> it impossible to access the data that was read.

It is a very difficult problem.

I don't try to solve the problem perfectly.

I think the data lost may be acceptable if an asynchronous
event is for termination.

If the purpose is termination, data lost may be happen
anyway.  For example, data from TCP is lost if a process
exits before data arrival.

If data lost is acceptable but termination delay is not
acceptable, Thread.blocking_interruptible(klass) { I/O } can
be used.

If data lost is not acceptable but termination delay is
acceptable, Thread.blocking_uninterruptible(klass) { I/O }
can be used.

If data lost and termination delay is not acceptable, it is
difficult.  One idea is
Thread.blocking_interruptible(klass) { IO.select } and
nonblocking I/O.  But nonblocking I/O causes inherently
partial result.  So there should be a read/write buffer.  If
termination procedure ignore the read buffer, data lost
occur.  If termination procedure flushes the write buffer,
it may blocks.  So data lost or termination delay again.  It
is the difficult problem.

I hope either data lost or termination delay is acceptable.

If an asynchronous event is used for non-termination, data
lost is not acceptable in general.  Assume a procedure is
called for the event.  If some delay is acceptable,
Thread.blocking_uninterruptible(klass) { I/O } can be used.
If the delay is also not acceptable, it is the difficult
problem.  However if the event is caused by Thread#raise,
I'm not sure why the procedure is not called directly
instead of Thread#raise.  If the event is caused by signal,
I have no good idea to do it.

So I think asynchronous events should be used only for
termination.  This is why I think exception on blocking
operation is tolerable.
B11f10c4cd9d53970e7be20caa43f940?d=identicon&s=25 Tanaka Akira (Guest)
on 2008-03-24 13:42
(Received via mailing list)
In article <47E4A106.7060405@sun.com>,
  Charles Oliver Nutter <charles.nutter@sun.com> writes:

> As mental has said, if threads are uninterruptible by default, this
> would make ensures safe. I think that's the only reasonable option.

If threads are uniterruptible by default for all kind of
asynchronous events, SIGINT and other signals are not
effective by default.  It is not reasonable.

So the default should be depended by kind of asynchronous
events.  I think it is reasonable that signals are
interruptible by default but others are uninterruptible by
default.
E0526a6bf302e77598ef142d91bdd31c?d=identicon&s=25 Daniel DeLorme (Guest)
on 2008-03-24 14:18
(Received via mailing list)
I think the specific problem mentionned was that asynchronous exceptions
can abort the cleanup within an ensure section. So what about a simple
solution to that specific problem, like: if an asynchronous exception is
raised inside an ensure section, the exception is queued and raised at
the end of the ensure section. Is that too naive?
Fe6a008c1e3065327d1f1b007d8f1362?d=identicon&s=25 Paul Brannan (cout)
on 2008-03-24 16:44
(Received via mailing list)
On Thu, Mar 20, 2008 at 07:21:06AM +0900, MenTaLguY wrote:
> Now, the question with IO and asynchronous exceptions in Ruby is
> whether there is a good way to report a partial result?

The existing API already returns the number of bytes successfully
written for both IO#write and IO#syswrite.  But how do we find out what
caused the operation to be interrupted?  Currently IO#write gives an
exception:

irb(main):001:0> r, w = IO.pipe
=> [#<IO:0x4039f734>, #<IO:0x4039f70c>]
irb(main):002:0> t = Thread.new { w.write("HELLO" * 20000) }
=> #<Thread:0x4039a0a4 run>
irb(main):003:0> r.close
=> nil
irb(main):004:0> t.join
Errno::EPIPE: Broken pipe
        from (irb):2:in `write'
        from (irb):4:in `join'
        from (irb):4

and IO#syswrite gives the number of bytes written:

irb(main):018:0> sock = TCPSocket.new('localhost', 2000)
=> #<TCPSocket:0x4043bf1c>
irb(main):019:0> t = Thread.new { p sock.syswrite("HELLO" * 200000) }
=> #<Thread:0x40428804 run>
irb(main):024:0> t.join
131072
=> #<Thread:0x4040d324 dead>

I think the behavior of IO#syswrite is correct (because there is only
one write(2) system call), but IO#write actually got *both* partial
success *and* an exceptional condition, so what should its behavior be?

Paul
Fe6a008c1e3065327d1f1b007d8f1362?d=identicon&s=25 Paul Brannan (cout)
on 2008-03-24 16:52
(Received via mailing list)
On Mon, Mar 24, 2008 at 09:41:29PM +0900, Tanaka Akira wrote:
> So the default should be depended by kind of asynchronous
> events.  I think it is reasonable that signals are
> interruptible by default but others are uninterruptible by
> default.

I think this is reasonable too, especially given that we can turn
signals off using trap.

Paul
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-24 17:08
(Received via mailing list)
On Mon, 24 Mar 2008 22:17:34 +0900, Daniel DeLorme <dan-ml@dan42.com>
wrote:
> I think the specific problem mentionned was that asynchronous exceptions
> can abort the cleanup within an ensure section. So what about a simple
> solution to that specific problem, like: if an asynchronous exception is
> raised inside an ensure section, the exception is queued and raised at
> the end of the ensure section. Is that too naive?

Maybe slightly.  I don't think it would necessarily be an issue for the
current implementation of 1.9, but you do also need to do the
registration/unregistration of ensure and catch clauses in an
uninterruptible fashion.

Basically you need to have the equivalent of this:

 Thread.uninterruptable do
   begin
     Thread.interruptable do
       # ...
     end
   rescue
     # ...
   ensure
     # ...
   end
 end

(in cases where the thread is currently interruptable)

See also section 4.2 of "Asynchronous Exceptions in Haskell":

 http://citeseer.ist.psu.edu/415348.html

-mental
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-24 17:11
(Received via mailing list)
On Mon, 24 Mar 2008 21:41:29 +0900, Tanaka Akira <akr@fsij.org> wrote:
> So the default should be depended by kind of asynchronous
> events.  I think it is reasonable that signals are
> interruptible by default but others are uninterruptible by
> default.

This seems sensible to me, though it would be nice to also have
an alternate way to handle POSIX signals that didn't involve
interruption.

-mental
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-24 17:26
(Received via mailing list)
On Mon, 24 Mar 2008 21:22:35 +0900, Tanaka Akira <akr@fsij.org> wrote:
> So I think asynchronous events should be used only for
> termination.  This is why I think exception on blocking
> operation is tolerable.

I think I agree.  However, that still leaves us with timeouts, which
are sometimes done for operations that will be retried in the future,
so data loss is not acceptable there.

Won't we also need to augment the existing stdlib APIs to allow the
specification of timeouts for any blocking operations that can't
already be done with non-blocking and select?

-mental
B11f10c4cd9d53970e7be20caa43f940?d=identicon&s=25 Tanaka Akira (Guest)
on 2008-03-25 11:14
(Received via mailing list)
In article <b52213a47751467b7fcc5c5e245fdba4@localhost>,
  MenTaLguY <mental@rydia.net> writes:

> Now, the question with IO and asynchronous exceptions in Ruby is
> whether there is a good way to report a partial result?

I think defining the partial results causes, ultimately,
event driven architecture.

The partial result is similar to a continuation.

For syswrite, it is number of bytes wrote.  It can be used
to resume syswrite.

If IO#gets returns a partial result on an interrupt, it
should be usable to resume IO#gets.  It may contain the
bytes read.  If the IO converts character code, it may
contain characters converted, a state of character code
conversion engine and the bytes not converted yet.

If http request reading procedure returns partial result on
an interrupt, it should be usable to resume the procedure.
It contains a state of http request parser.  If the parser
is recursive decent, it contains the stack.  It is the
continuation.

So defining the partial result of a complex I/O operation
such as http request reading is translating continuation to
some data structure.  It is done by an event driven
architecture.

If a program is not event driven, I think that defining the
partial results for complex I/O operations is too tedious.
B11f10c4cd9d53970e7be20caa43f940?d=identicon&s=25 Tanaka Akira (Guest)
on 2008-03-25 11:19
(Received via mailing list)
In article <7f2500ea06b9d4ae2a8ac86ebf513d10@localhost>,
  MenTaLguY <mental@rydia.net> writes:

> This seems sensible to me, though it would be nice to also have
> an alternate way to handle POSIX signals that didn't involve
> interruption.

Thread.delay_interrupt(klass) { ... } is intended for that.

Hm.  It may need to take "SIGTERM", etc. because Ruby has no
exception class corresponding to single signal, SIGTERM,
etc.
B11f10c4cd9d53970e7be20caa43f940?d=identicon&s=25 Tanaka Akira (Guest)
on 2008-03-25 11:34
(Received via mailing list)
In article <5ec9772607938ba6d0665ce1905904eb@localhost>,
  MenTaLguY <mental@rydia.net> writes:

> I think I agree.  However, that still leaves us with timeouts, which
> are sometimes done for operations that will be retried in the future,
> so data loss is not acceptable there.

For net/http, data loss is not big problem because
timeouted request is not resumed.  The timeout terminates
the http request.

I'm not sure your assumption.  If an operation will be
resumed later, why the operation needs timeout?  What a task
should be done between the timeout and the resuming?  Why
the task should not be done in another thread?

> Won't we also need to augment the existing stdlib APIs to allow the
> specification of timeouts for any blocking operations that can't
> already be done with non-blocking and select?

It is reasonable, I think.
Bec38d63650c8912b6ba9b557fb953b9?d=identicon&s=25 Roger Pack (rogerdpack)
on 2008-03-25 15:32
> Basically you need to have the equivalent of this:
>
>  Thread.uninterruptable do
>    begin
>      Thread.interruptable do
>        # ...
>      end
>    rescue
>      # ...
>    ensure
>      # ...
>    end
>  end

Yeah that would be work.  That or, as one poster put it, only use
exceptions to kill, not to raise (to snuff out the non-determinism and
force users to use queues).  Both of those would work.
Good luck.
-R
912c61d9da47754de7039f4271334a9f?d=identicon&s=25 Mental Guy (mental)
on 2008-03-26 03:08
(Received via mailing list)
Attachment: signature.asc (190 Bytes)
On Tue, 2008-03-25 at 19:33 +0900, Tanaka Akira wrote:
> I'm not sure your assumption.  If an operation will be
> resumed later, why the operation needs timeout?  What a task
> should be done between the timeout and the resuming?  Why
> the task should not be done in another thread?

That's a good point; if another thread is used most of the cases I was
thinking of are unnecessary.  So I think I'm satisfied.

-mental
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.