C Threads and Ruby

Hi,

i know the ruby interpreter isn’t thread safe and
it’s not a good idea to call rb_* functions from more
than one thread.

Nevertheless, is there a way to signal the interpreter
that i would like him to call one of my functions in a
c extension? (from another thread)

The only way i figured out so far would be to create a
pipe, let a ruby thread sleep on the fd and put a byte
in the queue from c the wake the thread. (good, bad, ugly?)

cheers

Simon

On 6/20/06, Kroeger, Simon (ext) [email protected] wrote:

The only way i figured out so far would be to create a
pipe, let a ruby thread sleep on the fd and put a byte
in the queue from c the wake the thread. (good, bad, ugly?)

cheers

Simon

It depends what you are trying to do. I just worked up a patch to
win32-service, that does this by creating a ruby thread in the C
extension that polls against a simple integer guarded by a mutex –
when an event occurs it is signaled there and the correct call back is
spun off in yet another ruby thread. I was worried this would be
“expensive”, but the rb_thread_polling method, worked like a charm.

This idea could be easily extended to allow for an event queue that
was filled by a native C thread and processed by a ruby thread. Note
that calling any rb_XXXX from anything other than the “ruby thread” is
a really bad idea.

pth

From: Patrick H. [mailto:[email protected]]
Sent: Tuesday, June 20, 2006 1:29 PM

It depends what you are trying to do. I just worked up a patch to
win32-service, that does this by creating a ruby thread in the C
extension that polls against a simple integer guarded by a mutex –
when an event occurs it is signaled there and the correct call back is
spun off in yet another ruby thread. I was worried this would be
“expensive”, but the rb_thread_polling method, worked like a charm.

Yeah, I’m doing something simmilar with rb_thread_wait_for instead of
rb_thread_polling. rb_thread_polling does sleep for 6ms. While this may
be acceptable it is not exactly what I was looking for. (I’m sleeping
for
only 1ms and there is no noticeable cpu requirement)

Perhaps my problem is: I just hate polling.

This idea could be easily extended to allow for an event queue that
was filled by a native C thread and processed by a ruby thread. Note
that calling any rb_XXXX from anything other than the “ruby thread” is
a really bad idea.

pth

thanks anyway,

Simon

Patrick: given your proposal for an event queue filled by native threads
in
C and consumed by a Ruby thread, notice that there is no mutex that
works
between native threads and Ruby threads. Your example of “signalling” by
changing the value of an integer probably works because on an Intel chip
(you said you did this on Windows), it only takes one memory-bus cycle
to
change the value of a 32-bit integer, so you can get away without a
mutex. I
doubt this would work with a more complex data structure.

Simon: the pipe idea will work (I used it in an early version of
eventmachine), but there are a few gotchas. First, the IO objects (pipe
and
socketpair, can’t remember exactly which one I used) are full-duplex on
Unix
but half-duplex on Windows. Next, you still won’t be able to synchronize
between Ruby and native threads. I infer from your description that you
want
Ruby to sleep while your native thread(s) run, and then to call a native
function. Presumably this will happen while other native threads
continue to
run.

I thought about writing an extension that would allow Ruby code to
access
the native mutexes and other synchronization objects, and I think one of
the
core guys has worked on this. It would be a bit tricky because condvars
don’t work perfectly on Windows, but you could wrap the native Windows
objects if you really had to.

On 6/20/06, Francis C. [email protected] wrote:

Patrick: given your proposal for an event queue filled by native threads in
C and consumed by a Ruby thread, notice that there is no mutex that works
between native threads and Ruby threads. Your example of “signalling” by
changing the value of an integer probably works because on an Intel chip
(you said you did this on Windows), it only takes one memory-bus cycle to
change the value of a 32-bit integer, so you can get away without a mutex. I
doubt this would work with a more complex data structure.

That is not true, it is completely possible to use an OS level
mutex/critical section etc, from inside a “green” Ruby thread. It will
block all Ruby threads which is not very nice, but I do not know of a
better solution.

I used a system mutex (actually a CriticalSection) in both threads –
yes it blocks ruby, but only long enough to read the memory. It would
be better if the Ruby C API provide a thread safe wait for non-native
threads – but this does work and is completely thread safe even if I
were not doing an atomic operation (for example working off a linked
list).

pth

On 6/20/06, Paul B. [email protected] wrote:

I typically use a pipe for this kind of synchronization.

Paul

I would agree a pipe is generally better; however, in my case it was
against a Win32 API that required the use of threads. Now I am curious
about TRAP_BEG/TRAP_END. Is it safe to access rb_XXX methods from
inside a TRAP_BEG/TRAP_END block not in the ruby interpreter thread?
If so that would make some things much easier.

Thanks
pth

That is not true, it is completely possible to use an OS level
mutex/critical section etc, from inside a “green” Ruby thread. It will
block all Ruby threads which is not very nice, but I do not know of a
better solution.<<<

That’s why I said you can’t do it ;-). It indeed is not very nice unless
you’re very disciplined about how you design your synchronization sets.
(And
one of my pet peeves is that most people have no clue how to do this
right.)
Bad possibility is crummy performance. Worse yet is deadlocks, which are
possible if the native code relies on something else Ruby is doing.

On Wed, Jun 21, 2006 at 01:39:53AM +0900, Patrick H. wrote:

That is not true, it is completely possible to use an OS level
mutex/critical section etc, from inside a “green” Ruby thread. It will
block all Ruby threads which is not very nice, but I do not know of a
better solution.

TRAP_BEG and TRAP_END work well for allowing Ruby to switch threads,
though the call between them must be interruptible. I believe this
should be true of sem_wait(), though I’ve been unsuccessful interrupting
it on linux.

I typically use a pipe for this kind of synchronization.

Paul

I typically use a pipe for this kind of synchronization.

Me too. Crude but it works.

Simon Kröger wrote:

Thread.new do
loop do
puts ‘foo!’

Might help:

  wr.puts 'foo!'

Well, b*****it:


$stdout.sync = true
rd, wr = IO.pipe

Thread.new do
loop do
puts ‘foo!’
sleep 0.5
end
end

sleep 2
puts(rd.gets)

<<<

Joel already caught the obvious error in this code. It’s also possible
you’ll hit a platform-dependence with half/full duplex descriptors. Are
you
using Windows or Unix-like?

Francis C. schrieb:

I typically use a pipe for this kind of synchronization.

Me too. Crude but it works.

Well, b*****it:
(sorry, no offense intended…)


$stdout.sync = true
rd, wr = IO.pipe

Thread.new do
loop do
puts ‘foo!’
sleep 0.5
end
end

sleep 2
puts(rd.gets)

ruby 1.8.4 (2005-12-24) [i386-mswin32]
gives me 4 foo!s and waits for the rest of eternity.
So blocking on a pipe isn’t an option for platform independent code.
(or am I doing something wrong, please say I’m doing something wrong!)

cheers

Simon

From: “Simon Kröger” [email protected]

end

sleep 2
puts(rd.gets)

ruby 1.8.4 (2005-12-24) [i386-mswin32]
gives me 4 foo!s and waits for the rest of eternity.
So blocking on a pipe isn’t an option for platform independent code.
(or am I doing something wrong, please say I’m doing something wrong!)

As far as I know, on mswin32 ruby, the only genuine nonblocking
I/O is with sockets.

I, too, have need to wake up a ruby thread from a native thread
in my application. Currently I’m using polling.

My thinking was to use UDP over loopback. My ruby thread would
sleep waiting for the C thread to send it a UDP packet.

I haven’t tried this yet. I’m certain it would work, technically,
but I’m wary of it because I’ve had experiences in the past where
simply creating a socket (even on localhost) causes the user’s
computer to decide to take the modem off-hook and try to connect
to their AOL account… :rolleyes:

(This was about six years ago. I’m not sure if Windows still
behaves like that with dialup connections or not.)

Regards,

Bill

The problem isn’t that rd.gets waits forever - i didn’t described my
problem well enough obviously. The problem is: i get 4 foo!s not one
more. Not only the main thread blocks at the gets but all other ruby
threads do also block. This defeats the whole purpose of using a pipe
in the first place.
<<<

Yes, this does fail on Windows as written (works fine on Unix), but it
has
nothing to do with threads. Take out the thread altogether and it will
still
block forever.

Try writing an extension in C and create your descriptors in native
Win32
code. Those will definitely work with Ruby on both Windows and Unix
(I’ve
done it). You may have to create two pairs because they will be
half-duplex
on Windows.

Francis C. wrote:

end

sleep 2
puts(rd.gets)

<<<

Joel already caught the obvious error in this code. It’s also possible
you’ll hit a platform-dependence with half/full duplex descriptors. Are you
using Windows or Unix-like?

Thanks to Joel and you but no.

The problem isn’t that rd.gets waits forever - i didn’t described my
problem well enough obviously. The problem is: i get 4 foo!s not one
more. Not only the main thread blocks at the gets but all other ruby
threads do also block. This defeats the whole purpose of using a pipe
in the first place.

And yes: ruby 1.8.4 (2005-12-24) [i386-mswin32] is on windows.

cheers

Simon

From: “Francis C.” [email protected]

Try this: create a pipe or socketpair in native code. Then you can create a
ruby IO object from the file-descriptor of the read-side of the pipe. Then
call Ruby’s select (NOT the native select) on the IO object. When it selects
readable, you run your Ruby code. This blocks neither your native threads
nor your other Ruby threads.

I know this works in both Unix and Windows, but the native code is slightly
different.

Cool. I didn’t realize one could wrap a native win32 pipe
handle in some way that will work with ruby’s select!

The impression I got from the windows select() documentation
was that it only works with winsock sockets.

This makes me wonder why ruby wouldn’t already be creating
the “proper kind” of pipes and socketpairs that work with its
select(), if it’s that easy to create one myself and wrap it
in a ruby IO object?

Strange . . .

Thanks,

Bill

I, too, have need to wake up a ruby thread from a native thread
in my application. Currently I’m using polling.

My thinking was to use UDP over loopback. My ruby thread would
sleep waiting for the C thread to send it a UDP packet.
<<<

Try this: create a pipe or socketpair in native code. Then you can
create a
ruby IO object from the file-descriptor of the read-side of the pipe.
Then
call Ruby’s select (NOT the native select) on the IO object. When it
selects
readable, you run your Ruby code. This blocks neither your native
threads
nor your other Ruby threads.

I know this works in both Unix and Windows, but the native code is
slightly
different.

The impression I got from the windows select() documentation
was that it only works with winsock sockets.
<<<

Ruby’s Kernel#select is quite different from the native select. It’s
deeply
intertwined with the green threads implementation. It basically knows
how to
handle anything that can be wrapped in a Ruby IO object.

From: Francis C. [mailto:[email protected]]
Sent: Wednesday, June 21, 2006 12:26 AM

and it will still
block forever.

Well… ??? yes of course, if you block the one and only thread
on an IO operation that never returns your program will block
forever. This should be true for linux also, right?

So the problem is that all threads block because of this IO
operation, how could that be unrelated to threads?

Try writing an extension in C and create your descriptors in
native Win32
code. Those will definitely work with Ruby on both Windows
and Unix (I’ve
done it). You may have to create two pairs because they will
be half-duplex
on Windows.

native Win32 means calling ‘CreatePipe’ instead of ‘_pipe’ ?
do you have code for a platform independent implementation?
(using #ifdef)

for the record: ruby 1.9.0 (2006-04-15) [i386-mswin32] seems
to have a fix for that problem.

cheers

Simon

Simon:

Well… ??? yes of course, if you block the one and only thread
on an IO operation that never returns your program will block
forever. This should be true for linux also, right?
<<<

It has nothing to do with threads. I only took a quick look at the Ruby
source code, but I suspect there is a bug in the Windows implementation.
Remember, I got the same result as you did even when I took out your
thread.
And it did work fine on both Linux and OSX. Ruby’s green threads
implementation does not block the whole process when waiting for an I/O
because its scheduler interacts with multiplexed I/O.

native Win32 means calling ‘CreatePipe’ instead of ‘_pipe’ ?
do you have code for a platform independent implementation?
(using #ifdef)
<<<

Windows does have Unix-like calls for doing this. (They’re wrappers over
Win32 APIs.) I forget which one I’ve used but I think it was pipe().
Also, there may be something in Daniel B.'s win32 library that may
help
you.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs