Ruby embed called from a pthread

Ruby
1

Hello guys,

I’m writing a opensync metamodule in order to allow opensync modules
written in ruby.
It already exists for python but I am a ruby guy. It is almost
complete except to the fact
that is segfalts :slight_smile: I’m using ruby 1.9.1.

Opensync is multithreaded by design. So, my module must accept this
behavior.
I mutexed all ruby access in order to avoid concurrent access of ruby
env.

If this is offtopic for this list, please, sorry and let me know.

Currently, my initialization (ruby_init, RUBY_INIT_STACK,…) run
inside the main thread and all following calls works with no problem.
However, when the first non-main thread GC, it segfalts. Is it allowed
to call ruby from a child thread?

I debugged and what I think that I noticed is that GC scans from
current thread stack to the machine “stack end”. For the main thread,
this is OK. However, when it is a child thread, between the child
stack and the main stack, there is some unallocated memory, which
segfalts on access. Something like this:

[child_stack_start…child_stack_end] unallocated
[main_stack_start…main_stack_end]

If I disable GC, it runs flawless. Is it a possible solution to
disable GC when running a method inside a pthread and reenable it when
main thread calls a method? Would it miss (not free) some objects
inside pthread stack?

One good solution would be some workarround in order to allow calls
and GC from any thread as the one I proposed. Is it possible?

Another great solution would be to allow all ruby stuck exclusively
inside a pthread. This way, I could isolate ruby in a dedicated
pthread with a dedicated stack. All other threads would use ruby
though this thread. However, my current knowledge tells me that ruby
must live inside the main thread (it depends on the main thread stack
end address). Am I right? Is something like running ruby from a single
pthread possible?

The best solution would be to allow ruby initialization inside any
pthread and ruby calls from any pthread. I guess ruby is too far away
from this one :wink:

I was hoping that 1.9.x would help me in any way as it uses pthreads
for ruby threads but I noticed no difference.

I really searched hard to find a solution before getting to the list
but nothing helped. Maybe ruby_bind_stack patch could help but I don’t
think it would get upstream too soon.

Thanks,

 Luiz Angelo Daros de Luca, Me.
        [email protected]
2

Hi,

(2011/06/23 16:37), Luiz Angelo Daros de Luca wrote:

Currently, my initialization (ruby_init, RUBY_INIT_STACK,…) run
inside the main thread and all following calls works with no problem.
However, when the first non-main thread GC, it segfalts. Is it allowed
to call ruby from a child thread?

No.

Ruby’s GC marks machine stack of (registered) all threads. However,
(maybe) your child thread is not registered.

Now, there are no way to register the other thread.

disable GC when running a method inside a pthread and reenable it when
must live inside the main thread (it depends on the main thread stack
I really searched hard to find a solution before getting to the list
but nothing helped. Maybe ruby_bind_stack patch could help but I don’t
think it would get upstream too soon.

One simple solution is making Ruby thread and communicate with queue or
something thread-safe data structure.

I think you can run Ruby interpreter in non-main thread. However, only
one interpreter.

3

Hello SASADA,

Thanks for your help but it still does not work.

 Luiz Angelo Daros de Luca, Me.
        [email protected]

2011/6/23 SASADA Koichi [email protected]:

Ruby’s GC marks machine stack of (registered) all threads. However,

from this one :wink:

I was hoping that 1.9.x would help me in any way as it uses pthreads
for ruby threads but I noticed no difference.

I really searched hard to find a solution before getting to the list
but nothing helped. Maybe ruby_bind_stack patch could help but I don’t
think it would get upstream too soon.

One simple solution is making Ruby thread and communicate with queue or
something thread-safe data structure.

If I understrood correcly, you suggested that I create a thread inside
ruby world and try to communicate with it from C using a threadsafe
data structure. However, because of the ruby global_vm_lock, the
thread I created in ruby world does not run after my code returns. It
never passes this line:

native_mutex_lock(&th->vm->global_vm_lock);

It seems that ruby does not use the global lock for C ruby method
(like rb_funcall, rb_eval). I think that it is used only for thread
stuff while inside ruby, unlocking before passing to another thread
and locking when a thread gets control.

So, when my rb_* call returns, global_vm_lock is locked for the main
thread and my ruby thread is not running.

I also get another try forcing my ruby thread to run (using ruby
ConditionalVariable). However, if it sleeps inside ruby, it will get
the same situation of the first case when my rb_* call returns:
“global_vm_lock is locked for the main thread”. I tried to sleep my
ruby thread inside C world (using an extension call). In this case, it
is my rb_call that never returns because it cannot get global_vm_lock
back.

I still get no solution.

I think you can run Ruby interpreter in non-main thread. However, only
one interpreter.

I tried to run all ruby stuff (RUBY_INIT_STACK, ruby_init… ) from a
single thread but it segfaults after some code is loaded, when
requiring some files. Maybe it is the GC that runs and crashes. I
didn’t debugged too much this one.

Do you have any project or example that runs ruby inside a thread?

Thanks,

4

On Thu, 23 Jun 2011 16:37:44 +0900, Luiz Angelo Daros de Luca wrote:

I mutexed all ruby access in order to avoid concurrent access of ruby
env.

We’ve solved this problem, which arised while trying to launch MRI as a
thread
in NodeJS. You can take a look at the code:
coldruby/libcoldruby/MRIRubyCompiler.cpp at master · whitequark/coldruby · GitHub

5

Thanks, I’ll take a look.

 Luiz Angelo Daros de Luca, Me.
        [email protected]

2011/6/23 Peter Z. [email protected]:

6

Luiz Angelo Daros de Luca wrote in post #1007025:

I debugged and what I think that I noticed is that GC scans from
current thread stack to the machine “stack end”. For the main thread,
this is OK. However, when it is a child thread, between the child
stack and the main stack, there is some unallocated memory, which
segfalts on access. Something like this:

[child_stack_start…child_stack_end] unallocated
[main_stack_start…main_stack_end]

Correct. I documented this issue in more detail here:

Feature #2294: [PATCH] ruby_bind_stack() to embed Ruby in coroutine - Ruby master - Ruby Issue Tracking System

The best solution would be to allow ruby initialization inside any
pthread and ruby calls from any pthread. I guess ruby is too far away
from this one :wink:

Indeed, it is possible to run Ruby 1.9 inside a pthread, ucontext, or
libpcl environment tied to a statically or dynamically allocated stack:

GitHub - sunaku/ruby-coroutine-example: How to embed Ruby inside C coroutines.

Cheers.

7

Wow Peter, what a pretty hack :slight_smile: It really works! Thanks again.
Shouldn’t ruby provide something like this? (but not hacky)

After a long period of developing, I finally soved all my ruby
problems. As a shared library, I receive call from many threads. So,
when the first thread needs ruby, I spaws a new pthread dedicated to
ruby and initialize all its stuff. I created some wrapper for each of
my C methods. When my method is called, I write its arguments to a
global variable, asks my ruby pthread (not a ruby thread) to run it
and it wakes me when it is done. All ruby calls uses the Peter’s Hack.

Again, maybe ruby could already provide something similar to my
solution.

Anyway, it now works!

 Luiz Angelo Daros de Luca, Me.
        [email protected]

2011/6/23 Peter Z. [email protected]: