Forum: Ruby-core [Ruby 1.9 - Feature #4614][Open] [RFC/PATCH] thread_pthread.c: lower RUBY_STACK_MIN_LIMIT to 64K

Issue #4614 has been reported by Eric Wong.

----------------------------------------
Feature #4614: [RFC/PATCH] thread_pthread.c: lower RUBY_STACK_MIN_LIMIT 
to 64K
http://redmine.ruby-lang.org/issues/4614

Author: Eric Wong
Status: Open
Priority: Low
Assignee:
Category: core
Target version: 1.9.x


The patch (committed) for Issue #4568 made this change possible.

Lowering stack size allows users on memory-constrained 32-bit
machines to spawn more native threads (which are easier (IMHO)
to use than Fibers).

Setting RUBY_STACK_MIN_LIMIT to 48K seems to work even with
check, test-rubyspec, and benchmark-each targets.  However, I'm
keeping it at 64K since that is what Symbian uses, so maybe
there's some code paths that need 64K.

I started experiencing failures with the Ruby tests with 44K on
x86_64, so 44K is definitely not safe.  Much more effort would
be required to make Ruby work with smaller stacks than 48K.

Also pullable via git: git pull git://bogomips.org/ruby.git stack-reduce

Issue #4614 has been updated by Koichi Sasada.

Assignee set to Koichi Sasada

GC eats large stack size if there are nested object (deep nested array 
and so on).  I'm not sure but I think this is why we allocate 512KB for 
stack.

Any comments?

I think that Thread.new should accept stack size.
----------------------------------------
Feature #4614: [RFC/PATCH] thread_pthread.c: lower RUBY_STACK_MIN_LIMIT 
to 64K
http://redmine.ruby-lang.org/issues/4614

Author: Eric Wong
Status: Open
Priority: Low
Assignee: Koichi Sasada
Category: core
Target version: 1.9.x


=begin
The patch (committed) for Issue #4568 made this change possible.

Lowering stack size allows users on memory-constrained 32-bit
machines to spawn more native threads (which are easier (IMHO)
to use than Fibers).

Setting RUBY_STACK_MIN_LIMIT to 48K seems to work even with
check, test-rubyspec, and benchmark-each targets.  However, I'm
keeping it at 64K since that is what Symbian uses, so maybe
there's some code paths that need 64K.

I started experiencing failures with the Ruby tests with 44K on
x86_64, so 44K is definitely not safe.  Much more effort would
be required to make Ruby work with smaller stacks than 48K.

Also pullable via git: git pull git://bogomips.org/ruby.git stack-reduce

=end

Koichi Sasada <redmine@ruby-lang.org> wrote:
> GC eats large stack size if there are nested object (deep nested array
> and so on).  I'm not sure but I think this is why we allocate 512KB
> for stack.

Good point.  I didn't think of that (rare case for me).  We would need a
non-recursive implementation of gc_mark_children.

A simpler idea would be a dedicated marking thread with a deeper stack;
but I think that's ugly.

> Any comments?
>
> I think that Thread.new should accept stack size.

I'm not sure what the API would be.  While it would help me, I think it
would be difficult to use and too low level for Ruby.

Meanwhile I can rebuild Ruby or use a trivial LD_PRELOAD:
http://yhbt.net/libministack.c

Issue #4614 has been updated by ko1 (Koichi Sasada).

Status changed from Assigned to Feedback
Priority changed from Low to High

Ad-hoc solution, but environment variable (such as 
RUBY_THREAD_MACHINE_STACK_SIZE) is enough for this issue?

----------------------------------------
Feature #4614: [RFC/PATCH] thread_pthread.c: lower RUBY_STACK_MIN_LIMIT 
to 64K
https://bugs.ruby-lang.org/issues/4614#change-31661

Author: normalperson (Eric Wong)
Status: Feedback
Priority: High
Assignee: ko1 (Koichi Sasada)
Category: core
Target version: 2.0.0


=begin
The patch (committed) for Issue #4568 made this change possible.

Lowering stack size allows users on memory-constrained 32-bit
machines to spawn more native threads (which are easier (IMHO)
to use than Fibers).

Setting RUBY_STACK_MIN_LIMIT to 48K seems to work even with
check, test-rubyspec, and benchmark-each targets.  However, I'm
keeping it at 64K since that is what Symbian uses, so maybe
there's some code paths that need 64K.

I started experiencing failures with the Ruby tests with 44K on
x86_64, so 44K is definitely not safe.  Much more effort would
be required to make Ruby work with smaller stacks than 48K.

Also pullable via git: git pull git://bogomips.org/ruby.git stack-reduce

=end

"ko1 (Koichi Sasada)" <redmine@ruby-lang.org> wrote:
> Ad-hoc solution, but environment variable (such as
> RUBY_THREAD_MACHINE_STACK_SIZE) is enough for this issue?

I think environment variable is too ugly, but it's your (or matz's)
choice to support it.

Also, it seems the recursive structure problem with gc_mark_children()
is gone since r37075.

Probably better to discuss #6695 instead.

Issue #4614 has been updated by mame (Yusuke Endoh).


ko1, what's the status?

Too many 2.0.0 tickets, especially big task ones, are assigned to ko1. 
ko1, please decide your priorities.

Welcome to volunteer for helping him.

--
Yusuke Endoh <mame@tsg.ne.jp>
----------------------------------------
Feature #4614: [RFC/PATCH] thread_pthread.c: lower RUBY_STACK_MIN_LIMIT 
to 64K
https://bugs.ruby-lang.org/issues/4614#change-33700

Author: normalperson (Eric Wong)
Status: Feedback
Priority: High
Assignee: ko1 (Koichi Sasada)
Category: core
Target version: 2.0.0


=begin
The patch (committed) for Issue #4568 made this change possible.

Lowering stack size allows users on memory-constrained 32-bit
machines to spawn more native threads (which are easier (IMHO)
to use than Fibers).

Setting RUBY_STACK_MIN_LIMIT to 48K seems to work even with
check, test-rubyspec, and benchmark-each targets.  However, I'm
keeping it at 64K since that is what Symbian uses, so maybe
there's some code paths that need 64K.

I started experiencing failures with the Ruby tests with 44K on
x86_64, so 44K is definitely not safe.  Much more effort would
be required to make Ruby work with smaller stacks than 48K.

Also pullable via git: git pull git://bogomips.org/ruby.git stack-reduce

=end

Issue #4614 has been updated by ko1 (Koichi Sasada).


Priority: *High*.

----------------------------------------
Feature #4614: [RFC/PATCH] thread_pthread.c: lower RUBY_STACK_MIN_LIMIT 
to 64K
https://bugs.ruby-lang.org/issues/4614#change-33907

Author: normalperson (Eric Wong)
Status: Feedback
Priority: High
Assignee: ko1 (Koichi Sasada)
Category: core
Target version: 2.0.0


=begin
The patch (committed) for Issue #4568 made this change possible.

Lowering stack size allows users on memory-constrained 32-bit
machines to spawn more native threads (which are easier (IMHO)
to use than Fibers).

Setting RUBY_STACK_MIN_LIMIT to 48K seems to work even with
check, test-rubyspec, and benchmark-each targets.  However, I'm
keeping it at 64K since that is what Symbian uses, so maybe
there's some code paths that need 64K.

I started experiencing failures with the Ruby tests with 44K on
x86_64, so 44K is definitely not safe.  Much more effort would
be required to make Ruby work with smaller stacks than 48K.

Also pullable via git: git pull git://bogomips.org/ruby.git stack-reduce

=end

Issue #4614 has been updated by ko1 (Koichi Sasada).

Priority changed from High to Normal

I add environment variable at r38479.

      - RUBY_THREAD_VM_STACK_SIZE: vm stack size used at thread 
creation.
        default: 128KB (32bit CPU) or 256KB (64bit CPU).
      - RUBY_THREAD_MACHINE_STACK_SIZE: machine stack size used at 
thread
        creation. default: 512KB or 1024KB.
      - RUBY_FIBER_VM_STACK_SIZE: vm stack size used at fiber creation.
        default: 64KB or 128KB.
      - RUBY_FIBER_MACHINE_STACK_SIZE: machine stack size used at fiber
        creation. default: 256KB or 256KB.

I want to ask feedback about this default setting.


----------------------------------------
Feature #4614: [RFC/PATCH] thread_pthread.c: lower RUBY_STACK_MIN_LIMIT 
to 64K
https://bugs.ruby-lang.org/issues/4614#change-34870

Author: normalperson (Eric Wong)
Status: Feedback
Priority: Normal
Assignee: ko1 (Koichi Sasada)
Category: core
Target version: 2.0.0


=begin
The patch (committed) for Issue #4568 made this change possible.

Lowering stack size allows users on memory-constrained 32-bit
machines to spawn more native threads (which are easier (IMHO)
to use than Fibers).

Setting RUBY_STACK_MIN_LIMIT to 48K seems to work even with
check, test-rubyspec, and benchmark-each targets.  However, I'm
keeping it at 64K since that is what Symbian uses, so maybe
there's some code paths that need 64K.

I started experiencing failures with the Ruby tests with 44K on
x86_64, so 44K is definitely not safe.  Much more effort would
be required to make Ruby work with smaller stacks than 48K.

Also pullable via git: git pull git://bogomips.org/ruby.git stack-reduce

=end

"ko1 (Koichi Sasada)" <redmine@ruby-lang.org> wrote:
> I add environment variable at r38479.

Thanks!

Btw, you put a giant "if false" around test/ruby/test_fiber.rb
with that commit.  Was that intentional?

> I want to ask feedback about this default setting.

I'm fine with the current defaults (including the r38592 update)

Issue #4614 has been updated by ko1 (Koichi Sasada).

Status changed from Feedback to Closed

Sorry I missed your comment.
And the `if false' sentences may be removed.


----------------------------------------
Feature #4614: [RFC/PATCH] thread_pthread.c: lower RUBY_STACK_MIN_LIMIT 
to 64K
https://bugs.ruby-lang.org/issues/4614#change-36205

Author: normalperson (Eric Wong)
Status: Closed
Priority: Normal
Assignee: ko1 (Koichi Sasada)
Category: core
Target version: 2.0.0


=begin
The patch (committed) for Issue #4568 made this change possible.

Lowering stack size allows users on memory-constrained 32-bit
machines to spawn more native threads (which are easier (IMHO)
to use than Fibers).

Setting RUBY_STACK_MIN_LIMIT to 48K seems to work even with
check, test-rubyspec, and benchmark-each targets.  However, I'm
keeping it at 64K since that is what Symbian uses, so maybe
there's some code paths that need 64K.

I started experiencing failures with the Ruby tests with 44K on
x86_64, so 44K is definitely not safe.  Much more effort would
be required to make Ruby work with smaller stacks than 48K.

Also pullable via git: git pull git://bogomips.org/ruby.git stack-reduce

=end