Forum: Ruby SizedQueue

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Ffc48400d1044e22209aa8fdacc2e053?d=identicon&s=25 Paul Tsung (ptsung)
on 2006-06-03 23:13
Does anyone know about threads and SizedQueue?

I want to limit or control the number of threads created in a script.  I
am not sure what is the best way to handle this.

I wrote a test script using SizedQueue to limit the number of threads.

require 'thread'

@items = %w{item1 item2 item3 item4 item5}
@queue = SizedQueue.new(1)
@thread = []
@nap = 3

def testthread(item)
	@thread << Thread.new(item) do | i |
	  puts Thread.list
		puts "#{i} will sleep for #{@nap} seconds"
		sleep(@nap)
		puts "#{i} woke up"
		@queue.deq
	end
end

@items.each do | item |
	@queue.enq( testthread(item) )
end

@thread.each { | th | th.join }

I set the SizedQueue max to one but the output I get is this.

#<Thread:0x28955c8>
#<Thread:0x28aa420>
item1 will sleep for 3 seconds
#<Thread:0x28aa420>
#<Thread:0x28955c8>
#<Thread:0x28aa0d8>
item2 will sleep for 3 seconds
item2 woke up
item1 woke up
#<Thread:0x28aa420>
#<Thread:0x28955c8>
#<Thread:0x28aa060>
item3 will sleep for 3 seconds
#<Thread:0x28aa060>
#<Thread:0x28955c8>
#<Thread:0x28a98f8>
item4 will sleep for 3 seconds
item4 woke up
item3 woke up
#<Thread:0x28955c8>
#<Thread:0x28a9b08>
item5 will sleep for 3 seconds
item5 woke up

The question is why are there three threads running concurrently from
the output.  I know one is the main thread.  Then I expect only one
other thread to be running.  Is this right or am I doing something
wrong?

Thanks for any suggestions or input.
D1be1ff625c6286f4734cb6019490301?d=identicon&s=25 Vlad GALU (Guest)
on 2006-06-03 23:52
(Received via mailing list)
On 6/4/06, Paul Tsung <ptsung@gmail.com> wrote:
> Does anyone know about threads and SizedQueue?
>
> I want to limit or control the number of threads created in a script.  I
> am not sure what is the best way to handle this.

   Subclass Thread, then use a class variable to hold a thread
counter. Then, simply return from the constructor when the counter
reaches a certain value.
E0d864d9677f3c1482a20152b7cac0e2?d=identicon&s=25 Robert Klemme (Guest)
on 2006-06-04 15:42
(Received via mailing list)
2006/6/3, Paul Tsung <ptsung@gmail.com>:
> Does anyone know about threads and SizedQueue?
>
> I want to limit or control the number of threads created in a script.  I
> am not sure what is the best way to handle this.

You probably haev a misconception on how a SizedQueue works. The limit
on the queue ensures that at most n elements reside in the queue at
any given point in time. You typically do that if the queue is filled
faster than it is emptied. When the queue is full an enqueue operation
blocks the calling thread.

What you probably want is a leigth weight processing framework.  The
basic idea is to have a single queue (sized limited or not) and a
fixed number of threads reading from the queue.

A typical setting:

require 'thread'

MAX_THREADS = 10
QUEUE = Queue.new

threads = (1..MAX_THREADS).map do |i|
  Thread.new(QUEUE) do |q|
    until ( q == ( task = q.deq ) )
      # process task
      print "thread-", i, " ", task.inspect, "\n"
    end
  end
end

# generate tasks
1000.times {|i| QUEUE.enq i}

# send terminators down the queue
threads.size.times { QUEUE.enq QUEUE}

# wait for threads to finish
threads.each {|t| t.join}

# eof

HTH

robert
Ffc48400d1044e22209aa8fdacc2e053?d=identicon&s=25 Paul Tsung (ptsung)
on 2006-06-05 03:22
Robert Klemme wrote:

Thanks Robert.  I will give it a try.

Paul
373e8ca5f9ee2167de04083c9c908b2c?d=identicon&s=25 Lune Lune (lune9)
on 2010-01-22 11:17
Hi Robert

Thanks for your example.

I wanted to play with it to really understand how that works. So I tried
this :
QUEUE = Queue.new
threads = (1..5).map do |a|
   Thread.new(QUEUE) do |q|
      until ( q == (task = q.deq))
        print "thread-", a, " ", task.inspect, "\n"
      end
   end
end

10.times do |t|
  puts "t #{t}"
  30.times { |s| puts "s #{s}"; QUEUE.enq(s) }
  threads.size.times { QUEUE.enq QUEUE }
  threads.each {|t| t.join}
end

In my loop "30 times" 5 * 6 threads are created.
But when I add "10 times" I should see 10 * what I had earlier. This is
not happening. I have my 5 * 6 threads created and that's all. Could you
explain why ?

Thanks
E088bb5c80fd3c4fd02c2020cdacbaf0?d=identicon&s=25 Jesús Gabriel y Galán (Guest)
on 2010-01-22 11:34
(Received via mailing list)
On Fri, Jan 22, 2010 at 11:17 AM, Lune Lune <verduc_m@epitech.net>
wrote:
>        print "thread-", a, " ", task.inspect, "\n"
>
> In my loop "30 times" 5 * 6 threads are created.
> But when I add "10 times" I should see 10 * what I had earlier. This is
> not happening. I have my 5 * 6 threads created and that's all. Could you
> explain why ?

The first time around you insert 30 elements and then one end-of-work
token (the QUEUE itself) per thread. Then you join the threads, so the
main thread will wait at that point. The 5 threads will consume the 30
tokens and eventually consume the end-of-work token and exit. After
that, the main thread goes through the next 9 iterations inserting
items in the queue, but there are no more threads to consume the queue
items.

Jesus.
This topic is locked and can not be replied to.