On Mon, May 16, 2011 at 3:19 AM, Zd Yu [email protected] wrote:
- the main thread open the input file and pass the file object to each
- the main thread send a “GO” command to all the working threads
Did you ever hear of blocking queues?
5.2 check all the working threads’ [:result] and extract them out and
do some aggregation.
5.3 for those threads whose result has been picked, send a signal to
let them proceed with the next line.
6. once all the working threads finish, the main thread output the
This sounds like a very typical application of farmer worker. You
create two queues, one for tasks and one for results. Then you start
a thread which fetches results from the result queue and processes
them. Then you start a number of threads which read from the tasks
queue, process tasks and place results in the result queue. Finally
you use Thread#value to join on the result processor.
WORKERS = 5
tasks = SizedQueue.new WORKERS * 10
results = SizedQueue.new WORKERS * 10
agg = Thread.new do
state = Hash.new 0
th = WORKERS
while th > 0
x = results.deq
if Thread === x
th -= 1
state[x] += 1
workers = WORKERS.times.map do
until (it = tasks.deq) == tasks