# Threads, Queues and possible memory leak

**URL:** <https://www.ruby-forum.com/t/threads-queues-and-possible-memory-leak/141499>\
**Category:** Ruby\
**Created:** [June 26, 2008, 1:39pm UTC](https://www.ruby-forum.com/t/threads-queues-and-possible-memory-leak/141499 "2008-06-26T13:39:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![gnufied](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/ea666f/32.png) [@gnufied](https://www.ruby-forum.com/u/gnufied)\
**Post date:** [June 26, 2008, 1:39pm UTC](https://www.ruby-forum.com/t/threads-queues-and-possible-memory-leak/141499/1 "2008-06-26T13:39:07Z")

</div>

I have a simple thread pool of [default] 20 threads running. They read  
a task from a queue and execute it and then go back to waiting on the  
queue. Task popped from the queue contains a block and gets executed  
within thread pool.

Code is here:

require “thread”

class WorkData  
attr\_accessor :data,:block,:job\_key  
def initialize(args,job\_key,&block)  
@data = args  
@job\_key = job\_key  
@block = block  
end  
end

class ThreadPool  
attr\_accessor :size,:threads,:work\_queue  
def initialize(size)  
@size = size  
@threads = []  
@work\_queue = Queue.new  
puts Time.now  
@size.times { add\_thread }  
puts “all threads started: #{Time.now}”  
end  
def defer(\*args,&block)  
job\_key = Thread.current[:job\_key]  
@work\_queue \<\< WorkData.new(args,job\_key,&block)  
end

def add\_thread  
@threads \<\< Thread.new do  
Thread.current[:job\_key] = nil  
while true  
task = @work\_queue.pop  
Thread.current[:job\_key] = task.job\_key  
block\_result = run\_task(task)  
end  
end  
end  
def run\_task task  
block\_arity = task.block.arity  
begin  
t\_data = task.data  
result = nil  
if block\_arity != 0  
result = t\_data.is\_a?(Array) ? (task.block.call(\*t\_data)) :  
(task.block.call(t\_data))  
else  
result = task.block.call  
end  
return result  
rescue  
return nil  
end  
end  
end

class SomeLoop  
attr\_accessor :thread\_pool  
def initialize  
@thread\_pool = ThreadPool.new(20)  
end

def start  
count = 0  
counter = 0  
Signal.trap(“USR1”) { request\_generator }  
loop do  
if count \< 600  
make\_requests()  
sleep(0.05)  
count += 1  
end  
sleep(0.005)  
if (counter%100 == 0 && length != 0)  
puts “Queue Length: #{length}”  
puts “Count : #{count}”  
elsif(length == 0 && count \>= 600)  
puts “All tasks has been executed”  
end  
counter += 1  
end  
end

def request\_generator  
puts “lets make request”  
600.times {  
make\_requests()  
sleep(0.05)  
}  
end

def length  
thread\_pool.work\_queue.length  
end

def some\_job count  
sleep 0.8  
end

def make\_requests  
100.times { |i|  
thread\_pool.defer(i,&method(:some\_job))  
}  
end  
end  
Thread.abort\_on\_exception = true  
a = SomeLoop.new  
a.start

Its being used in a live application and I have managed to extract the  
problematic part.

Now the problem is, as tasks get added in the queue, my program starts  
using more memory and thats fair. But even after all the tasks are  
popped out from the queue the memory stays the same and further  
addition of tasks leads to more memory usage and gets eventually  
restarted by monitoring app.

You can simulate the scenario by running above program and by sending  
“USR1” signal to it. Memory usage just keeps climbing.

I have tried detecting the leak with bleakhouse and can paste the dump  
if useful.  
I have also tried using ‘dike’, but couldn’t get any meaningful dump (  
Ara, may be you can throw some pointers).

Any ideas?

---

<div class="post-metadata">

**Author:** ![gnufied](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/ea666f/32.png) [@gnufied](https://www.ruby-forum.com/u/gnufied)\
**Post date:** [June 26, 2008, 4:32pm UTC](https://www.ruby-forum.com/t/threads-queues-and-possible-memory-leak/141499/2 "2008-06-26T16:32:21Z")

</div>

On Jun 26, 2008, at 5:36 AM, hemant wrote:

> I have also tried using ‘dike’, but couldn’t get any meaningful dump (  
> Ara, may be you can throw some pointers).

will dike you need at least two dumps and then take deltas. typically  
you’d actually look at the deltas of deltas to see long term trends,  
which is a manual process. do you mean you couldn’t dump or that the  
dumps didn’t make sense?

cheers.

a @ [http://codeforpeople.com/](http://codeforpeople.com/)

---

<div class="post-metadata">

**Author:** ![gnufied](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/ea666f/32.png) [@gnufied](https://www.ruby-forum.com/u/gnufied)\
**Post date:** [June 26, 2008, 7:28pm UTC](https://www.ruby-forum.com/t/threads-queues-and-possible-memory-leak/141499/3 "2008-06-26T19:28:11Z")

</div>

> Now the problem is, as tasks get added in the queue, my program starts  
> using more memory and thats fair. But even after all the tasks are  
> popped out from the queue the memory stays the same and further  
> addition of tasks leads to more memory usage and gets eventually  
> restarted by monitoring app.

If you think there’s a ‘real’ memory leak then valgrind is your friend.  
Also trying ruby from svn might help [latest].  
-R

---

<div class="post-metadata">

**Author:** ![gnufied](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/ea666f/32.png) [@gnufied](https://www.ruby-forum.com/u/gnufied)\
**Post date:** [June 26, 2008, 7:46pm UTC](https://www.ruby-forum.com/t/threads-queues-and-possible-memory-leak/141499/4 "2008-06-26T19:46:09Z")

</div>

On 6/26/08, ara.t.howard [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> is a manual process. do you mean you couldn’t dump or that the dumps didn’t  
> make sense?

Well, i didn’t get any dump at all. I got 0,1 files in log directory,  
but they had nothing in it, except [] ( in one of them and other one  
was empty). I will try sometime later today.

> If you think there’s a ‘real’ memory leak then valgrind is your friend.  
> Also trying ruby from svn might help [latest].

I have tried running with ruby1.9 trunk and leaks seems to occur  
really slow. I mean, with some 150 thousand messages in queue, in  
ruby1.8 memory shoots up to 400 MB, but with ruby1.9 it stays around  
52MB.