# Memory leak problem

**URL:** <https://www.ruby-forum.com/t/memory-leak-problem/229179>\
**Category:** Ruby\
**Created:** [October 8, 2008, 6:41pm UTC](https://www.ruby-forum.com/t/memory-leak-problem/229179 "2008-10-08T18:41:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![elioncho](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/ed8c4c/32.png) [@elioncho](https://www.ruby-forum.com/u/elioncho)\
**Post date:** [October 8, 2008, 6:41pm UTC](https://www.ruby-forum.com/t/memory-leak-problem/229179/1 "2008-10-08T18:41:08Z")

</div>

I have a txt file with some data that i need to import to de database.  
I’m using ruby to import those data but i have a major problem, when i  
've  
a large amount of data i run out of memory.

File.open("#{RAILS\_ROOT}/public/files/Neighborsville.TXT").each() do  
|line|  
@stringArray = line.split("|")  
@i += 1  
puts @i  
@pid = @stringArray[0]  
@chain\_id = @stringArray[1]  
@business = Business.find\_by\_pid\_and\_chain\_id(@pid,@chain\_id);  
#Check PID + CHAIN\_ID  
@business.pid = @stringArray[0]  
@business.chain\_id = @stringArray[1]  
@business.cityname = @stringArray[17]  
@business.state = @stringArray[18]  
@business.business =  
Business.find\_by\_pid\_and\_chain\_id(@pid,@chain\_id);  
@business.city = City.new  
@business.business\_category = get\_category\_id(@stringArray[40])  
@business.address = @stringArray[8] +" “+ @stringArray[9] +” “+  
@stringArray[10]+” “+ @stringArray[11] +” “+@stringArray[12]+”  
“+@stringArray[13]+” "+@stringArray[14]  
if @chain\_id == nil  
@chain\_id = “”  
end  
business.save  
end  
end

I belive that ruby use in every cycle of the do new blocks of memories  
por my instances  
of Business. Can someone help me please?

Thanks,

Elioncho

---

<div class="post-metadata">

**Author:** ![elioncho](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/ed8c4c/32.png) [@elioncho](https://www.ruby-forum.com/u/elioncho)\
**Post date:** [October 10, 2008, 7:24am UTC](https://www.ruby-forum.com/t/memory-leak-problem/229179/2 "2008-10-10T07:24:19Z")

</div>

Elias O. wrote:

> I have a txt file with some data that i need to import to de database.  
> I’m using ruby to import those data but i have a major problem, when i  
> 've  
> a large amount of data i run out of memory.
> 
> File.open("#{RAILS\_ROOT}/public/files/Neighborsville.TXT").each() do  
> |line|  
> @stringArray = line.split("|")  
> @i += 1  
> puts @i  
> @pid = @stringArray[0]  
> @chain\_id = @stringArray[1]  
> @business = Business.find\_by\_pid\_and\_chain\_id(@pid,@chain\_id);  
> #Check PID + CHAIN\_ID  
> @business.pid = @stringArray[0]  
> @business.chain\_id = @stringArray[1]  
> @business.cityname = @stringArray[17]  
> @business.state = @stringArray[18]  
> @business.business =  
> Business.find\_by\_pid\_and\_chain\_id(@pid,@chain\_id);  
> @business.city = City.new  
> @business.business\_category = get\_category\_id(@stringArray[40])  
> @business.address = @stringArray[8] +" “+ @stringArray[9] +” “+  
> @stringArray[10]+” “+ @stringArray[11] +” “+@stringArray[12]+”  
> “+@stringArray[13]+” "+@stringArray[14]  
> if @chain\_id == nil  
> @chain\_id = “”  
> end  
> business.save  
> end  
> end
> 
> I belive that ruby use in every cycle of the do new blocks of memories  
> for my instances  
> of Business.

Yes that’s right. You’ve possibly run into an infamous “ruby’s GC is  
broken!” bug. Then again, I could be wrong.  
Question:  
don’t you want @business.save?

Anyway ways to avoid this:  
you may be able to use ar extensions, which allows for multiple inserts  
into the DB. Oh wait, except that you are doing multiple updates. Never  
mind.  
In that case, as gross as it seems, you could try forking once per loop  
[or once every x lines of the input file]–that way the forked process  
will die [with its high RAM consumption] allowing the parent process to  
continue [and fork more].

require ‘forkoff’  
File.open("#{RAILS\_ROOT}/public/files/Neighborsville.TXT").each() do  
|line|  
[1].forkoff {  
# do your stuff  
}  
end

Maybe it won’t help. I know with huge data sets it helps me avoid the  
GC. I’ve never tried it with sql and rails.  
Good luck.  
-=R

---

<div class="post-metadata">

**Author:** ![elioncho](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/ed8c4c/32.png) [@elioncho](https://www.ruby-forum.com/u/elioncho)\
**Post date:** [October 10, 2008, 9:55am UTC](https://www.ruby-forum.com/t/memory-leak-problem/229179/3 "2008-10-10T09:55:45Z")

</div>

On Oct 9, 3:39 am, Elias O. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> ```
> @pid = @stringArray[0]
> @business.business_category = get_category_id(@stringArray[40])
> 
> ```
> 
> I belive that ruby use in every cycle of the do new blocks of memories  
> por my instances  
> of Business. Can someone help me please?
> 
> Thanks,
> 
> ## Elioncho
> 
> Posted viahttp://www.ruby-forum.com/.

I would suggest that you don’t use member variables in this situation  
if it is possible (ie ‘business’ rather than ‘@business’, ‘i’ rather  
then ‘@i’). It doesn’t look like you need member variables here and it  
could will mean that the GC won’t collect some memory that it  
otherwise could.

You could also try a patched version of ruby with better memory  
collection ([http://lloydforge.org/projects/ruby/](http://lloydforge.org/projects/ruby/),  
[Online Project Management and Scrum Software - Acunote Blog](http://blog.pluron.com/2008/01/ruby-on-rails-i/comments/page/2/) or  
[http://www.rubyenterpriseedition.com/](http://www.rubyenterpriseedition.com/))

Also try explicitly calling GC.start at the beginning of every loop  
(or every few loops). This will slow your code down a lot but I’ve  
occasionally seen cases where it has helped.

All that said, the Ruby GC is a bit crap and you might just have use  
the fork approach suggested by Roger.

Dan