WIN32OLE memory leaks

In my ruby code I am making a call out to a COM object that is
returning an array of objects. I extract some information from these
objects and then request a new batch to do the same work all over again.

Even though I am clearing the received array out, my program slowly
leaks memory until it hits its limit (around 1.4 GB on Windows) and
dies.

I have tried adding my_object.ole_free calls everywhere, but that has
not helped at all. I’ve looked through the archives (back to 2004) and
saw that this was a common issue years ago. I had hoped it would be
fixed by now.

Does anyone have any suggestions on how to force these unused objects
to get GC’ed? This program needs to run a long time so a memory leak
is absolutely fatal.

cr

On Tue, Jan 12, 2010 at 4:19 PM, Chuck R. [email protected]
wrote:

Does anyone have any suggestions on how to force these unused objects to get
GC’ed? This program needs to run a long time so a memory leak is absolutely
fatal.

Some code might be helpful (and maybe a reference to the old article
that you think is the same issue?)

(but, I don’t have a Windows box to test it on for you anyways)

Would it be possible to run each batch in a new ruby process?

On Jan 12, 6:19 pm, Chuck R. [email protected] wrote:

saw that this was a common issue years ago. I had hoped it would be
fixed by now.

Does anyone have any suggestions on how to force these unused objects
to get GC’ed? This program needs to run a long time so a memory leak
is absolutely fatal.

Please include exact and complete version of Ruby you’re using (ruby -
v) and under which Windows flavor.

Also, if this can be reproducible with any COM object, include a
sample script for us to reproduce in other Windows version and Ruby
version combinations.

Once we have that and we can confirm, I’ll recommend submitting a bug
report to Ruby own bug tracking system:

There select the proper version and report the issue.

On Jan 12, 2010, at 4:10 PM, Luis L. wrote:

I have tried adding my_object.ole_free calls everywhere, but that has
v) and under which Windows flavor.

Also, if this can be reproducible with any COM object, include a
sample script for us to reproduce in other Windows version and Ruby
version combinations.

Sorry for not providing more details. I actually thought this might be
a “known” issue.

Here’s some code. This is running against 1.8.6-p383 (from
rubyinstaller.org) on Windows XP SP3 plus all the latest patches
(Windows Update runs weekly).

def register_callbacks
event.on_event(‘DataResolved’) do |collection, error|
save collection unless error
@flag1 = false
end
end

def save collection
collection.each do |data|
doc = {
‘field1’ => data.Property1.to_i,
‘field2’ => data.Property2.to_f,
‘field3’ => data.Property3
}

 @mongo_collection.insert doc
 data.ole_free

end
collection.ole_free
end

def request_data start_date, end_date
@start_date = start_date
@end_date = end_date

@flag1 = true

request = @com.CreateDataRequest

range_start_method = request.ole_method_help ‘RangeStart’
range_end_method = request.ole_method_help ‘RangeEnd’

request._setproperty range_start_method.dispid, [@start_date],
[VT_DATE]
request._setproperty range_end_method.dispid, [@end_date], [VT_DATE]

@com.RequestData request
end

I have a method that makes a function call on the COM object
(#request_data). When the function has data to return, it generates a
“DataResolved” event which triggers my callback which was registered
once at the beginning of execution. The callback calls #save and loops
through the collection and pulls some properties into a hash which
then gets inserted into mongodb. When the collection is exhausted, I
go back to #request_data and make another call on the COM object to
request the next batch.

The collection returned is usually around 30MB. I don’t leak all 30MB
each time through; it leaks a big percentage of it though (30 to 50%).

When Task Manager tells me the ruby.exe process is taking up around
1.4GB, the process just exits.

Unfortunately this application is not available to the general public
so I can’t provide a script for someone to independently verify this. :frowning:

cr

On Wed, Jan 13, 2010 at 10:55 AM, Chuck R. [email protected]
wrote:

Please include exact and complete version of Ruby you’re using (ruby -
v) and under which Windows flavor.

Here’s some code. This is running against 1.8.6-p383 (from
rubyinstaller.org) on Windows XP SP3 plus all the latest patches (Windows
Update runs weekly).

Are you able to do a test run with another ruby build, e.g. 1.8.7/1.9.1
from:
Download Ruby

You see the same behavior even skipping the interaction with mongo?

Running it with 1.9.1-p243 had the same results. I couldn’t figure out how
to install the “one-click” installers from the main ruby page. I kept
getting regular zip files that didn’t have any installation script. (Perhaps
I just couldn’t find it.)

Yeah, just zip files; basically unzip and run \bin\ruby.exe.

You see the same behavior even skipping the interaction with mongo?

I ran these tests without having mongo loaded at all (I commented out that
code). The leak remains.

Okay, (assumed that, but) good to have it verified.

Hello,

On Thu, Jan 14, 2010 at 12:55:02AM +0900, Chuck R. wrote:

Does anyone have any suggestions on how to force these unused objects
to get GC’ed? This program needs to run a long time so a memory leak
is absolutely fatal.

(snip)

I have a method that makes a function call on the COM object
(#request_data). When the function has data to return, it generates a
“DataResolved” event which triggers my callback which was registered
once at the beginning of execution. The callback calls #save and loops
through the collection and pulls some properties into a hash which then
gets inserted into mongodb. When the collection is exhausted, I go back
to #request_data and make another call on the COM object to request the
next batch.

How about calling GC.start before requesting the next batch?

If you had used GC.start already, ignore this suggestion.

Regards,
Masaki S.

On Jan 13, 2010, at 10:49 AM, [email protected] wrote:

Even though I am clearing the received array out, my program slowly
Update runs weekly).

Are you able to do a test run with another ruby build, e.g.
1.8.7/1.9.1 from:
Download Ruby

Running it with 1.9.1-p243 had the same results. I couldn’t figure out
how to install the “one-click” installers from the main ruby page. I
kept getting regular zip files that didn’t have any installation
script. (Perhaps I just couldn’t find it.)

You see the same behavior even skipping the interaction with mongo?

I ran these tests without having mongo loaded at all (I commented out
that code). The leak remains.

cr

On Jan 15, 2010, at 8:44 AM, Masaki S. wrote:

(snip)

to #request_data and make another call on the COM object to request
the
next batch.

How about calling GC.start before requesting the next batch?

If you had used GC.start already, ignore this suggestion.

Calling GC.start does not fix the problem. The program continues to
leak memory and run out.

cr

Jan 13, 2010, Chuck R.:

Also, if this can be reproducible with any COM object, include a
(#request_data). When the function has data to return, it generates a
the process just exits.

Unfortunately this application is not available to the general public so I
can’t provide a script for someone to independently verify this. :frowning:

I’ve never used it but “a simple memory leak detector for ruby”:
http://codeforpeople.com/lib/ruby/dike/dike-0.0.4/README

Or, maybe roll your own:

def object_count
count = Hash.new(0)
ObjectSpace.each_object{|o| count[o.class] += 1}
count
end

require ‘pp’

loop {
do_stuff
pp object_count
}

On Jan 15, 2010, at 12:28 PM, [email protected] wrote:

1.8.7/1.9.1
from:
Download Ruby

Running it with 1.9.1-p243 had the same results.

IronRuby 0.9 appears to have added Win32OLE:
Jimmy Schementi > IronRuby 0.9 Released!

I’ll give ironruby a try on Monday when I get back into the office.
Thanks for the pointer to it. I had completely forgotten about it.

cr

Jan 13, 2010, Chuck R.:

Here’s some code. This is running against 1.8.6-p383 (from
rubyinstaller.org) on Windows XP SP3 plus all the latest patches (Windows
Update runs weekly).

Are you able to do a test run with another ruby build, e.g. 1.8.7/1.9.1
from:
Download Ruby

Running it with 1.9.1-p243 had the same results.

IronRuby 0.9 appears to have added Win32OLE:
http://blog.jimmy.schementi.com/2009/08/ironruby-09-released.html#win32ole

On Sat, Jan 16, 2010 at 01:59:55AM +0900, Chuck R. wrote:

How about calling GC.start before requesting the next batch?

If you had used GC.start already, ignore this suggestion.

Calling GC.start does not fix the problem. The program continues to leak
memory and run out.

Thank you for testing calling GC.start.
Hmmm…, I don’t have any idea to fix this issue.

BTW,

On Jan 12, 6:19 pm, Chuck R. [email protected] wrote:

I have tried adding my_object.ole_free calls everywhere, but that has
not helped at all. I’ve looked through the archives (back to 2004)
and
saw that this was a common issue years ago. I had hoped it would be
fixed by now.

I had fixed some Win32OLE memory leaks on 2007.
So, I think this issue is not same issue in 2004.

And I have been trying to create simple script which leaks memory
like this issue, but I have not succeeded yet.

I’ll investigate this issue more, but I think it is not easy to
fix this issue because I have not created the sample script yet.

FYI, WIN32OLE#ole_free does not GC WIN32OLE object.
WIN32OLE#ole_free release COM wrapped by WIN32OLE object.
GC.start does GC WIN32OLE objects. (And when WIN32OLE object GCed,
the COM wrapped by the WIN32OLE object is released automatically.)

Regards,
Masaki S.

On Jan 15, 2010, at 3:45 PM, Chuck R. wrote:

leaks memory until it hits its limit (around 1.4 GB on
Update runs weekly).

I’ll give ironruby a try on Monday when I get back into the office.
Thanks for the pointer to it. I had completely forgotten about it.

I had a few minutes to try this out before I left for the day. Alas,
the IronRuby support for win32ole is extremely rudimentary. It does
not handle events or variants. I need both for my script.

cr

On Jan 16, 2010, at 5:25 AM, Masaki S. wrote:

Hmmm…, I don’t have any idea to fix this issue.

I had fixed some Win32OLE memory leaks on 2007.
So, I think this issue is not same issue in 2004.

And I have been trying to create simple script which leaks memory
like this issue, but I have not succeeded yet.

I’ll investigate this issue more, but I think it is not easy to
fix this issue because I have not created the sample script yet.

I have new information on this WIN32OLE memory leak.

ruby 1.9.1 p378 (from rubyinstaller.org)

I believe I have narrowed the problem down to its core, but I need some
help to prove it.

In my program I attach to a COM object and register for several events.
One of the events delivers a large chunk of data (anywhere from a few K
to several megabytes). I process this data (write it to a database) and
then wait for the next event.

I believe that the data delivered through to the event is never
garbage collected. I proved to myself that it is not my code leaking
memory by commenting out my event handler. I am still receiving those
events and watching as my memory footprint grows until the ruby
executable finally dies at around 1.5 GB. BTW, it exits with an exit
code of 239 but my exit handler (#at_exit block) is never executed in
this out of memory situation.

Also, I have run the program with GC.stress = true and it still leaks
until it dies. For some reason the garbage collector either doesn’t know
about this memory or thinks that there is still a valid reference to it
somewhere.

I could absolutely prove this leaking theory if I had a small windows
program that just issued COM events and passed some data with the event.
We would see that the data is never freed.

cr

Chuck R. wrote:

http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7553&group_id=426

This particular problem wasn’t related to events per se, but to leaking WIN32OLE objects. I’m betting they are related.

cr

This should get you started:

Complex or my fairly elaborate protocol may be a little strong. MS has
never been great about documenting important things in a way they can be
found. COM is at the very bottom of a deep stack of MS frameworks and
each of which seems to try to plaster over the magic in the one below.
It seems to be assumed that everyone will buy their expensive tool that
solves all the problems.

– Bill

If you have a problem, and address them with a big complicated tool, you
then have two problems.

On Mar 18, 2010, at 2:15 PM, William R. wrote:

COM has a fairly elaborate protocol for memory management responsibilities across interfaces. Its been quite a while since I have had anything to do with it, but I am pretty sure that the receiver and generator of COM data needs to explicitly free via the appropriate COM procedure.

I’ve looked all over the Microsoft dev site at the COM documentation. I
can’t find anything that describes a complex memory management protocol.
That isn’t to say it isn’t real just that I can’t find it.

BTW, searching through the archives I found a bug report from early 2007
that describes a similar leak.

http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7553&group_id=426

This particular problem wasn’t related to events per se, but to leaking
WIN32OLE objects. I’m betting they are related.

cr

I have continued to chase this problem down. I think I may have a lead,
but I need someone familiar with the win32ole.c code to confirm or deny
my suspicions.

I have pastied the invoke method here: gist:342716 · GitHub

Take a look around line 118 of the pastie. This code is making copies OR
references to each parameter before invoking the event sink callback.

After the block is invoked, I think at least one of the code paths does
NOT call VariantClear() on the passed parameters (see line 215). Won’t
that lead to a memory leak for these parameters?

Could this be the leak that has been plaguing me?

cr

Chuck R. wrote:

I have tried adding my_object.ole_free calls everywhere, but that has

In my program I attach to a COM object and register for several events. One of the events delivers a large chunk of data (anywhere from a few K to several megabytes). I process this data (write it to a database) and then wait for the next event.

I believe that the data delivered through to the event is never garbage collected. I proved to myself that it is not my code leaking memory by commenting out my event handler. I am still receiving those events and watching as my memory footprint grows until the ruby executable finally dies at around 1.5 GB. BTW, it exits with an exit code of 239 but my exit handler (#at_exit block) is never executed in this out of memory situation.

Also, I have run the program with GC.stress = true and it still leaks until it dies. For some reason the garbage collector either doesn’t know about this memory or thinks that there is still a valid reference to it somewhere.

I could absolutely prove this leaking theory if I had a small windows program that just issued COM events and passed some data with the event. We would see that the data is never freed.

cr

COM has a fairly elaborate protocol for memory management
responsibilities across interfaces. Its been quite a while since I have
had anything to do with it, but I am pretty sure that the receiver and
generator of COM data needs to explicitly free via the appropriate COM
procedure.

– Bill

On Mar 24, 2010, at 3:12 PM, Chuck R. wrote:

Could this be the leak that has been plaguing me?

I should have noted that the code snippet comes from ruby 1.9.1 p-376.

I have discovered the root cause of this problem. The good news is that
it isn’t a problem with the Ruby WIN32OLE implementation.

I don’t know if all COM servers work this way, but the specific
application I am using memory-maps all WIN32OLE_EVENT data parameters.
The space consumed by this memory-mapped data is registered in the ruby
process and not against the COM server.

Since COM objects do memory management via reference counting, I had to
specifically release the data given to my event callbacks. I didn’t know
this was a required step (the COM server API docs are silent on this
issue) so I was slowly leaking these references every time my event
callbacks were fired. I have now corrected my misuse of this COM
server’s API and all is well.

cr