Why does this code leak?

ara_howard · January 9, 2008, 4:23am

cfp2:~ > cat a.rb
#! /usr/bin/env ruby

require ‘net/http’

3.times do
Net::HTTP.start(‘www.google.com’) do |http|
http_response = http.get ‘/’
end

p ObjectSpace.each_object(Net::HTTPResponse){}
end

cfp2:~ > ruby a.rb
1
2
3

why is this leaking Net::HTTPResponse objects?

a @ http://codeforpeople.com/

ara_howard · January 9, 2008, 4:36am

why is this leaking Net::HTTPResponse objects?

I don’t know, but jruby has the exact same behaviour. Maybe that helps?

ara_howard · January 9, 2008, 4:54am

Why are you assuming it’s leaking without giving the GC a chance to run?

require ‘net/http’

3.times do
Net::HTTP.start(‘www.google.com’) do |http|
http_response = http.get ‘/’
end

p ObjectSpace.each_object(Net::HTTPResponse){}
end

p “Force GC”
ObjectSpace.garbage_collect
p ObjectSpace.each_object(Net::HTTPResponse){}

C:>ruby test.rb
1
2
3
“Force GC”
0

MBL

ara_howard · January 9, 2008, 5:24am

On Wednesday 09 January 2008 04:23:23 ara howard wrote:

why is this leaking Net::HTTPResponse objects?
It does not leak, objects are just not immediately released by the GC.
Try it
running 1000 times (on local server preferably).

Jan

ara_howard · January 9, 2008, 5:28am

On Jan 8, 2008, at 8:53 PM, Michael Bevilacqua-Linn wrote:

Why are you assuming it’s leaking without giving the GC a chance to
run?

because i’m an idiot and pasted the wrong buffer. here is the code:

distilled behaviour from dike.rb

class Object
Methods = instance_methods.inject(Hash.new){|h, m| h.update m =>
instance_method(m)}
end

class Class
Methods = instance_methods.inject(Hash.new){|h, m| h.update m =>
instance_method(m)}

 def new *a, &b
   object = Methods["new"].bind(self).call *a, &b
 ensure
   ObjectSpace.define_finalizer object, finalizer
 end

 def finalizer
   lambda{}
 end

end

the above makes this code leaks, but only Net::HTTPOK objects

require “net/http”

def leak!
Net::HTTP.start(“localhost”) do |http|
puts http.get(‘/’).code
end
end

3.times {
puts “—”
p ObjectSpace.each_object(Net::HTTPResponse){}
leak!
GC.start
}

a @ http://codeforpeople.com/

ara_howard · January 9, 2008, 10:41am

2008/1/9, ara howard [email protected]:

so… why does installing a static finalizer work ok, but a dynamic
one leaks memory!?

i’m nice and confused now.

I’d say you picked the wrong class for your tests - apparently a
lambda uses an array somehow. This is what I see from the modified
script (attached):

10:38:52 /cygdrive/c/Temp
$ ./a.rb leak

Foo: 0

Foo: 1

Foo: 1
10:39:04 /cygdrive/c/Temp
$ ./a.rb noleak

Foo: 0

Foo: 1

Foo: 1
10:39:08 /cygdrive/c/Temp
$ ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-cygwin]

Kind regards

robert

ara_howard · January 9, 2008, 6:21am

On Jan 8, 2008, at 8:45 PM, Jan D. wrote:

It does not leak, objects are just not immediately released by the
GC. Try it
running 1000 times (on local server preferably).

as i already posted - i posted the wrong code. here is an even more
distilled version:

cfp2:~ > cat a.rb

distilled behaviour from dike.rb

class Class
Finalizer = lambda {}

 def leak_free_finalizer
   Finalizer
 end

 def leaky_finalizer
   lambda{}
 end

 def finalizer
   %r/leak/ =~ ARGV.first ? leaky_finalizer : leak_free_finalizer
 end

 def new *a, &b
   object = allocate
   object.send :initialize, *a, &b
   object
 ensure
   ObjectSpace.define_finalizer object, finalizer
 end

end

the above makes this code leak iff ARGV has “leak” in it

require “yaml”

7.times {
GC.start

 y Array.name => ObjectSpace.each_object(Array){}

 Array.new

}

cfp2:~ > ruby a.rb

Array: 21

Array: 49

Array: 58

cfp2:~ > ruby a.rb leak

Array: 21

Array: 38

Array: 54

Array: 67

Array: 79

Array: 91

Array: 103

so… why does installing a static finalizer work ok, but a dynamic
one leaks memory!?

i’m nice and confused now.

a @ http://codeforpeople.com/

ara_howard · January 9, 2008, 5:14pm

On Jan 9, 2008, at 2:41 AM, Robert K. wrote:

'd say you picked the wrong class for your tests - apparently a
lambda uses an array somehow. This is what I see from the modified
script (attached):

hmmm. doing this in irb suggests not:

list = []
GC.start; p ObjectSpace.each_object(Array){}; list << lambda{}
GC.start; p ObjectSpace.each_object(Array){}; list << lambda{}
GC.start; p ObjectSpace.each_object(Array){}; list << lambda{}
GC.start; p ObjectSpace.each_object(Array){}; list << lambda{}

the number of Array’s will remain static. hrrrmmm…

a @ http://codeforpeople.com/

ara_howard · January 9, 2008, 5:40pm

Here’s an even simpler example. You don’t need anything but the
following to demonstrate the problem.

class Foo
Finalizer = lambda{}
def initialize
ObjectSpace.define_finalizer(self,Finalizer)
end
end
def test
10.times do
GC.start
count = ObjectSpace.each_object(Foo) {}
p “Count: #{count}”
Foo.new
end
end
test # → no leak
“Count: 0”
“Count: 1”
“Count: 2”
“Count: 2”
“Count: 2”
etc.

Now, re-open Foo and add an inline finalizer.

class Foo
def initialize
ObjectSpace.define_finalizer(self,lambda{})
end
end
test # → now it leaks
“Count: 1”
“Count: 2”
“Count: 3”
“Count: 4”
etc.

I realize the scope of the lambda invocation is different in this
example, but since the behavior is so similar, I thought it likely
pointed to the same underlying issue.

Dan

dev.zeraweb.com

ara_howard · January 9, 2008, 5:25pm

so… why does installing a static finalizer work ok, but a dynamic
one leaks memory!?

My guess is that the lambda is keeping the scope of the new invocation
around, which includes a reference to the newly created array.

Would it be a better test not to use Array (instances of which seems
to be created also by ObjectSpace / YAML)? Ex:

class Foo ; end
7.times {
GC.start
count = ObjectSpace.each_object(Foo)
p “Count: #{count}”
Foo.new
}

-Dan

ara_howard · January 9, 2008, 5:46pm

On Jan 9, 2008, at 9:39 AM, dan yoder wrote:

I realize the scope of the lambda invocation is different in this
example, but since the behavior is so similar, I thought it likely
pointed to the same underlying issue.

i think it’s actually some strange interaction with yaml. check this
out:

cfp2:~ > cat a.rb
class Class
def finalizer
lambda{}
end

def new *a, &b
object = allocate
object.send :initialize, *a, &b
object
ensure
ObjectSpace.define_finalizer object, finalizer
end
end

class Foo; end
class Bar < Foo; end

c = Array

if ARGV.detect{|arg| arg[“leak”]}
require “yaml”
7.times {
GC.start
y c.name => ObjectSpace.each_object(c){}
c.new
}
else
7.times {
GC.start
puts “—”
puts “#{ c.name }: #{ ObjectSpace.each_object(c){} }”
c.new
}
end

cfp2:~ > ruby a.rb

Array: 6

Array: 11

Array: 14

Array: 18

Array: 20

cfp2:~ > ruby a.rb leak

Array: 21

Array: 38

Array: 54

Array: 67

Array: 79

Array: 91

Array: 103

a @ http://codeforpeople.com/

ara_howard · January 9, 2008, 7:40pm

On 09.01.2008 19:26, Rick DeNatale wrote:

require “yaml”
c.new

Not sure how you got there Ara, I don’t see where the OP ever mentioned YAML.

Rick, OP == ara => true.

I think the key is where the lambda is created. The lambda is
capturing the binding.

In his revisited example (msg id
[email protected]) the non leaky finalizer
had self bound to Class and the non leaky to the particular class
instance. I believe Ara’s confusion stems from the question how a class
instance can keep instances in memory.

In the first case the lambda is being created in the bindig context of
the class, and in particular self is the class.

In the second case, the lambda is being created in the binding context
of the new instance, and self is that new instance, so the lambda in
the finalizer is hanging on to it.

That’s not true for the posting I mentioned above and also not for the
last one. There was only one finalizer but one branch used yaml while
the other did not.

Kind regards

robert

ara_howard · January 9, 2008, 7:28pm

On Jan 9, 2008 11:45 AM, ara howard [email protected] wrote:

On Jan 9, 2008, at 9:39 AM, dan yoder wrote:

I realize the scope of the lambda invocation is different in this
example, but since the behavior is so similar, I thought it likely
pointed to the same underlying issue.

i think it’s actually some strange interaction with yaml. check this
out:
…
puts “—”
puts “#{ c.name }: #{ ObjectSpace.each_object(c){} }”
c.new
}
end

cfp2:~ > ruby a.rb

Array: 6

cfp2:~ > ruby a.rb leak

Array: 21

Not sure how you got there Ara, I don’t see where the OP ever mentioned
YAML.

I think the key is where the lambda is created. The lambda is
capturing the binding.

In the first case the lambda is being created in the bindig context of
the class, and in particular self is the class.

In the second case, the lambda is being created in the binding context
of the new instance, and self is that new instance, so the lambda in
the finalizer is hanging on to it.

That’s my theory anyway.

–
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

ara_howard · January 9, 2008, 9:43pm

Ah, missed that.

Anyway ara re-asked the question on ruby-core and Matz answered him
pretty much the same way I did.

On Jan 9, 2008 1:39 PM, Robert K. [email protected]
wrote:

…
puts "---"
[email protected]) the non leaky finalizer

That’s not true for the posting I mentioned above and also not for the
last one. There was only one finalizer but one branch used yaml while
the other did not.

Kind regards
    robert

–
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

ara_howard · January 9, 2008, 10:52pm

On Jan 9, 2008 3:46 PM, ara howard [email protected] wrote:

the ‘object’ ended up being bound. it was a local var of another
function. makes no sense.

I think that this is the code you are talking about right?

class Class
Finalizer = lambda { }

def leak_free_finalizer
  Finalizer
end

def leaky_finalizer
  # What's self here? It's the same as for the finalizer method
  # which
  lambda{}
end

def finalizer
  %r/leak/ =~ ARGV.first ? leaky_finalizer : leak_free_finalizer
end

def new *a, &b
  object = allocate
  object.send :initialize, *a, &b
  object
ensure
  ObjectSpace.define_finalizer object, finalizer
end

end

Looking at eval.c, it looks like lambda actually copies information
from the invocation stack, not just from the current frame. In the
leaky finalizer case we have the following on the stack when
leaky_finalizer is called, with the binding represented in hash
notation.

leaky_finalizer :self => Class
finalizer :self => Class
new :self=> Class, :object => the new Array
instance
code which called Array.new

The leak free finalizer lambda was created once at a time when no
instance to be finalized was on the stack.

–
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

ara_howard · January 9, 2008, 9:47pm

On Jan 9, 2008, at 1:42 PM, Rick DeNatale wrote:

Ah, missed that.

Anyway ara re-asked the question on ruby-core and Matz answered him
pretty much the same way I did.

sort of - i still don’t see how

the ‘object’ ended up being bound. it was a local var of another
function. makes no sense.
why this happened for only say, 1 of 100000 objects

i still think it’s a bug, but a have a work around - see dike
announcement.

a @ http://codeforpeople.com/

ara_howard · January 9, 2008, 11:19pm

On Jan 9, 2008, at 2:52 PM, Rick DeNatale wrote:

code which called Array.new

The leak free finalizer lambda was created once at a time when no
instance to be finalized was on the stack.

yeah i think that may be true - but it doesn’t make sense. the

def leaky_finalizer
lambda{}
end

is the current paradigm for preventing lambdas from enclosing a
reference to an object. what you are saying is that this call
encloses a local variable from another (the calling in the this case)
function.

how would that not be a bug? why enclose a variable that cannot
possible be reached in the code ran? this seems, to me just like this
code

void * leak (){ return(&malloc(42)) }

i just cannot image why lambda would crawl up the stack outside the
current function. if that is true then they are useless and any
invocation means every object in memory at the time of creation can
never be freed while that lambda exists. doesn’t that seem
excessive? also i use tons of lambdas in code that does not leak so
this just seems impossible.

nevertheless you may be right!

cheers.

a @ http://codeforpeople.com/

ara_howard · January 10, 2008, 12:05am

On Jan 9, 2008 5:18 PM, ara howard [email protected] wrote:

finalizer :self => Class
lambda{}
end

is the current paradigm for preventing lambdas from enclosing a
reference to an object. what you are saying is that this call
encloses a local variable from another (the calling in the this case)
function.

Well I’m positing that based on a rather superficial read of eval.c.

how would that not be a bug? why enclose a variable that cannot
possible be reached in the code ran?

Well I suspect that it’s because no real analysis is done of what’s
inside the block when a proc is created, so the assumption is that the
entire binding is needed. The Smalltalk compilers I recall would
produce different types of block objects depending on whether or not
the block contained references to variables outside the block, and/or
contained a return.

Now why it goes back down the stack, if it indeed does, I’m not sure.
Perhaps it has something to do with the lambda vs. Proc.new
differences. I think that Proc.new and lambda/proc both use the
proc_alloc function where this seems to be happening.

In any event this is probably more a topic for ruby-core so I’m
cross-posting this reply there.

this just seems impossible.

nevertheless you may be right!

Or not

Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/

ara_howard · January 10, 2008, 1:26pm

Just trying to answer the question if it is a bug by making a minimum
version of the leaking version (and correcting Ara’s terrible bug how
to write 7 and running it with 1.8 and 1.9
591/92 > cat leak.rb

vim: sw=2 ts=2 ft=ruby expandtab tw=0 nu syn:

class Foo
def initialize
ObjectSpace.define_finalizer self, lambda{}
end
end

(42/6).times {
GC.start

p "Foo" => ObjectSpace.each_object(Foo){}

Foo.new

}

robert@roma:~/log/ruby/theory 13:19:00
592/93 > ruby leak.rb
{“Foo”=>0}
{“Foo”=>1}
{“Foo”=>2}
{“Foo”=>3}
{“Foo”=>4}
{“Foo”=>5}
{“Foo”=>6}
robert@roma:~/log/ruby/theory 13:19:06
593/94 > ruby1.9 leak.rb
{“Foo”=>0}
{“Foo”=>1}
{“Foo”=>1}
{“Foo”=>1}
{“Foo”=>1}
{“Foo”=>1}
{“Foo”=>1}

I do not know if this is good enough to say it is a bug in 1.8, but I
would somehow suspect so.

Cheers
Robert

–
http://ruby-smalltalk.blogspot.com/

Whereof one cannot speak, thereof one must be silent.
Ludwig Wittgenstein

ara_howard · January 10, 2008, 5:15pm

On Jan 10, 2008, at 5:26 AM, Robert D. wrote:

end

(42/6).times {
GC.start

nice!

so i posted something like this over on ruby-core, which i’ll add for
posterity:

"

to add a note to the end of the thread the fix to my problem was
essentially this

class Class
New = instance_method :new
Objects = Hash.new
Destroy = lambda{|object_id| Objects.delete object_id}

def new *a, &b
object = allocate
Objects[object.object_id] = caller
object.send :initialize *a, &b
object
ensure
ObjectSpace.define_finalizer object, Destroy
end
end

and that

class Class
def destroy
lambda{}
end

…
ObjectSpace.define_finalizer object, destroy
…
end

perhaps i have not explained this adequately, but i still feel this is
a bug. the ‘self’ that is enclosed is never ‘object’ and that self
has no reference, save a local variable in another function, that
refers to ‘object’.

in any case i have a workaround and dike.rb is better than ever (see
[ANN] on ruby-talk) so thanks all for the input!

"

a @ http://codeforpeople.com/

Why does this code leak?

distilled behaviour from dike.rb

the above makes this code leaks, but only Net::HTTPOK objects

10:38:52 /cygdrive/c/Temp $ ./a.rb leak

Foo: 0

Foo: 1

Foo: 1

Foo: 1

Foo: 1

Foo: 1

Foo: 1 10:39:04 /cygdrive/c/Temp $ ./a.rb noleak

Foo: 0

Foo: 1

Foo: 1

Foo: 1

Foo: 1

Foo: 1

distilled behaviour from dike.rb

the above makes this code leak iff ARGV has “leak” in it

cfp2:~ > ruby a.rb

Array: 21

Array: 49

Array: 58

Array: 58

Array: 58

Array: 58

cfp2:~ > ruby a.rb leak

Array: 21

Array: 38

Array: 54

Array: 67

Array: 79

Array: 91

cfp2:~ > ruby a.rb

Array: 6

Array: 11

Array: 14

Array: 18

Array: 20

Array: 20

cfp2:~ > ruby a.rb leak

Array: 21

Array: 38

Array: 54

Array: 67

Array: 79

Array: 91

cfp2:~ > ruby a.rb

cfp2:~ > ruby a.rb leak

Or not

vim: sw=2 ts=2 ft=ruby expandtab tw=0 nu syn:

10:38:52 /cygdrive/c/Temp
$ ./a.rb leak

Foo: 1
10:39:04 /cygdrive/c/Temp
$ ./a.rb noleak