On 10/24/07, Bob H. [email protected] wrote:
<snip
original @list[0].
So, tmp ‘remembers’ the original @list[0]. The ‘@list[0] = nil’
removes the reference to what is now in tmp,
I believe that this is not true, imagine the GC freeing all objects
that are nil!
Seems very strange to me.
R.
Sorry for replying to my own post, can somebody confirm please that
the following code proves that Array#shift does not leak?
Thx in advance
def count
i = 0
ObjectSpace.each_object{ i += 1 }
i
end
a = []
10.times do
10_000.times do
a << “a”
a.shift
end
ObjectSpace.garbage_collect
p count
end
381
382
382
382
382
382
382
382
382
382
Robert
Hi –
On Thu, 25 Oct 2007, Todd B. wrote:
Bob, I can only guess that the grief you are talking about has
#output => 3
For the newbies out there, keep this in mind. The name b put on
a[0]'s shoes and kept them on. Somebody correct me if I’m wrong.
I’m not seeing what you’re seeing:
irb(main):014:0> a = 1,2,3
=> [1, 2, 3]
irb(main):015:0> a[0].id
=> 3
irb(main):016:0> b = a
=> [1, 2, 3]
irb(main):017:0> b.id
=> 1669980 # Are you really seeing 3 here?
irb(main):018:0> a[0] = 7
=> 7
irb(main):019:0> a[0].id
=> 15
irb(main):020:0> b.id
=> 1669980 # And here?
When you do b = a, you’re binding b to the same object as a. I don’t
know why b would think you had bound it to a[0].
David
On 10/24/07, David A. Black [email protected] wrote:
puts a[0].id
For the newbies out there, keep this in mind. The name b put on
irb(main):017:0> b.idDavid
You’re right David. That was supposed to be b = a[0] and not b = a.
Todd
On 24-Oct-07, at 9:54 AM, Todd B. wrote:
def dequeue
something to do with memory management, but I don’t see it.
It has to do with an optimisation in the implementation of shift
(there’s a series of posts regarding this earlier in this thread). In
brief, shift optimises itself by ‘shifting’ the start of the array
rather than shifting the contents of the array. The effect is that
using shift leaves an old part of the array in front of the start of
the array (but you cannot access it). The problem is that the shift
implementation doesn’t ‘clear’ the original 0th element, so it is
still in memory, is not accessible, but the garbage collector can see
it. In effect, the object referenced isn’t collected because there is
a reference. This is a memory leak. Sooner or later it’ll get fixed
up, but sometimes it really is later.
Can someone explain – with the code above – why tmp does in fact
not take on NilClass? I think it would be good info for newbies.
In the above code, the shift method, through an involved but hidden
technique implemented by Ruby, arranges for the 0th element of the
list to ‘disappear’, for the original @list[1] to become @list[0] and
so on, for the array size to be reduced by one, and returns the
original @list[0].
So, tmp ‘remembers’ the original @list[0]. The ‘@list[0] = nil’
removes the reference to what is now in tmp, and @list.shift makes
the original @list[0] ‘disappear’, and the last reference to tmp
returns it as the result of the method.
Unfortunately it isn’t as simple as everyone would like it to be.
But, as I’ve mentioned elsewhere, this will all go away in some
future version of Ruby. Already Ruby 1.8.5 is much better than 1.8.4
Cheers,
Bob
Todd
Bob H. – tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. – weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ – works on http://www.raconteur.info/
cms-for-static-content/home/
Hi,
On 24-Oct-07, at 3:09 PM, Robert D. wrote:
10.times do
382
382
382
382
382
382
382Robert
Try it this way:
class Thing
end
def count
i = 0
ObjectSpace.each_object(Thing){ i += 1 }
i
end
a = []
10.times do
10_000.times do
a << Thing.new
end
ObjectSpace.garbage_collect
10_000.times do
a.shift
end
ObjectSpace.garbage_collect
puts “count(1): #{count} size: #{a.size}”
a.push(nil).pop
ObjectSpace.garbage_collect
puts “count(2): #{count} size: #{a.size}”
10_000.times do
a << Thing.new
end
ObjectSpace.garbage_collect
10_000.times do
a[0] = nil # <<<<<< !!!
a.shift
end
ObjectSpace.garbage_collect
puts “count(3): #{count} size: #{a.size}”
end
count(1): 10000 size: 0
count(2): 0 size: 0
count(3): 0 size: 0
count(1): 10000 size: 0
count(2): 0 size: 0
count(3): 0 size: 0
count(1): 10000 size: 0
count(2): 0 size: 0
count(3): 0 size: 0
count(1): 10000 size: 0
count(2): 0 size: 0
count(3): 0 size: 0
count(1): 10000 size: 0
count(2): 0 size: 0
count(3): 0 size: 0
count(1): 10000 size: 0
count(2): 0 size: 0
count(3): 0 size: 0
count(1): 10000 size: 0
count(2): 0 size: 0
count(3): 0 size: 0
count(1): 10000 size: 0
count(2): 0 size: 0
count(3): 0 size: 0
count(1): 10000 size: 0
count(2): 0 size: 0
count(3): 0 size: 0
count(1): 10000 size: 0
count(2): 0 size: 0
count(3): 0 size: 0
Bob H. – tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. – weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ – works on http://www.raconteur.info/
cms-for-static-content/home/
On 24-Oct-07, at 5:03 PM, Robert D. wrote:
end
The code in Ruby that deals with the stuff before the start of the
array is non-trivial. The push(nil).pop thing I did, or the use of <<
you did triggers a cleanup. If you do your latest example without
interleaving the << and shift you’ll have a problem (change the
implementation of the Thing class in my example to allocate the
million element array and see what happens – well DON’T actually,
you won’t like what it does to your machine).
The trouble here is that the normal use of queues involves filling
them up then emptying them. If you use an Array and Array.shift to
empty the queue there’s going to be at least one element ‘stuck’ in
front of the array, likely more than 1. This might not matter but it
might. It mattered to my cache implementation, and it mattered to
Mongrel’s task management.
Cheers,
Bob
Bob H. – tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. – weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ – works on http://www.raconteur.info/
cms-for-static-content/home/
On 10/24/07, Bob H. [email protected] wrote:
a.shift
you won’t like what it does to your machine).The trouble here is that the normal use of queues involves filling
them up then emptying them. If you use an Array and Array.shift to
empty the queue there’s going to be at least one element ‘stuck’ in
front of the array, likely more than 1. This might not matter but it
might. It mattered to my cache implementation, and it mattered to
Mongrel’s task management.
Well I guess I have to believe you, but I would love to reproduce it
if I do
times do
<<
end
times do
shift
end
it will leak?
R.
On 10/24/07, Bob H. [email protected] wrote:
i
endi = 0
10_000.times do
end
count(2): 0 size: 0
count(1): 10000 size: 0
count(3): 0 size: 0
Recursive Design Inc. – weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ – works on http://www.raconteur.info/
cms-for-static-content/home/
Amazing Bob,
it has nothing to do with nil though, but with the assignment to a[0]
change a[0] = nil to a[0]=42, you get the same results.
You can even do the following
def count
i = 0
ObjectSpace.each_object{ i += 1 }
i
end
a = []
10_000.times do
a << Object.new
end
ObjectSpace.garbage_collect
10_000.times do
a.shift
end
ObjectSpace.garbage_collect
puts “count(1): #{count} size: #{a.size}”
a.push(nil).pop
ObjectSpace.garbage_collect
puts “count(2): #{count} size: #{a.size}”
10_000.times do
a << Object.new
end
ObjectSpace.garbage_collect
10_000.times do
a[0] = Object.new
a.shift
end
ObjectSpace.garbage_collect
puts “count(3): #{count} size: #{a.size}”
count(1): 10386 size: 0
count(2): 389 size: 0
count(3): 392 size: 0
There remains one thing to be clarified, is this really a bug?
I do not think so, I just ran this until I got 200 points on my terminal
:-0
a = []
loop do
a << Array.new( 1_000_000){ Object.new }
a.shift
$stdout.print “.”
$stdout.flush
end
on my Zenwalk box memory is allocated and released regualrely, I doubt
if ObjectSpace is the right tool to use.
Anyway forgive me to be blunt but everybody should just use Array#shift.
Cheers
Robert
Other people, however, believe that you enter a queue at
the head and leave at the tail, in the manner of food passing through a
snake. Queues written in that way appear in places that might be
considered authoritiative, such as the GNU/Linux operating system, making
the point of view hard to dismiss however repugnant you find the idea of
getting your data from a snake’s rear-end."
Repugnant is definitely the word. I don’t like the idea of getting my
information from a snake’s asshole. If I wanted to do that, I’d talk
to Steven Ballmer.
–
Giles B.
Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org
Tumblelog: http://giles.tumblr.com/
On 10/25/07, Bob H. [email protected] wrote:
shift
end
it will leak?How else would you describe the 10k Things left in memory after the
shifts in the modified version of your test case?
But that is assuming that the ObjectSpace tools tell the truth, it
does however not look like that on my box.
I have therefore no reason to believe that the above structure of the
code will leak, and have asked you if it will.
I will kill my box tonight(or not).
R.
Hi,
On 25-Oct-07, at 1:12 AM, Robert D. wrote:
Well I guess I have to believe you, but I would love to reproduce it
if I do
times do
<<
end
times do
shift
end
it will leak?
How else would you describe the 10k Things left in memory after the
shifts in the modified version of your test case?
Cheers,
Bob
R.
Bob H. – tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. – weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ – works on http://www.raconteur.info/
cms-for-static-content/home/
On 25-Oct-07, at 9:55 AM, Robert D. wrote:
end
code will leak, and have asked you if it will.
I will kill my box tonight
I guess you’ll have to convince your self 
Cheers,
Bob
R.
–
what do I think about Ruby?
http://ruby-smalltalk.blogspot.com/
Bob H. – tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. – weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ – works on http://www.raconteur.info/
cms-for-static-content/home/
On 10/25/07, Bob H. [email protected] wrote:
shift
end
it will leak?How else would you describe the 10k Things left in memory after the
shifts in the modified version of your test case?
The problem is that Ruby’s COW (copy-on-write) scheme is messed up for
arrays. I’ve found much worse cases performance and memory-wise. Try
doing a small #slice on a big array a bunch and modify the small
slices. I made a 1.9 patch a couple years ago that fixed the problem
and made ALL operations near the front just as fast as the operations
on the back. Unfortunately, it never made it in. The way I did it
might be unique, not sure. I thought C++ deque does something
similiar, but it doesn’t. When I find time, I’ll try to do the fix
again.
On 10/25/07, Bob H. [email protected] wrote:
if I do
But that is assuming that the ObjectSpace tools tell the truth, it
does however not look like that on my box.
I have therefore no reason to believe that the above structure of the
code will leak, and have asked you if it will.
I will kill my box tonightI guess you’ll have to convince your self
yes I have because it just did not happen,
526/28 > uname -a && ruby -v
Linux zenwalk 2.6.18.1 #1 SMP PREEMPT Mon Oct 30 15:32:27 CET 2006
i686 athlon-4 i386 GNU/Linux
ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-linux]
the program was
N = 1000
M = 1000
a = []
loop do
N.times do
a << Array.new( M ){ Object.new }
end
N.times do
a.shift
end
end
Ruby allocated up to 43M and then decided to release 3M regularly,
seems a very reasonable strategy to me.
I feel that it would be nice to show the exact leaking code so that I
can run it.
Cheers
Robert
On 25-Oct-07, at 12:40 PM, Eric M. wrote:
The problem is that Ruby’s COW (copy-on-write) scheme is messed up for
arrays. I’ve found much worse cases performance and memory-wise. Try
doing a small #slice on a big array a bunch and modify the small
slices. I made a 1.9 patch a couple years ago that fixed the problem
and made ALL operations near the front just as fast as the operations
on the back. Unfortunately, it never made it in. The way I did it
might be unique, not sure. I thought C++ deque does something
similiar, but it doesn’t. When I find time, I’ll try to do the fix
again.
Eric, thanks for the information.
Cheers,
Bob
Bob H. – tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. – weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ – works on http://www.raconteur.info/
cms-for-static-content/home/
On 10/25/07, Bob H. [email protected] wrote:
again.
Eric, thanks for the information.
Now this is all very great but could you share with us what is going
on, right now I just see people complaining that it is broken, no
reference to a bug report, no test code, I feel that this is highly
unfair to Ruby.
Whatever I am not going to try to reproduce an error that might not
even be there.
I have not found any reference to this on Ruby core, could you please
give a pointer to your patch Eric?
Robert
On 10/25/07, Giles B. [email protected] wrote:
information from a snake’s asshole. If I wanted to do that, I’d talk
to Steven Ballmer.
Or, in a similar vein perhaps Disney.
Repugnant or not?:
–
Rick DeNatale
My blog on Ruby
http://talklikeaduck.denhaven2.com/
On 25-Oct-07, at 4:41 PM, Robert D. wrote:
Now this is all very great but could you share with us what is going
on, right now I just see people complaining that it is broken, no
reference to a bug report, no test code, I feel that this is highly
unfair to Ruby.Whatever I am not going to try to reproduce an error that might not
even be there.
I have not found any reference to this on Ruby core, could you please
give a pointer to your patch Eric?
Robert, I’ve provided several pieces of code to illustrate this problem.
I’m afraid this thread has the potential to degenerate very very
quickly. I’m sorry I’m involved, and I’d very much like to be
uninvolved as of now.
Cheers,
Bob
Robert
–
what do I think about Ruby?
http://ruby-smalltalk.blogspot.com/
Bob H. – tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. – weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ – works on http://www.raconteur.info/
cms-for-static-content/home/
On 10/25/07, Robert D. [email protected] wrote:
similiar, but it doesn’t. When I find time, I’ll try to do the fix
even be there.
I have not found any reference to this on Ruby core, could you please
give a pointer to your patch Eric?Robert
Here it is from over 2 years ago:
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/5861
I believe matz tried incorporating it about a year ago, but there were
too many changes in the code since I patched it. You might find a
discussion from then.
The message above also has a testbench (for some reason it is
base-64). Last I tried it, there were some improvements, but there
were still some tests that looked clearly leaky and/or slow.
Eric
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
Sponsor our Newsletter | Privacy Policy | Terms of Service | Remote Ruby Jobs