Writing long-running daemons without memory leaks?

Hi all,

I am in the middle of writing some code that will exist as a
long-running process on a number of machines. I am having trouble
keeping memory utilization down, which is important in this case,
because other processes on these machines will have priority to use the
majority of RAM on these boxes for other tasks. The daemon itself is
pretty simple, just reading and writing files and keeping some small
state, but it also needs to cache large writes from multiple clients.
When it does this, I find that this memory is not freed.

My question is, I’ve heard from numerous accounts that WEBrick also
leaks memory, and I would suspect that Mongrel does, too (though I
haven’t confirmed that) for the same reason. I understand that Ruby’s GC
is conservative, but can anyone point me to documentation, reference,
source code, anything really that will help me better understand how to
write Ruby code that doesn’t “leak” and can be sustained as a
long-running process without an ever-increasing memory footprint?

Any help would be greatly appreciated. Thanks in advance.

P.S. I’ve read the Pickaxe’s lone page on the subject (in the Duck
Typing chapter) and I’ve seen Why’s article about the same
(http://whytheluckystiff.net/articles/theFullyUpturnedBin.html).

P.P.S. Has Minero A.'s Ruby book been translated to English anywhere?
I hear that it has a whole chapter on GC…


Toby DiPasquale

On Sat, 18 Mar 2006, Toby DiPasquale wrote:

P.S. I’ve read the Pickaxe’s lone page on the subject (in the Duck
Typing chapter) and I’ve seen Why’s article about the same
(http://whytheluckystiff.net/articles/theFullyUpturnedBin.html).

P.P.S. Has Minero A.'s Ruby book been translated to English anywhere?
I hear that it has a whole chapter on GC…

i’m running dozens of ruby daemons, some of which are extremely (months
at a
time) long lived and have not seen any gc issues. can you post a
specific
(minimal) example that you find leaks memory?

regards.

-a

unknown wrote:

i’m running dozens of ruby daemons, some of which are extremely (months
at a
time) long lived and have not seen any gc issues. can you post a
specific
(minimal) example that you find leaks memory?

Sure. Here is a server and client, resp, that exhibit the behavior I am
referring to:

puma:> cat leak.rb

vim:set ts=4 sw=4 ai:

require ‘socket’

class Buffer
def initialize size
@size = size
@buffer = []
@length = 0
end
attr_reader :length, :size
def full? ; @length == @size end
def empty? ; @length.zero? end
# Lets you fill up to capacity and then returns
# what’s left over
def fill data
if full?
data
elsif @length + data.length < @size
@buffer << data
@length += data.length
nil
else
l = @size - @length
@buffer << data[0, l]
data[l…-1]
end
end
def expunge
buf = @buffer.join
@buffer.clear
@length = 0
buf
end
end # class Buffer

def read_long s
s.read( 4).unpack( “N”)[0]
end

main client handling thread logic

client_handler = lambda do |s|
begin
len = read_long s
buf = Buffer.new len
until buf.full?
x = s.read 4096
break if x.nil?
buf.fill x
end
puts “writing #{buf.length} bytes”
f = File.open “/dev/null”, “w”
f.write buf.expunge
f.close
ensure
s.close
end
end

main server loop

ss = TCPServer.new ‘127.0.0.1’, 10001
begin
while true
s = ss.accept
puts “got connection”
Thread.start s, &client_handler
end
ensure
ss.close
end

puma:~> cat leak_client.rb

vim:set ts=4 sw=4 ai:

require ‘socket’

def write_long s, l
s.write( [l].pack( “N”))
end

str = “a” * 65536

t = TCPSocket.new ‘127.0.0.1’, 10001
write_long t, 1024 * str.length
1024.times { t.write str }
t.close

puma:~>

And here’s the output from a simple test to show the issue:

puma:~> for i in 1 2 3 4 5 6 7 8 9 10; do ps aux | grep [l]eak.rb ; ruby
leak_client.rb ; done
toby 8452 0.0 0.1 3116 1684 pts/3 S+ 11:23 0:00 ruby
leak.rb
toby 8452 3.8 12.8 143032 133432 pts/3 Sl+ 11:23 0:01 ruby
leak.rb
toby 8452 6.4 15.8 173664 164020 pts/3 Sl+ 11:23 0:02 ruby
leak.rb
toby 8452 7.8 15.2 167988 158472 pts/3 Sl+ 11:23 0:03 ruby
leak.rb
toby 8452 9.2 15.8 173532 164004 pts/3 Sl+ 11:23 0:03 ruby
leak.rb
toby 8452 10.7 15.2 168024 158508 pts/3 Sl+ 11:23 0:04 ruby
leak.rb
toby 8452 12.0 15.8 173568 164000 pts/3 Sl+ 11:23 0:04 ruby
leak.rb
toby 8452 13.2 15.8 173568 164008 pts/3 Sl+ 11:23 0:05 ruby
leak.rb
toby 8452 14.3 15.8 173568 164012 pts/3 Sl+ 11:23 0:06 ruby
leak.rb
toby 8452 15.7 15.0 165012 155496 pts/3 Sl+ 11:23 0:06 ruby
leak.rb
puma:~> ps aux | grep [l]eak.rb
toby 8452 13.0 22.1 239132 229544 pts/3 Sl+ 11:23 0:07 ruby
leak.rb
puma:~>

This memory never really goes away. Also notice that the client is
sending 64MB, but the first time the leak.rb image jumps to 143MB. Any
clues? Am I just doing something really stupid? TIA.


Toby DiPasquale

On Sat, 18 Mar 2006, Toby DiPasquale wrote:

Sure. Here is a server and client, resp, that exhibit the behavior I am
referring to:

i cannot reproduce it:

 harp:~ > cat a.rb
 require 'socket'

 class Buffer
    def initialize size
        @size = size
        @buffer = []
        @length = 0
    end
    attr_reader :length, :size
    def full?  ; @length == @size end
    def empty? ; @length.zero? end
    # Lets you fill up to capacity and then returns
    # what's left over
    def fill data
        if full?
            data
        elsif @length + data.length < @size
            @buffer << data
            @length += data.length
            nil
        else
            l = @size - @length
            @buffer << data[0, l]
            data[l..-1]
        end
    end
    def expunge
        buf = @buffer.join
        @buffer.clear
        @length = 0
        buf
    end
 end     # class Buffer

 def read_long s
    s.read( 4).unpack( "N")[0]
 end

 # main client handling thread logic
 client_handler = lambda do |s|
    begin
        len = read_long s
        buf = Buffer.new len
        until buf.full?
            x = s.read 4096
            break if x.nil?
            buf.fill x
        end
        puts "writing #{buf.length} bytes"
        f = File.open "/dev/null", "w"
        f.write buf.expunge
        f.close
    ensure
        s.close
    end
 end

 # main server loop
 ss = TCPServer.new '127.0.0.1', 10001
 begin
    while true
        s = ss.accept
        puts "got connection"
        Thread.start s, &client_handler
    end
 ensure
    ss.close
 end



 harp:~ > cat b.rb
 require 'socket'

 def write_long s, l
        s.write( [l].pack( "N"))
 end

 str = "a" * 65536

 t = TCPSocket.new '127.0.0.1', 10001
 write_long t, 1024 * str.length
 1024.times { t.write str }
 t.close



 harp:~ > ps aux|head -1;  while true;do ruby b.rb && ps aux|grep 

‘ruby a.rb’|grep -v grep ;done
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
COMMAND
ahoward 13671 12.7 23.1 239328 237956 pts/26 S 10:14 0:44 ruby
a.rb
ahoward 13671 12.7 21.2 219404 218032 pts/26 S 10:14 0:45 ruby
a.rb
ahoward 13671 12.8 22.3 230784 229396 pts/26 S 10:14 0:45 ruby
a.rb
ahoward 13671 13.0 22.3 230784 229412 pts/26 S 10:14 0:45 ruby
a.rb
ahoward 13671 13.1 22.3 230916 229416 pts/26 S 10:14 0:46 ruby
a.rb
ahoward 13671 13.2 15.9 165364 163876 pts/26 S 10:14 0:46 ruby
a.rb
ahoward 13671 13.3 22.3 230904 229424 pts/26 S 10:14 0:47 ruby
a.rb
ahoward 13671 13.4 21.5 225116 221504 pts/26 R 10:14 0:47 ruby
a.rb
ahoward 13671 13.5 28.7 296328 294940 pts/26 S 10:14 0:48 ruby
a.rb
ahoward 13671 13.6 21.5 222480 221108 pts/26 S 10:14 0:48 ruby
a.rb
ahoward 13671 13.7 22.3 230796 229400 pts/26 S 10:14 0:49 ruby
a.rb
ahoward 13671 13.8 21.8 230804 224296 pts/26 R 10:14 0:49 ruby
a.rb
ahoward 13671 13.9 27.3 296344 281256 pts/26 R 10:14 0:50 ruby
a.rb
ahoward 13671 14.0 25.4 265672 261500 pts/26 R 10:14 0:50 ruby
a.rb
ahoward 13671 14.1 22.3 230880 229384 pts/26 S 10:14 0:51 ruby
a.rb
ahoward 13671 14.2 15.9 165340 163864 pts/26 S 10:14 0:51 ruby
a.rb
ahoward 13671 14.3 20.8 230888 214376 pts/26 R 10:14 0:52 ruby
a.rb
ahoward 13671 14.4 19.3 200220 198844 pts/26 S 10:14 0:52 ruby
a.rb
ahoward 13671 14.5 21.2 230852 218484 pts/26 R 10:14 0:53 ruby
a.rb
ahoward 13671 14.6 15.9 165312 163872 pts/26 R 10:14 0:53 ruby
a.rb
ahoward 13671 14.8 28.1 290632 289256 pts/26 S 10:14 0:54 ruby
a.rb
ahoward 13671 14.8 16.0 230776 164724 pts/26 R 10:14 0:54 ruby
a.rb
ahoward 13671 14.9 21.6 230908 222412 pts/26 R 10:14 0:54 ruby
a.rb
ahoward 13671 15.0 15.0 219356 154356 pts/26 R 10:14 0:55 ruby
a.rb
ahoward 13671 15.1 16.2 230864 166464 pts/26 R 10:14 0:55 ruby
a.rb
ahoward 13671 15.2 25.7 265764 264320 pts/26 S 10:14 0:56 ruby
a.rb
ahoward 13671 15.4 22.3 230852 229388 pts/26 S 10:14 0:56 ruby
a.rb
ahoward 13671 15.4 15.9 165312 163864 pts/26 S 10:14 0:57 ruby
a.rb
ahoward 13671 15.5 15.9 165320 163880 pts/26 R 10:14 0:57 ruby
a.rb
ahoward 13671 15.6 28.5 294368 292992 pts/26 S 10:14 0:58 ruby
a.rb
ahoward 13671 15.7 22.3 230808 229388 pts/26 S 10:14 0:58 ruby
a.rb
ahoward 13671 15.8 21.7 230816 223624 pts/26 R 10:14 0:59 ruby
a.rb
ahoward 13671 15.9 15.9 165276 163872 pts/26 R 10:14 0:59 ruby
a.rb
ahoward 13671 16.0 20.6 213384 212008 pts/26 S 10:14 0:59 ruby
a.rb
ahoward 13671 16.1 22.3 230812 229396 pts/26 S 10:14 1:00 ruby
a.rb
ahoward 13671 16.2 22.3 230812 229408 pts/26 S 10:14 1:00 ruby
a.rb
ahoward 13671 16.2 15.9 165280 163872 pts/26 R 10:14 1:01 ruby
a.rb
ahoward 13671 16.3 22.3 230812 229408 pts/26 S 10:14 1:01 ruby
a.rb
ahoward 13671 16.4 19.3 200212 198836 pts/26 S 10:14 1:02 ruby
a.rb
ahoward 13671 16.5 22.3 230840 229400 pts/26 S 10:14 1:02 ruby
a.rb
ahoward 13671 16.5 15.4 159584 158204 pts/26 R 10:14 1:02 ruby
a.rb
ahoward 13671 16.7 27.4 296424 281408 pts/26 R 10:14 1:03 ruby
a.rb
ahoward 13671 16.8 19.3 203424 199044 pts/26 R 10:14 1:03 ruby
a.rb
ahoward 13671 16.9 22.1 230888 227576 pts/26 R 10:14 1:04 ruby
a.rb
ahoward 13671 17.0 25.7 265720 264108 pts/26 S 10:14 1:04 ruby
a.rb
ahoward 13671 17.0 15.9 230812 163952 pts/26 R 10:14 1:05 ruby
a.rb
ahoward 13671 17.1 27.1 296352 278484 pts/26 R 10:14 1:05 ruby
a.rb
ahoward 13671 17.2 28.1 290640 289264 pts/26 S 10:14 1:06 ruby
a.rb
ahoward 13671 17.3 15.9 165248 163860 pts/26 R 10:14 1:06 ruby
a.rb
ahoward 13671 17.4 21.8 230920 224092 pts/26 R 10:14 1:07 ruby
a.rb
ahoward 13671 17.5 22.0 228288 226912 pts/26 S 10:14 1:07 ruby
a.rb
ahoward 13671 17.5 15.9 165264 163872 pts/26 R 10:14 1:07 ruby
a.rb
ahoward 13671 17.6 22.0 228228 226852 pts/26 S 10:14 1:08 ruby
a.rb
ahoward 13671 17.7 22.3 230868 229400 pts/26 S 10:14 1:08 ruby
a.rb
ahoward 13671 17.8 14.8 153684 152304 pts/26 R 10:14 1:09 ruby
a.rb
ahoward 13671 17.9 22.3 230812 229400 pts/26 S 10:14 1:09 ruby
a.rb
ahoward 13671 18.0 21.7 225108 223732 pts/26 S 10:14 1:10 ruby
a.rb
ahoward 13671 18.1 20.6 230924 212128 pts/26 R 10:14 1:10 ruby
a.rb
ahoward 13671 18.1 14.4 149944 148564 pts/26 R 10:14 1:11 ruby
a.rb
ahoward 13671 18.3 28.7 296332 294936 pts/26 S 10:14 1:11 ruby
a.rb
ahoward 13671 18.3 19.5 212736 200952 pts/26 R 10:14 1:12 ruby
a.rb
ahoward 13671 18.4 21.1 230824 216768 pts/26 R 10:14 1:12 ruby
a.rb
ahoward 13671 18.5 22.3 230816 229404 pts/26 S 10:14 1:12 ruby
a.rb

so memory builds to about 25 and, presumably when the gc kicks in, drops
to 15

  • but it certainly stays in this range.

what os and ruby version?

-a

On 3/17/06, Toby DiPasquale [email protected] wrote:

I am in the middle of writing some code that will exist as a
long-running process on a number of machines. I am having trouble
keeping memory utilization down, which is important in this case,
[snip]
source code, anything really that will help me better understand how to
write Ruby code that doesn’t “leak” and can be sustained as a
long-running process without an ever-increasing memory footprint?

Any help would be greatly appreciated. Thanks in advance.

Maybe you already knew this stuff, but, here’s some things that might
help you:

Set things to nil after you’re finished using them.

Avoid continuations. If I recall correctly, the GC doesn’t handle them
right.

Be careful with blocks (and bindings); they keep a reference to all
local variables that are (potentially) visible in the block, even if
those variables aren’t actually used in the block. Any values attached
to those local vars not be garbage collected until the whole block is.
Here’s an example:

def foo
a1=%w[foo bar baz]
a2=%w[shish boom bah]
a3=%w[this one’s actually used]
return bar { p a3 }
end

def bar(&block)
return block
end

a1 and a2 are leaked (temporarily). Even tho they’re unused, they
don’t actually get GC’d until block is also GC’d.

You can also use ObjectSpace.each_object to iterate over currently
live objects; this might help you determine which objects are being
leaked.

unknown wrote:

 harp:~ > ps aux|head -1;  while true;do ruby b.rb && ps aux|grep 

[…]

 ahoward  13671 18.3 19.5 212736 200952 pts/26 R   10:14   1:12 ruby 

a.rb
ahoward 13671 18.4 21.1 230824 216768 pts/26 R 10:14 1:12 ruby
a.rb
ahoward 13671 18.5 22.3 230816 229404 pts/26 S 10:14 1:12 ruby
a.rb

so memory builds to about 25 and, presumably when the gc kicks in, drops
to 15

  • but it certainly stays in this range.

what os and ruby version?

ruby 1.8.4 on Ubuntu Linux, kernel 2.6.12-10 on a P4 Thinkpad with 1GB
RAM. The other systems having this problem are all also running Linux
2.6.x with ruby 1.8.4.

One thing about the above: you do realize that’s ~250MB and ~150MB, not
25 and 15, right?


Toby DiPasquale

-----BEGIN PGP SIGNED MESSAGE-----

In article [email protected],
Toby DiPasquale [email protected] wrote:

_ This is generally true of all unix processes. They do not
return allocated memory back to the system until they exit.
Freeing memory inside the code merely allows you to use it
again, it does not make it available to the system.

_ If your process is not using the memory, it may stay swapped
out, ( look at the real and virtual sizes ). If this is really
an issue, the standard way around it is to do the memory
intensive part in a forked subprocess that exits after it’s
done.

_ Booker C. Bense

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQCVAwUBRBsqMGTWTAjn5N/lAQHVbwP/XYqiyZsWzCqsYNfzP4rCe5cEzWbKBBfU
54A/x/DDnImzpM3MYcIG4Um2AAGFBYMrylxEeTv0Ot6RMi7Nl2NJLIzIg/6lGh1M
6d1ojbnddmUGfJXuYDZ2/lcWDWYzYCM6hYbWGh7h6eMYtNVZizE1k9ebA4zL+H0i
jFU8K8Aoe7k=
=ZbII
-----END PGP SIGNATURE-----

On Sat, 18 Mar 2006, Toby DiPasquale wrote:

One thing about the above: you do realize that’s ~250MB and ~150MB, not
25 and 15, right?

yes - but it’s definitely not leaking. i’m running

harp:~ > ruby -v
ruby 1.8.4 (2006-01-12) [i686-linux]

harp:~ > cat /etc/redhat-release
Red Hat Enterprise Linux WS release 3 (Taroon Update 6)

harp:~ > uname -srm
Linux 2.4.21-37.0.1.EL i686

btw.

it’s clear to me that your code is using tons of memory - it’s not clear
that
it’s leaking, however.

if you looking for why your code is so big look to you handler:
changing it
to

Thread.start(s){|s| loop{ s.read(8192) or break }}

will result in runs that look like this:

harp:~ > ps aux|head -1; while true;do ruby b.rb && ps aux|grep
‘ruby a.rb’|grep -v grep ;done
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
COMMAND
ahoward 15535 4.1 0.4 6004 4556 pts/26 S 10:53 0:00 ruby
a.rb
ahoward 15535 7.1 0.7 8940 7520 pts/26 S 10:53 0:00 ruby
a.rb
ahoward 15535 9.4 0.2 4080 2704 pts/26 S 10:53 0:00 ruby
a.rb
ahoward 15535 10.7 0.5 6820 5432 pts/26 S 10:53 0:01 ruby
a.rb
ahoward 15535 13.0 0.7 9676 8212 pts/26 S 10:53 0:01 ruby
a.rb
ahoward 15535 13.2 0.3 4552 3176 pts/26 S 10:53 0:01 ruby
a.rb
ahoward 15535 15.0 0.5 7156 5712 pts/26 S 10:53 0:01 ruby
a.rb
ahoward 15535 15.9 0.8 9684 8300 pts/26 S 10:53 0:01 ruby
a.rb
ahoward 15535 17.1 0.2 4432 3056 pts/26 S 10:53 0:02 ruby
a.rb
ahoward 15535 19.2 0.5 6892 5392 pts/26 S 10:53 0:02 ruby
a.rb
ahoward 15535 19.5 0.7 9228 7796 pts/26 S 10:53 0:02 ruby
a.rb
ahoward 15535 21.1 0.2 3712 2332 pts/26 S 10:53 0:02 ruby
a.rb
ahoward 15535 21.6 0.4 5908 4484 pts/26 S 10:53 0:03 ruby
a.rb

so it’s the handler/Buffer class that consumes (but does’t leak) so much
memory.

in any case, i think you are making the code too hard : ruby already
does
massive internal buffering - you can do what you want with less code i
think:

harp:~ > cat netstring.rb
class NetString < ::String
def initialize arg = “”
arg.respond_to?(“read”) ? read(arg) : super(arg)
end
def clear
self.replace “”
end
def read port
clear
self.replace(port.read(port.read(4).unpack(“N”).first))
end
def write port
port << [size].pack(“N”)
port << self
end
end

harp:~ > cat a.rb
require ‘socket’
require ‘netstring’

STDOUT.sync = true

ss = TCPServer.new ‘127.0.0.1’, 10001

loop do
s = ss.accept
puts “got connection”
Thread.start(s) do |s|
File.open("/dev/null", “w”) do |f|
ns = NetString.new s
puts “received #{ ns.size } bytes”
written = f.write ns
puts “wrote #{ written } bytes”
end
end
end

harp:~ > cat b.rb
require ‘socket’
require ‘netstring’

loop do
TCPSocket.open(‘127.0.0.1’, 10001) do |t|
NetString::new(“a” * 65536).write t
end
end

harp:~ > ruby a.rb # terminal one

harp:~ > ruby b.rb # terminal two

harp:~ > ps aux|head -1;{ while true;do ps aux|grep ‘ruby a.rb’|grep
-v grep;sleep 1;done; }|head -42
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
COMMAND
ahoward 17990 10.7 1.1 13480 11868 pts/26 S 11:30 0:27 ruby
a.rb
ahoward 17990 10.7 1.1 13164 11704 pts/26 S 11:30 0:27 ruby
a.rb
ahoward 17990 10.7 1.1 13660 12144 pts/26 S 11:30 0:27 ruby
a.rb
ahoward 17990 10.7 1.1 13424 11904 pts/26 S 11:30 0:27 ruby
a.rb
ahoward 17990 10.7 1.1 13328 11792 pts/26 S 11:30 0:27 ruby
a.rb
ahoward 17990 10.7 1.1 13784 12184 pts/26 S 11:30 0:27 ruby
a.rb
ahoward 17990 10.7 0.9 11040 9496 pts/26 S 11:30 0:27 ruby
a.rb
ahoward 17990 10.8 0.9 10904 9428 pts/26 S 11:30 0:28 ruby
a.rb
ahoward 17990 10.7 0.9 11968 10220 pts/26 S 11:30 0:28 ruby
a.rb
ahoward 17990 10.7 1.1 13464 12056 pts/26 S 11:30 0:28 ruby
a.rb
ahoward 17990 10.7 0.9 11000 9476 pts/26 S 11:30 0:28 ruby
a.rb
ahoward 17990 10.7 0.9 10992 9472 pts/26 S 11:30 0:28 ruby
a.rb
ahoward 17990 10.8 1.1 13364 11912 pts/26 S 11:30 0:28 ruby
a.rb
ahoward 17990 10.8 0.9 10976 9464 pts/26 S 11:30 0:28 ruby
a.rb
ahoward 17990 10.8 1.1 13356 11908 pts/26 S 11:30 0:28 ruby
a.rb
ahoward 17990 10.8 1.1 13796 12228 pts/26 S 11:30 0:29 ruby
a.rb
ahoward 17990 10.8 1.1 13572 12116 pts/26 S 11:30 0:29 ruby
a.rb
ahoward 17990 10.8 1.1 13836 12244 pts/26 S 11:30 0:29 ruby
a.rb
ahoward 17990 10.8 1.1 13872 12284 pts/26 S 11:30 0:29 ruby
a.rb
ahoward 17990 10.8 1.2 14112 12436 pts/26 S 11:30 0:29 ruby
a.rb
ahoward 17990 10.8 1.2 14224 12428 pts/26 S 11:30 0:29 ruby
a.rb
ahoward 17990 10.8 1.1 13888 12292 pts/26 S 11:30 0:29 ruby
a.rb
ahoward 17990 10.8 1.1 13896 12296 pts/26 S 11:30 0:29 ruby
a.rb
ahoward 17990 10.8 1.2 14200 12416 pts/26 S 11:30 0:30 ruby
a.rb
ahoward 17990 10.8 1.1 13856 12276 pts/26 S 11:30 0:30 ruby
a.rb
ahoward 17990 10.8 1.2 13960 12328 pts/26 S 11:30 0:30 ruby
a.rb
ahoward 17990 10.8 1.1 13624 12164 pts/26 S 11:30 0:30 ruby
a.rb
ahoward 17990 10.8 1.2 14048 12384 pts/26 S 11:30 0:30 ruby
a.rb
ahoward 17990 10.8 1.1 13984 12308 pts/26 S 11:30 0:30 ruby
a.rb
ahoward 17990 10.8 1.2 14088 12424 pts/26 S 11:30 0:30 ruby
a.rb
ahoward 17990 10.8 1.1 13744 12220 pts/26 S 11:30 0:30 ruby
a.rb
ahoward 17990 10.8 1.2 14040 12396 pts/26 S 11:30 0:31 ruby
a.rb
ahoward 17990 10.8 1.1 13496 12100 pts/26 S 11:30 0:31 ruby
a.rb
ahoward 17990 10.8 1.2 14056 12472 pts/26 S 11:30 0:31 ruby
a.rb
ahoward 17990 10.8 1.2 14080 12452 pts/26 S 11:30 0:31 ruby
a.rb
ahoward 17990 10.8 1.1 13520 12108 pts/26 S 11:30 0:31 ruby
a.rb
ahoward 17990 10.8 1.1 13656 12276 pts/26 S 11:30 0:31 ruby
a.rb
ahoward 17990 10.8 1.1 13624 12160 pts/26 S 11:30 0:31 ruby
a.rb
ahoward 17990 10.8 1.2 13824 12392 pts/26 S 11:30 0:31 ruby
a.rb
ahoward 17990 10.8 1.2 13912 12432 pts/26 S 11:30 0:31 ruby
a.rb
ahoward 17990 10.8 1.2 14152 12520 pts/26 S 11:30 0:31 ruby
a.rb
ahoward 17990 10.8 1.1 13768 12232 pts/26 S 11:30 0:32 ruby
a.rb

kind regards.

-a

ahoward wrote:

if you looking for why your code is so big look to you handler:
changing it
to

Thread.start(s){|s| loop{ s.read(8192) or break }}

will result in runs that look like this:

Interesting.

so it’s the handler/Buffer class that consumes (but does’t leak) so much
memory.

in any case, i think you are making the code too hard : ruby already
does
massive internal buffering - you can do what you want with less code i
think:

harp:~ > cat netstring.rb
class NetString < ::String
def initialize arg = “”
arg.respond_to?(“read”) ? read(arg) : super(arg)
end
def clear
self.replace “”
end
def read port
clear
self.replace(port.read(port.read(4).unpack(“N”).first))
end
def write port
port << [size].pack(“N”)
port << self
end
end

harp:~ > cat a.rb
require ‘socket’
require ‘netstring’

STDOUT.sync = true

ss = TCPServer.new ‘127.0.0.1’, 10001

loop do
s = ss.accept
puts “got connection”
Thread.start(s) do |s|
File.open("/dev/null", “w”) do |f|
ns = NetString.new s
puts “received #{ ns.size } bytes”
written = f.write ns
puts “wrote #{ written } bytes”
end
end
end

harp:~ > cat b.rb
require ‘socket’
require ‘netstring’

loop do
TCPSocket.open(‘127.0.0.1’, 10001) do |t|
NetString::new(“a” * 65536).write t
end
end

Thanks, a. I will try to incorporate your suggestions when I get it
working again.


Toby DiPasquale

Booker C.Bense wrote:

_ This is generally true of all unix processes. They do not
return allocated memory back to the system until they exit.
Freeing memory inside the code merely allows you to use it
again, it does not make it available to the system.

I thought they did return freed memory. Am I misreading the following?

irb(main):003:0> puts ps
PID TTY TIME CMD
7791 pts/7 00:00:00 zsh
13495 pts/7 00:00:00 irb
13501 pts/7 00:00:00 ps
=> nil
irb(main):004:0> puts ps -Ovsz
PID VSZ S TTY TIME COMMAND
7791 6068 S pts/7 00:00:00 /bin/zsh
13495 8404 S pts/7 00:00:00 irb
13502 3664 R pts/7 00:00:00 ps -Ovsz
=> nil
irb(main):005:0> puts ps -Ovsz,rss
PID VSZ RSS S TTY TIME COMMAND
7791 6068 2736 S pts/7 00:00:00 /bin/zsh
13495 8404 5100 S pts/7 00:00:00 irb
13503 3668 888 R pts/7 00:00:00 ps -Ovsz,rss
=> nil
irb(main):006:0> def usemem(sz); @x = Array.new(sz); nil; end
=> nil
irb(main):007:0> usemem 10_000_000
=> nil
irb(main):008:0> puts ps -Ovsz,rss
PID VSZ RSS S TTY TIME COMMAND
7791 6068 2736 S pts/7 00:00:00 /bin/zsh
13495 47468 44192 S pts/7 00:00:00 irb
13504 3668 892 R pts/7 00:00:00 ps -Ovsz,rss
=> nil
irb(main):009:0> usemem 10_000
=> nil
irb(main):010:0> puts ps -Ovsz,rss
PID VSZ RSS S TTY TIME COMMAND
7791 6068 2736 S pts/7 00:00:00 /bin/zsh
13495 47468 44232 S pts/7 00:00:00 irb
13505 3664 884 R pts/7 00:00:00 ps -Ovsz,rss
=> nil
irb(main):011:0> GC.start
=> nil
irb(main):012:0> puts ps -Ovsz,rss
PID VSZ RSS S TTY TIME COMMAND
7791 6068 2736 S pts/7 00:00:00 /bin/zsh
13495 8404 5168 S pts/7 00:00:00 irb
13506 3664 888 R pts/7 00:00:00 ps -Ovsz,rss
=> nil