Forum: Ruby Writing long-running daemons without memory leaks?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
159cd33667611645164e8623de42f67e?d=identicon&s=25 Toby DiPasquale (Guest)
on 2006-03-17 16:36
Hi all,

I am in the middle of writing some code that will exist as a
long-running process on a number of machines. I am having trouble
keeping memory utilization down, which is important in this case,
because other processes on these machines will have priority to use the
majority of RAM on these boxes for other tasks. The daemon itself is
pretty simple, just reading and writing files and keeping some small
state, but it also needs to cache large writes from multiple clients.
When it does this, I find that this memory is not freed.

My question is, I've heard from numerous accounts that WEBrick also
leaks memory, and I would suspect that Mongrel does, too (though I
haven't confirmed that) for the same reason. I understand that Ruby's GC
is conservative, but can anyone point me to documentation, reference,
source code, anything really that will help me better understand how to
write Ruby code that doesn't "leak" and can be sustained as a
long-running process without an ever-increasing memory footprint?

Any help would be greatly appreciated. Thanks in advance.

P.S. I've read the Pickaxe's lone page on the subject (in the Duck
Typing chapter) and I've seen Why's article about the same
(http://whytheluckystiff.net/articles/theFullyUptur...).

P.P.S. Has Minero Aoki's Ruby book been translated to English anywhere?
I hear that it has a whole chapter on GC...

--
Toby DiPasquale
Cb48ca5059faf7409a5ab3745a964696?d=identicon&s=25 unknown (Guest)
on 2006-03-17 16:48
(Received via mailing list)
On Sat, 18 Mar 2006, Toby DiPasquale wrote:

>
> P.S. I've read the Pickaxe's lone page on the subject (in the Duck
> Typing chapter) and I've seen Why's article about the same
> (http://whytheluckystiff.net/articles/theFullyUptur...).
>
> P.P.S. Has Minero Aoki's Ruby book been translated to English anywhere?
> I hear that it has a whole chapter on GC...

i'm running dozens of ruby daemons, some of which are extremely (months
at a
time) long lived and have not seen any gc issues.  can you post a
specific
(minimal) example that you find leaks memory?

regards.

-a
159cd33667611645164e8623de42f67e?d=identicon&s=25 Toby DiPasquale (Guest)
on 2006-03-17 17:29
unknown wrote:
> i'm running dozens of ruby daemons, some of which are extremely (months
> at a
> time) long lived and have not seen any gc issues.  can you post a
> specific
> (minimal) example that you find leaks memory?

Sure. Here is a server and client, resp, that exhibit the behavior I am
referring to:

puma:> cat leak.rb
# vim:set ts=4 sw=4 ai:
require 'socket'

class Buffer
    def initialize size
        @size = size
        @buffer = []
        @length = 0
    end
    attr_reader :length, :size
    def full?  ; @length == @size end
    def empty? ; @length.zero? end
    # Lets you fill up to capacity and then returns
    # what's left over
    def fill data
        if full?
            data
        elsif @length + data.length < @size
            @buffer << data
            @length += data.length
            nil
        else
            l = @size - @length
            @buffer << data[0, l]
            data[l..-1]
        end
    end
    def expunge
        buf = @buffer.join
        @buffer.clear
        @length = 0
        buf
    end
end     # class Buffer

def read_long s
    s.read( 4).unpack( "N")[0]
end

# main client handling thread logic
client_handler = lambda do |s|
    begin
        len = read_long s
        buf = Buffer.new len
        until buf.full?
            x = s.read 4096
            break if x.nil?
            buf.fill x
        end
        puts "writing #{buf.length} bytes"
        f = File.open "/dev/null", "w"
        f.write buf.expunge
        f.close
    ensure
        s.close
    end
end

# main server loop
ss = TCPServer.new '127.0.0.1', 10001
begin
    while true
        s = ss.accept
        puts "got connection"
        Thread.start s, &client_handler
    end
ensure
    ss.close
end

puma:~> cat leak_client.rb
# vim:set ts=4 sw=4 ai:
require 'socket'

def write_long s, l
        s.write( [l].pack( "N"))
end

str = "a" * 65536

t = TCPSocket.new '127.0.0.1', 10001
write_long t, 1024 * str.length
1024.times { t.write str }
t.close

puma:~>

And here's the output from a simple test to show the issue:

puma:~> for i in 1 2 3 4 5 6 7 8 9 10; do ps aux | grep [l]eak.rb ; ruby
leak_client.rb ; done
toby      8452  0.0  0.1   3116  1684 pts/3    S+   11:23   0:00 ruby
leak.rb
toby      8452  3.8 12.8 143032 133432 pts/3   Sl+  11:23   0:01 ruby
leak.rb
toby      8452  6.4 15.8 173664 164020 pts/3   Sl+  11:23   0:02 ruby
leak.rb
toby      8452  7.8 15.2 167988 158472 pts/3   Sl+  11:23   0:03 ruby
leak.rb
toby      8452  9.2 15.8 173532 164004 pts/3   Sl+  11:23   0:03 ruby
leak.rb
toby      8452 10.7 15.2 168024 158508 pts/3   Sl+  11:23   0:04 ruby
leak.rb
toby      8452 12.0 15.8 173568 164000 pts/3   Sl+  11:23   0:04 ruby
leak.rb
toby      8452 13.2 15.8 173568 164008 pts/3   Sl+  11:23   0:05 ruby
leak.rb
toby      8452 14.3 15.8 173568 164012 pts/3   Sl+  11:23   0:06 ruby
leak.rb
toby      8452 15.7 15.0 165012 155496 pts/3   Sl+  11:23   0:06 ruby
leak.rb
puma:~> ps aux | grep [l]eak.rb
toby      8452 13.0 22.1 239132 229544 pts/3   Sl+  11:23   0:07 ruby
leak.rb
puma:~>

This memory never really goes away. Also notice that the client is
sending 64MB, but the first time the leak.rb image jumps to 143MB. Any
clues? Am I just doing something really stupid? TIA.

--
Toby DiPasquale
Ab870531383eea6e4d9110317f5401e7?d=identicon&s=25 Caleb Clausen (Guest)
on 2006-03-17 18:01
(Received via mailing list)
On 3/17/06, Toby DiPasquale <toby@cbcg.net> wrote:
> I am in the middle of writing some code that will exist as a
> long-running process on a number of machines. I am having trouble
> keeping memory utilization down, which is important in this case,
[snip]
> source code, anything really that will help me better understand how to
> write Ruby code that doesn't "leak" and can be sustained as a
> long-running process without an ever-increasing memory footprint?
>
> Any help would be greatly appreciated. Thanks in advance.

Maybe you already knew this stuff, but, here's some things that might
help you:

Set things to nil after you're finished using them.

Avoid continuations. If I recall correctly, the GC doesn't handle them
right.

Be careful with blocks (and bindings); they keep a reference to all
local variables that are (potentially) visible in the block, even if
those variables aren't actually used in the block. Any values attached
to those local vars not be garbage collected until the whole block is.
Here's an example:

  def foo
     a1=%w[foo bar baz]
     a2=%w[shish boom bah]
     a3=%w[this one's actually used]
     return bar { p a3 }
  end

  def bar(&block)
    return block
  end

a1 and a2 are leaked (temporarily). Even tho they're unused, they
don't actually get GC'd until block is also GC'd.

You can also use ObjectSpace.each_object to iterate over currently
live objects; this might help you determine which objects are being
leaked.
Cb48ca5059faf7409a5ab3745a964696?d=identicon&s=25 unknown (Guest)
on 2006-03-17 18:28
(Received via mailing list)
On Sat, 18 Mar 2006, Toby DiPasquale wrote:

>
> Sure. Here is a server and client, resp, that exhibit the behavior I am
> referring to:

i cannot reproduce it:


     harp:~ > cat a.rb
     require 'socket'

     class Buffer
        def initialize size
            @size = size
            @buffer = []
            @length = 0
        end
        attr_reader :length, :size
        def full?  ; @length == @size end
        def empty? ; @length.zero? end
        # Lets you fill up to capacity and then returns
        # what's left over
        def fill data
            if full?
                data
            elsif @length + data.length < @size
                @buffer << data
                @length += data.length
                nil
            else
                l = @size - @length
                @buffer << data[0, l]
                data[l..-1]
            end
        end
        def expunge
            buf = @buffer.join
            @buffer.clear
            @length = 0
            buf
        end
     end     # class Buffer

     def read_long s
        s.read( 4).unpack( "N")[0]
     end

     # main client handling thread logic
     client_handler = lambda do |s|
        begin
            len = read_long s
            buf = Buffer.new len
            until buf.full?
                x = s.read 4096
                break if x.nil?
                buf.fill x
            end
            puts "writing #{buf.length} bytes"
            f = File.open "/dev/null", "w"
            f.write buf.expunge
            f.close
        ensure
            s.close
        end
     end

     # main server loop
     ss = TCPServer.new '127.0.0.1', 10001
     begin
        while true
            s = ss.accept
            puts "got connection"
            Thread.start s, &client_handler
        end
     ensure
        ss.close
     end



     harp:~ > cat b.rb
     require 'socket'

     def write_long s, l
            s.write( [l].pack( "N"))
     end

     str = "a" * 65536

     t = TCPSocket.new '127.0.0.1', 10001
     write_long t, 1024 * str.length
     1024.times { t.write str }
     t.close



     harp:~ > ps aux|head -1;  while true;do ruby b.rb && ps aux|grep
'ruby a.rb'|grep -v grep ;done
     USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME
COMMAND
     ahoward  13671 12.7 23.1 239328 237956 pts/26 S   10:14   0:44 ruby
a.rb
     ahoward  13671 12.7 21.2 219404 218032 pts/26 S   10:14   0:45 ruby
a.rb
     ahoward  13671 12.8 22.3 230784 229396 pts/26 S   10:14   0:45 ruby
a.rb
     ahoward  13671 13.0 22.3 230784 229412 pts/26 S   10:14   0:45 ruby
a.rb
     ahoward  13671 13.1 22.3 230916 229416 pts/26 S   10:14   0:46 ruby
a.rb
     ahoward  13671 13.2 15.9 165364 163876 pts/26 S   10:14   0:46 ruby
a.rb
     ahoward  13671 13.3 22.3 230904 229424 pts/26 S   10:14   0:47 ruby
a.rb
     ahoward  13671 13.4 21.5 225116 221504 pts/26 R   10:14   0:47 ruby
a.rb
     ahoward  13671 13.5 28.7 296328 294940 pts/26 S   10:14   0:48 ruby
a.rb
     ahoward  13671 13.6 21.5 222480 221108 pts/26 S   10:14   0:48 ruby
a.rb
     ahoward  13671 13.7 22.3 230796 229400 pts/26 S   10:14   0:49 ruby
a.rb
     ahoward  13671 13.8 21.8 230804 224296 pts/26 R   10:14   0:49 ruby
a.rb
     ahoward  13671 13.9 27.3 296344 281256 pts/26 R   10:14   0:50 ruby
a.rb
     ahoward  13671 14.0 25.4 265672 261500 pts/26 R   10:14   0:50 ruby
a.rb
     ahoward  13671 14.1 22.3 230880 229384 pts/26 S   10:14   0:51 ruby
a.rb
     ahoward  13671 14.2 15.9 165340 163864 pts/26 S   10:14   0:51 ruby
a.rb
     ahoward  13671 14.3 20.8 230888 214376 pts/26 R   10:14   0:52 ruby
a.rb
     ahoward  13671 14.4 19.3 200220 198844 pts/26 S   10:14   0:52 ruby
a.rb
     ahoward  13671 14.5 21.2 230852 218484 pts/26 R   10:14   0:53 ruby
a.rb
     ahoward  13671 14.6 15.9 165312 163872 pts/26 R   10:14   0:53 ruby
a.rb
     ahoward  13671 14.8 28.1 290632 289256 pts/26 S   10:14   0:54 ruby
a.rb
     ahoward  13671 14.8 16.0 230776 164724 pts/26 R   10:14   0:54 ruby
a.rb
     ahoward  13671 14.9 21.6 230908 222412 pts/26 R   10:14   0:54 ruby
a.rb
     ahoward  13671 15.0 15.0 219356 154356 pts/26 R   10:14   0:55 ruby
a.rb
     ahoward  13671 15.1 16.2 230864 166464 pts/26 R   10:14   0:55 ruby
a.rb
     ahoward  13671 15.2 25.7 265764 264320 pts/26 S   10:14   0:56 ruby
a.rb
     ahoward  13671 15.4 22.3 230852 229388 pts/26 S   10:14   0:56 ruby
a.rb
     ahoward  13671 15.4 15.9 165312 163864 pts/26 S   10:14   0:57 ruby
a.rb
     ahoward  13671 15.5 15.9 165320 163880 pts/26 R   10:14   0:57 ruby
a.rb
     ahoward  13671 15.6 28.5 294368 292992 pts/26 S   10:14   0:58 ruby
a.rb
     ahoward  13671 15.7 22.3 230808 229388 pts/26 S   10:14   0:58 ruby
a.rb
     ahoward  13671 15.8 21.7 230816 223624 pts/26 R   10:14   0:59 ruby
a.rb
     ahoward  13671 15.9 15.9 165276 163872 pts/26 R   10:14   0:59 ruby
a.rb
     ahoward  13671 16.0 20.6 213384 212008 pts/26 S   10:14   0:59 ruby
a.rb
     ahoward  13671 16.1 22.3 230812 229396 pts/26 S   10:14   1:00 ruby
a.rb
     ahoward  13671 16.2 22.3 230812 229408 pts/26 S   10:14   1:00 ruby
a.rb
     ahoward  13671 16.2 15.9 165280 163872 pts/26 R   10:14   1:01 ruby
a.rb
     ahoward  13671 16.3 22.3 230812 229408 pts/26 S   10:14   1:01 ruby
a.rb
     ahoward  13671 16.4 19.3 200212 198836 pts/26 S   10:14   1:02 ruby
a.rb
     ahoward  13671 16.5 22.3 230840 229400 pts/26 S   10:14   1:02 ruby
a.rb
     ahoward  13671 16.5 15.4 159584 158204 pts/26 R   10:14   1:02 ruby
a.rb
     ahoward  13671 16.7 27.4 296424 281408 pts/26 R   10:14   1:03 ruby
a.rb
     ahoward  13671 16.8 19.3 203424 199044 pts/26 R   10:14   1:03 ruby
a.rb
     ahoward  13671 16.9 22.1 230888 227576 pts/26 R   10:14   1:04 ruby
a.rb
     ahoward  13671 17.0 25.7 265720 264108 pts/26 S   10:14   1:04 ruby
a.rb
     ahoward  13671 17.0 15.9 230812 163952 pts/26 R   10:14   1:05 ruby
a.rb
     ahoward  13671 17.1 27.1 296352 278484 pts/26 R   10:14   1:05 ruby
a.rb
     ahoward  13671 17.2 28.1 290640 289264 pts/26 S   10:14   1:06 ruby
a.rb
     ahoward  13671 17.3 15.9 165248 163860 pts/26 R   10:14   1:06 ruby
a.rb
     ahoward  13671 17.4 21.8 230920 224092 pts/26 R   10:14   1:07 ruby
a.rb
     ahoward  13671 17.5 22.0 228288 226912 pts/26 S   10:14   1:07 ruby
a.rb
     ahoward  13671 17.5 15.9 165264 163872 pts/26 R   10:14   1:07 ruby
a.rb
     ahoward  13671 17.6 22.0 228228 226852 pts/26 S   10:14   1:08 ruby
a.rb
     ahoward  13671 17.7 22.3 230868 229400 pts/26 S   10:14   1:08 ruby
a.rb
     ahoward  13671 17.8 14.8 153684 152304 pts/26 R   10:14   1:09 ruby
a.rb
     ahoward  13671 17.9 22.3 230812 229400 pts/26 S   10:14   1:09 ruby
a.rb
     ahoward  13671 18.0 21.7 225108 223732 pts/26 S   10:14   1:10 ruby
a.rb
     ahoward  13671 18.1 20.6 230924 212128 pts/26 R   10:14   1:10 ruby
a.rb
     ahoward  13671 18.1 14.4 149944 148564 pts/26 R   10:14   1:11 ruby
a.rb
     ahoward  13671 18.3 28.7 296332 294936 pts/26 S   10:14   1:11 ruby
a.rb
     ahoward  13671 18.3 19.5 212736 200952 pts/26 R   10:14   1:12 ruby
a.rb
     ahoward  13671 18.4 21.1 230824 216768 pts/26 R   10:14   1:12 ruby
a.rb
     ahoward  13671 18.5 22.3 230816 229404 pts/26 S   10:14   1:12 ruby
a.rb


so memory builds to about 25 and, presumably when the gc kicks in, drops
to 15
- but it certainly stays in this range.

what os and ruby version?

-a
159cd33667611645164e8623de42f67e?d=identicon&s=25 Toby DiPasquale (Guest)
on 2006-03-17 18:44
unknown wrote:
>      harp:~ > ps aux|head -1;  while true;do ruby b.rb && ps aux|grep
[...]
>      ahoward  13671 18.3 19.5 212736 200952 pts/26 R   10:14   1:12 ruby
> a.rb
>      ahoward  13671 18.4 21.1 230824 216768 pts/26 R   10:14   1:12 ruby
> a.rb
>      ahoward  13671 18.5 22.3 230816 229404 pts/26 S   10:14   1:12 ruby
> a.rb
>
>
> so memory builds to about 25 and, presumably when the gc kicks in, drops
> to 15
> - but it certainly stays in this range.
>
> what os and ruby version?

ruby 1.8.4 on Ubuntu Linux, kernel 2.6.12-10 on a P4 Thinkpad with 1GB
RAM. The other systems having this problem are all also running Linux
2.6.x with ruby 1.8.4.

One thing about the above: you do realize that's ~250MB and ~150MB, not
25 and 15, right?

--
Toby DiPasquale
Cb48ca5059faf7409a5ab3745a964696?d=identicon&s=25 unknown (Guest)
on 2006-03-17 19:38
(Received via mailing list)
On Sat, 18 Mar 2006, Toby DiPasquale wrote:

>>
> One thing about the above: you do realize that's ~250MB and ~150MB, not
> 25 and 15, right?

yes - but it's definitely not leaking.  i'm running

   harp:~ > ruby -v
   ruby 1.8.4 (2006-01-12) [i686-linux]

   harp:~ > cat /etc/redhat-release
   Red Hat Enterprise Linux WS release 3 (Taroon Update 6)

   harp:~ > uname -srm
   Linux 2.4.21-37.0.1.EL i686

btw.

it's clear to me that your code is using tons of memory - it's not clear
that
it's leaking, however.

if you looking for why your code is so big look to you handler:
changing it
to

   Thread.start(s){|s| loop{ s.read(8192) or break }}

will result in runs that look like this:

   harp:~ > ps aux|head -1;  while true;do ruby b.rb && ps aux|grep
'ruby a.rb'|grep -v grep ;done
   USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME
COMMAND
   ahoward  15535  4.1  0.4  6004 4556 pts/26   S    10:53   0:00 ruby
a.rb
   ahoward  15535  7.1  0.7  8940 7520 pts/26   S    10:53   0:00 ruby
a.rb
   ahoward  15535  9.4  0.2  4080 2704 pts/26   S    10:53   0:00 ruby
a.rb
   ahoward  15535 10.7  0.5  6820 5432 pts/26   S    10:53   0:01 ruby
a.rb
   ahoward  15535 13.0  0.7  9676 8212 pts/26   S    10:53   0:01 ruby
a.rb
   ahoward  15535 13.2  0.3  4552 3176 pts/26   S    10:53   0:01 ruby
a.rb
   ahoward  15535 15.0  0.5  7156 5712 pts/26   S    10:53   0:01 ruby
a.rb
   ahoward  15535 15.9  0.8  9684 8300 pts/26   S    10:53   0:01 ruby
a.rb
   ahoward  15535 17.1  0.2  4432 3056 pts/26   S    10:53   0:02 ruby
a.rb
   ahoward  15535 19.2  0.5  6892 5392 pts/26   S    10:53   0:02 ruby
a.rb
   ahoward  15535 19.5  0.7  9228 7796 pts/26   S    10:53   0:02 ruby
a.rb
   ahoward  15535 21.1  0.2  3712 2332 pts/26   S    10:53   0:02 ruby
a.rb
   ahoward  15535 21.6  0.4  5908 4484 pts/26   S    10:53   0:03 ruby
a.rb

so it's the handler/Buffer class that consumes (but does't leak) so much
memory.

in any case, i think you are making the code too hard : ruby already
does
massive internal buffering - you can do what you want with less code i
think:

   harp:~ > cat netstring.rb
   class NetString < ::String
     def initialize arg = ""
       arg.respond_to?("read") ? read(arg) : super(arg)
     end
     def clear
       self.replace ""
     end
     def read port
       clear
       self.replace(port.read(port.read(4).unpack("N").first))
     end
     def write port
       port << [size].pack("N")
       port << self
     end
   end

   harp:~ > cat a.rb
   require 'socket'
   require 'netstring'

   STDOUT.sync = true

   ss = TCPServer.new '127.0.0.1', 10001

   loop do
     s = ss.accept
     puts "got connection"
     Thread.start(s) do |s|
       File.open("/dev/null", "w") do |f|
         ns = NetString.new s
         puts "received #{ ns.size } bytes"
         written = f.write ns
         puts "wrote #{ written } bytes"
       end
     end
   end


   harp:~ > cat b.rb
   require 'socket'
   require 'netstring'

   loop do
     TCPSocket.open('127.0.0.1', 10001) do |t|
       NetString::new("a" * 65536).write t
     end
   end


   harp:~ > ruby a.rb  # terminal one

   harp:~ > ruby b.rb  # terminal two

   harp:~ > ps aux|head -1;{ while true;do ps aux|grep 'ruby a.rb'|grep
-v grep;sleep 1;done; }|head -42
   USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME
COMMAND
   ahoward  17990 10.7  1.1 13480 11868 pts/26  S    11:30   0:27 ruby
a.rb
   ahoward  17990 10.7  1.1 13164 11704 pts/26  S    11:30   0:27 ruby
a.rb
   ahoward  17990 10.7  1.1 13660 12144 pts/26  S    11:30   0:27 ruby
a.rb
   ahoward  17990 10.7  1.1 13424 11904 pts/26  S    11:30   0:27 ruby
a.rb
   ahoward  17990 10.7  1.1 13328 11792 pts/26  S    11:30   0:27 ruby
a.rb
   ahoward  17990 10.7  1.1 13784 12184 pts/26  S    11:30   0:27 ruby
a.rb
   ahoward  17990 10.7  0.9 11040 9496 pts/26   S    11:30   0:27 ruby
a.rb
   ahoward  17990 10.8  0.9 10904 9428 pts/26   S    11:30   0:28 ruby
a.rb
   ahoward  17990 10.7  0.9 11968 10220 pts/26  S    11:30   0:28 ruby
a.rb
   ahoward  17990 10.7  1.1 13464 12056 pts/26  S    11:30   0:28 ruby
a.rb
   ahoward  17990 10.7  0.9 11000 9476 pts/26   S    11:30   0:28 ruby
a.rb
   ahoward  17990 10.7  0.9 10992 9472 pts/26   S    11:30   0:28 ruby
a.rb
   ahoward  17990 10.8  1.1 13364 11912 pts/26  S    11:30   0:28 ruby
a.rb
   ahoward  17990 10.8  0.9 10976 9464 pts/26   S    11:30   0:28 ruby
a.rb
   ahoward  17990 10.8  1.1 13356 11908 pts/26  S    11:30   0:28 ruby
a.rb
   ahoward  17990 10.8  1.1 13796 12228 pts/26  S    11:30   0:29 ruby
a.rb
   ahoward  17990 10.8  1.1 13572 12116 pts/26  S    11:30   0:29 ruby
a.rb
   ahoward  17990 10.8  1.1 13836 12244 pts/26  S    11:30   0:29 ruby
a.rb
   ahoward  17990 10.8  1.1 13872 12284 pts/26  S    11:30   0:29 ruby
a.rb
   ahoward  17990 10.8  1.2 14112 12436 pts/26  S    11:30   0:29 ruby
a.rb
   ahoward  17990 10.8  1.2 14224 12428 pts/26  S    11:30   0:29 ruby
a.rb
   ahoward  17990 10.8  1.1 13888 12292 pts/26  S    11:30   0:29 ruby
a.rb
   ahoward  17990 10.8  1.1 13896 12296 pts/26  S    11:30   0:29 ruby
a.rb
   ahoward  17990 10.8  1.2 14200 12416 pts/26  S    11:30   0:30 ruby
a.rb
   ahoward  17990 10.8  1.1 13856 12276 pts/26  S    11:30   0:30 ruby
a.rb
   ahoward  17990 10.8  1.2 13960 12328 pts/26  S    11:30   0:30 ruby
a.rb
   ahoward  17990 10.8  1.1 13624 12164 pts/26  S    11:30   0:30 ruby
a.rb
   ahoward  17990 10.8  1.2 14048 12384 pts/26  S    11:30   0:30 ruby
a.rb
   ahoward  17990 10.8  1.1 13984 12308 pts/26  S    11:30   0:30 ruby
a.rb
   ahoward  17990 10.8  1.2 14088 12424 pts/26  S    11:30   0:30 ruby
a.rb
   ahoward  17990 10.8  1.1 13744 12220 pts/26  S    11:30   0:30 ruby
a.rb
   ahoward  17990 10.8  1.2 14040 12396 pts/26  S    11:30   0:31 ruby
a.rb
   ahoward  17990 10.8  1.1 13496 12100 pts/26  S    11:30   0:31 ruby
a.rb
   ahoward  17990 10.8  1.2 14056 12472 pts/26  S    11:30   0:31 ruby
a.rb
   ahoward  17990 10.8  1.2 14080 12452 pts/26  S    11:30   0:31 ruby
a.rb
   ahoward  17990 10.8  1.1 13520 12108 pts/26  S    11:30   0:31 ruby
a.rb
   ahoward  17990 10.8  1.1 13656 12276 pts/26  S    11:30   0:31 ruby
a.rb
   ahoward  17990 10.8  1.1 13624 12160 pts/26  S    11:30   0:31 ruby
a.rb
   ahoward  17990 10.8  1.2 13824 12392 pts/26  S    11:30   0:31 ruby
a.rb
   ahoward  17990 10.8  1.2 13912 12432 pts/26  S    11:30   0:31 ruby
a.rb
   ahoward  17990 10.8  1.2 14152 12520 pts/26  S    11:30   0:31 ruby
a.rb
   ahoward  17990 10.8  1.1 13768 12232 pts/26  S    11:30   0:32 ruby
a.rb


kind regards.


-a
159cd33667611645164e8623de42f67e?d=identicon&s=25 Toby DiPasquale (Guest)
on 2006-03-17 20:25
ahoward wrote:
> if you looking for why your code is so big look to you handler:
> changing it
> to
>
>    Thread.start(s){|s| loop{ s.read(8192) or break }}
>
> will result in runs that look like this:

<snip>

Interesting.

> so it's the handler/Buffer class that consumes (but does't leak) so much
> memory.
>
> in any case, i think you are making the code too hard : ruby already
> does
> massive internal buffering - you can do what you want with less code i
> think:
>
>    harp:~ > cat netstring.rb
>    class NetString < ::String
>      def initialize arg = ""
>        arg.respond_to?("read") ? read(arg) : super(arg)
>      end
>      def clear
>        self.replace ""
>      end
>      def read port
>        clear
>        self.replace(port.read(port.read(4).unpack("N").first))
>      end
>      def write port
>        port << [size].pack("N")
>        port << self
>      end
>    end
>
>    harp:~ > cat a.rb
>    require 'socket'
>    require 'netstring'
>
>    STDOUT.sync = true
>
>    ss = TCPServer.new '127.0.0.1', 10001
>
>    loop do
>      s = ss.accept
>      puts "got connection"
>      Thread.start(s) do |s|
>        File.open("/dev/null", "w") do |f|
>          ns = NetString.new s
>          puts "received #{ ns.size } bytes"
>          written = f.write ns
>          puts "wrote #{ written } bytes"
>        end
>      end
>    end
>
>
>    harp:~ > cat b.rb
>    require 'socket'
>    require 'netstring'
>
>    loop do
>      TCPSocket.open('127.0.0.1', 10001) do |t|
>        NetString::new("a" * 65536).write t
>      end
>    end
>

<snip>

Thanks, a. I will try to incorporate your suggestions when I get it
working again.

--
Toby DiPasquale
1835ad5f9240a272c2990c55873904b7?d=identicon&s=25 Booker C.Bense (Guest)
on 2006-03-17 22:36
(Received via mailing list)
-----BEGIN PGP SIGNED MESSAGE-----

In article <7d20c3f1c9ff08ff73faab5feefc8eb2@ruby-forum.com>,
Toby DiPasquale  <toby@cbcg.net> wrote:
>
_ This is generally true of all unix processes. They do not
return allocated memory back to the system until they exit.
Freeing memory inside the code merely allows you to use it
again, it does not make it available to the system.

_ If your process is not using the memory, it may stay swapped
out, ( look at the real and virtual sizes ). If this is really
an issue, the standard way around it is to do the memory
intensive part in a forked subprocess that exits after it's
done.

_ Booker C. Bense


-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQCVAwUBRBsqMGTWTAjn5N/lAQHVbwP/XYqiyZsWzCqsYNfzP4rCe5cEzWbKBBfU
54A/x/DDnImzpM3MYcIG4Um2AAGFBYMrylxEeTv0Ot6RMi7Nl2NJLIzIg/6lGh1M
6d1ojbnddmUGfJXuYDZ2/lcWDWYzYCM6hYbWGh7h6eMYtNVZizE1k9ebA4zL+H0i
jFU8K8Aoe7k=
=ZbII
-----END PGP SIGNATURE-----
47b1910084592eb77a032bc7d8d1a84e?d=identicon&s=25 Joel VanderWerf (Guest)
on 2006-03-18 18:47
(Received via mailing list)
Booker C.Bense wrote:
> _ This is generally true of all unix processes. They do not
> return allocated memory back to the system until they exit.
> Freeing memory inside the code merely allows you to use it
> again, it does not make it available to the system.

I thought they did return freed memory. Am I misreading the following?


irb(main):003:0> puts `ps`
  PID TTY          TIME CMD
 7791 pts/7    00:00:00 zsh
13495 pts/7    00:00:00 irb
13501 pts/7    00:00:00 ps
=> nil
irb(main):004:0> puts `ps -Ovsz`
  PID    VSZ S TTY          TIME COMMAND
 7791   6068 S pts/7    00:00:00 /bin/zsh
13495   8404 S pts/7    00:00:00 irb
13502   3664 R pts/7    00:00:00 ps -Ovsz
=> nil
irb(main):005:0> puts `ps -Ovsz,rss`
  PID    VSZ   RSS S TTY          TIME COMMAND
 7791   6068  2736 S pts/7    00:00:00 /bin/zsh
13495   8404  5100 S pts/7    00:00:00 irb
13503   3668   888 R pts/7    00:00:00 ps -Ovsz,rss
=> nil
irb(main):006:0> def usemem(sz); @x = Array.new(sz); nil; end
=> nil
irb(main):007:0> usemem 10_000_000
=> nil
irb(main):008:0> puts `ps -Ovsz,rss`
  PID    VSZ   RSS S TTY          TIME COMMAND
 7791   6068  2736 S pts/7    00:00:00 /bin/zsh
13495  47468 44192 S pts/7    00:00:00 irb
13504   3668   892 R pts/7    00:00:00 ps -Ovsz,rss
=> nil
irb(main):009:0> usemem 10_000
=> nil
irb(main):010:0> puts `ps -Ovsz,rss`
  PID    VSZ   RSS S TTY          TIME COMMAND
 7791   6068  2736 S pts/7    00:00:00 /bin/zsh
13495  47468 44232 S pts/7    00:00:00 irb
13505   3664   884 R pts/7    00:00:00 ps -Ovsz,rss
=> nil
irb(main):011:0> GC.start
=> nil
irb(main):012:0> puts `ps -Ovsz,rss`
  PID    VSZ   RSS S TTY          TIME COMMAND
 7791   6068  2736 S pts/7    00:00:00 /bin/zsh
13495   8404  5168 S pts/7    00:00:00 irb
13506   3664   888 R pts/7    00:00:00 ps -Ovsz,rss
=> nil
This topic is locked and can not be replied to.