Ruby, memory and speed

Ruby
21

Le 18 août 06, à 00:07, John C. a écrit :

Stole that for
http://rubygarden.org:3000/Ruby/page/show/RubyOptimization

Suggestions for the Original Poster…

  • Browse that Wiki page, it may have something for you. (Alternatively,
    once you solve your problem add the solution to that page!)

Thanks for the pointer.

I also used Mmap#scan. It is pretty elegant compare to the usual:

io.each { |l|
l.chomp!
next unless l =? /blah/
…
}

I would think it is faster too (no formal testing done).

Watch the “si” and “so”. (Swap In Swap Out) If you are swapping 2 or
more swaps every 5 seconds, then you don’t have ruby GC problems, you
have memory problems. ie. Tweaking GC won’t help. You have to store
less
in ram full stop. Remember to set any dangling references that you
won’t
use again to nil, especially from class variables and globals.

Checked that. But no, the machine has plenty of memory and si/so was
always 0.

I finally went back to doing stream parsing. Instead of aggregating the
information from many file in one big hash, I read and write one record
at a time in a format suitable for sort (the UNIX command). Then pipe
it to sort. Finally, I merge all relevant files together with ‘sort
-m’. This lead to the result in a few hours only.

Thank you all for your suggestions,
Guillaume.

22

Le 17 août 06, à 20:55, [email protected] a écrit :

|> Not for 1.8.5. Maybe later.
|
|Do you withdraw [ruby-dev:29004]?

Ah, I am terribly sorry to forget. It’s checked in to 1.8.

  					matz.

Great. I’ll compile ruby from CVS and give it a try.

In my particular case, it didn’t seem to make much of a difference.

Thanks,
Guillaume.

23

On Fri, Aug 18, 2006 at 09:50:08AM +0900, [email protected]
wrote:

No. Any hint on what value I can raised it to? Any upper limit that I
should not cross?

You have so much RAM you could probably afford to set it to a few
hundred
MBs… If malloc_limit grows quickly enough it won’t really make a
difference, though. It’d be nice to know if it actually does in your
case,
for future reference.