# Querying persistent ruby objects in memory **URL:** \ **Category:** Ruby\ **Created:** [May 26, 2007, 11:05pm UTC](https://www.ruby-forum.com/t/querying-persistent-ruby-objects-in-memory/105188 "2007-05-26T23:05:48Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![braver](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/a88e57/32.png) [@braver](https://www.ruby-forum.com/u/braver)\ **Post date:** [May 26, 2007, 11:05pm UTC](https://www.ruby-forum.com/t/querying-persistent-ruby-objects-in-memory/105188/1 "2007-05-26T23:05:48Z")
I have a data-mining task which loads data as a big XML tree (10+ MB) and then reorganizes it. Even loading it with Hpricot takes 10-20 seconds. I don’t want to do it for every manilupation I want to try, especially for sequences of transformations. Thus I wonder what’s a good way to keep the huge object in memory between the runs of querying scripts. Can Rails be used for that? I’d rather avoid writing a client-server platform, or using it per se, unless there’s already an existing one. A vague intuition is, it should be something like threads – one thread parses XML and keeps it in memory, another starts up later, somehow joins the memory space of the first one, queries/transforms it, and ends. Then other queries/ transformations can all be run. Is there anything like it? Cheers, Alexy ---
**Author:** ![braver](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/a88e57/32.png) [@braver](https://www.ruby-forum.com/u/braver)\ **Post date:** [May 26, 2007, 11:27pm UTC](https://www.ruby-forum.com/t/querying-persistent-ruby-objects-in-memory/105188/2 "2007-05-26T23:27:51Z")
On 26.05.2007 23:00, braver wrote: > in memory, another starts up later, somehow joins the memory space of > the first one, queries/transforms it, and ends. Then other queries/ > transformations can all be run. Is there anything like it? I’d consider using Marshal. Kind regards robert ---
**Author:** ![braver](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/a88e57/32.png) [@braver](https://www.ruby-forum.com/u/braver)\ **Post date:** [May 27, 2007, 12:36am UTC](https://www.ruby-forum.com/t/querying-persistent-ruby-objects-in-memory/105188/3 "2007-05-27T00:36:59Z")
On May 26, 2:24 pm, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I’d consider using Marshal. That’s just plain serialization, isn’t it? I’ve seen that and Madelaine; but my wish is to keep the objects in memory without the need to dump/reload it, however fast. (That would be a last resort.) The question is, can we keep an object in memory in one thread, and explore/change it from another? In the worst case, we can probably quickly dump an object into a memory region and reload it back via Marshal – I guess a crude solution is forming here, using shared memory or RAM disk – have to see what’s there for macs… But still I wonder what folks think in terms of all kinds of RAM persistence in ruby solutions. Cheers, Alexy ---
**Author:** ![braver](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/a88e57/32.png) [@braver](https://www.ruby-forum.com/u/braver)\ **Post date:** [May 27, 2007, 1:14am UTC](https://www.ruby-forum.com/t/querying-persistent-ruby-objects-in-memory/105188/4 "2007-05-27T01:14:01Z")
On 5/26/07, braver [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > quickly dump an object into a memory region and reload it back via > Marshal – I guess a crude solution is forming here, using shared > memory or RAM disk – have to see what’s there for macs… But still > I wonder what folks think in terms of all kinds of RAM persistence in > ruby solutions. Aren’t you overengineering a little? You want to amortize a ten-second startup cost over a (presumably) large number of operations against some dataset. But you keep talking about threads. That tells me that your process will run for a long time and will know all the operations it has to execute upfront. In that case, forget about threads and just serialize your operations. Your life will be much simpler. But on the other hand, you talk about shared memory and about not wanting to write a client/server application. That suggests that you’re thinking of keeping this dataset around and having other PROCESSES sent requests to it at arbitrary times. In that case, don’t use threads either, or shared-memory for that matter. Life is too short to debug all that stuff. Write yourself a little client-server application and be done with it. If you don’t want to deal with the network programming, use EventMachine. ---
**Author:** ![braver](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/a88e57/32.png) [@braver](https://www.ruby-forum.com/u/braver)\ **Post date:** [May 27, 2007, 10:40am UTC](https://www.ruby-forum.com/t/querying-persistent-ruby-objects-in-memory/105188/5 "2007-05-27T10:40:40Z")
On 27.05.2007 00:33, braver wrote: > On May 26, 2:24 pm, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > I’d consider using Marshal. > > That’s just plain serialization, isn’t it? I’ve seen that and > Madelaine; but my wish is to keep the objects in memory without the > need to dump/reload it, however fast. (That would be a last resort.) I find that odd. Keeping something in memory is usually a _solution_ for some kind of _business requirement_ (e.g. to make things fast). Why would you want to keep something in mem if it can be persisted on disk really fast? I don’t know the volume of what you need to handle but did you actually try out how fast it is? > The question is, can we keep an object in memory in one thread, and > explore/change it from another? Yes, of course. Easily sharing memory is one (if not _the_) major aspect of multithreaded applications. But reading your other posting I am not sure whether you have the proper idea of MT programming. If you only want to do one set of manipulations at a time you do not need multiple threads because there is no concurrency involved. > In the worst case, we can probably > quickly dump an object into a memory region and reload it back via > Marshal – I guess a crude solution is forming here, using shared > memory or RAM disk – have to see what’s there for macs… But still > I wonder what folks think in terms of all kinds of RAM persistence in > ruby solutions. As James suggested using DRb is one option. Then you can decide whether to manipulate the object graph in the server process or send it off to the client (and probably send it back after doing your changes). It’s probably the best solution in your case because you can start arbitrary client processes and manipulate state in the server. But you should make sure that access is proper synchronized to cope with multiple clients that connect concurrently. Kind regards robert ---
**Author:** ![braver](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/a88e57/32.png) [@braver](https://www.ruby-forum.com/u/braver)\ **Post date:** [May 27, 2007, 8:12am UTC](https://www.ruby-forum.com/t/querying-persistent-ruby-objects-in-memory/105188/6 "2007-05-27T08:12:28Z")
Someone else was talking about this kind of problem the other day in #ruby-lang. Another posted an elegant solution to the problem (which incidentally was refused as it was another process), however: #!ruby raise ‘You need to install win32/process’ unless require ‘win32/process’ if RUBY\_PLATFORM.include? ‘mswin32’ # parent forks off and dies, leaving child as daemon exit 0 if !fork.nil? # daemon code starts here require ‘drb/drb’ require ‘thread’ require ‘server’ $SAFE = 1 # disable eval() and friends DRb.start\_service(“druby://:2020”, Server.new) puts DRb.uri DRb.thread.join