ActiveRecord retrieval style? Lazy initialtization?

I’m using ActiveRecord for an Extract and Transform operation, so it is
outside of its native environment of Rails. I am running into a problem
with memory consumption when I retrieve large amounts of data. By large
amounts I am talking about roughly 12,700 records that each have a one
to many relationship with about 15 additional types of information. One
of those types happens to be a table with binaries in it.

Based on the way the system seems to be responding, its as if
ActiveRecord is creating an array and then using lazy initialization for
the records. Creating the ET app was very easy, but I’m running into a
brick wall here. I know that ActiveRecord was designed for web systems,
so being able to process 12,700 complex records is outside of what you
might consider normal use.

The question I have is whether I can access it in a more fly-weight
model. In short, once I’m done with a record I don’t need it any more.
Most database related stuff I’ve used operates this way. In short you
have one object (in Java it would be a ResultSet), and its contents are
mutated as you iterate over the whole recordset. Instead of trying to
create 12,700 objects and its ancillary information, merely creating one
object and changing its value as you iterate.

Am I going to have to go to a lower level API to do what I need? If so,
I would still like to use the ActiveRecord API for each individual
record… Using the low level API to get the list of IDs and then
ActiveRecord to retrieve just the record for that ID.

There’s got to be something I’m missing.


Berin L.
Owner

Work: 571-215-7708
Email: [email protected] mailto:[email protected]
IM: [email protected] (MSN)
*http://www.linkedin.com/in/bloritsch
*

    • 	*D-Haven* <http://d-haven.org>
      

20044 Great Falls Forest Dr.
http://maps.google.com/maps?q=20044+Great+Falls+Forest+Dr.%2CGreat+Falls%2CVA+22066%2CUSA&hl=en
Great Falls, VA 22066
USA

See who we know in common http://www.linkedin.com/e/wwk/5678360/ Want
a signature like this? http://www.linkedin.com/e/sig/5678360/

Hi Berin,

aren’t you able to use an approach similare to what the paginate method
does?

In a first step cound the records you are going to process then iterate
on
them in a paginated way by using ‘where id in (id, id, id, id)’

Cheers,
Jan

So is your problem memory consumption, processing time, excessive SQL
calls?

Cheers
Dr Nic

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs