I’m using ActiveRecord for an Extract and Transform operation, so it is
outside of its native environment of Rails. I am running into a problem
with memory consumption when I retrieve large amounts of data. By large
amounts I am talking about roughly 12,700 records that each have a one
to many relationship with about 15 additional types of information. One
of those types happens to be a table with binaries in it.
Based on the way the system seems to be responding, its as if
ActiveRecord is creating an array and then using lazy initialization for
the records. Creating the ET app was very easy, but I’m running into a
brick wall here. I know that ActiveRecord was designed for web systems,
so being able to process 12,700 complex records is outside of what you
might consider normal use.
The question I have is whether I can access it in a more fly-weight
model. In short, once I’m done with a record I don’t need it any more.
Most database related stuff I’ve used operates this way. In short you
have one object (in Java it would be a ResultSet), and its contents are
mutated as you iterate over the whole recordset. Instead of trying to
create 12,700 objects and its ancillary information, merely creating one
object and changing its value as you iterate.
Am I going to have to go to a lower level API to do what I need? If so,
I would still like to use the ActiveRecord API for each individual
record… Using the low level API to get the list of IDs and then
ActiveRecord to retrieve just the record for that ID.
There’s got to be something I’m missing.
20044 Great Falls Forest Dr.
Great Falls, VA 22066