Forum: Ruby on Rails ActiveRecord row-by-row processing

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
80d9f4c5aceadbdf9c8f1e226797c3f9?d=identicon&s=25 Steve Downey (Guest)
on 2006-03-14 19:28
(Received via mailing list)
I want to use ActiveRecord outside of Rails to do processing on large
tables.  I want to be able to process the rows as they are retrieved
rather than having to store the results in an intermediate array.

Any suggestions?

Looks like I could hack the adapter to provide an accessor to the
underlying DBI handle.  Anything cleaner than that?
F3dc06f587d1ff4c7366b102bfda9204?d=identicon&s=25 David Mitchell (Guest)
on 2006-03-14 21:28
(Received via mailing list)
2a73a242ac582b27e841b4bce6ba1373?d=identicon&s=25 Kamil Kukura (Guest)
on 2006-03-14 22:50
(Received via mailing list)
Once I needed to export a big table to CVS file so I did it with this
soultion:

    render :text => Proc.new { |response, output|
        # Connect to database and start generation of CSV.
        # First we send a first row, which is sorted field names, then
we write
        # a row for each group of same response_id. Everything is done
by streaming
        # into the output
        copt = ActiveRecord::Base.configurations[RAILS_ENV]
        db = Mysql.real_connect(copt['host'], copt['username'],
copt['password'], copt['database'])
        mysql_result = db.query('SELECT * FROM table ORDER BY id')

        while r = mysql_result.fetch_row
            # use r[1], r[2], r[3] as columns here
            # output.write(r[1] + "\n")
        end
        db.close
    }

The above is used with MySQL, for other DB engine you need to use
different adapter.

--
Kamil
5d15c6821f3c3054c04b85471824ba7c?d=identicon&s=25 Bob Silva (Guest)
on 2006-03-15 04:51
(Received via mailing list)
If you want to do this in ActiveRecord, you wont have much luck. I
worked on some code to provide functionality like you want but there are
multiple problems.

class CoolBob < ActiveRecord::Base
end

CoolBob.each do |bob|
  // DO something with each record (as it is returned from the adapter)
end

In this implementation, only 1 CoolBob object is created, and its
@attributes are updated with each row fetch. The problem here is that
Ruby being a GC'd language, it doesnt save you anything OTHER than
object creation time since it doesnt create a new object with each row.
It doesn't save on any memory use really. Each @attributes (which is the
row data as an array) will still use memory until collected. So not much
different than creating all the objects up front.

Also, the Ruby MySQL adapter (as well as others) doesn't support
unbuffered queries which is where you would get your speed increase and
judicial use of memory from anyways. So in short, what you want to do
isn't really possible using AR, but is possible using the method in the
previous post, even though that still buffers the complete result set
before being given access to it.

Bob Silva
http://www.railtie.net/
This topic is locked and can not be replied to.