# ActiveRecord row-by-row processing

**URL:** <https://www.ruby-forum.com/t/activerecord-row-by-row-processing/54839>\
**Category:** Rails\
**Created:** [March 14, 2006, 7:28pm UTC](https://www.ruby-forum.com/t/activerecord-row-by-row-processing/54839 "2006-03-14T19:28:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![stevedogers](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/9dc877/32.png) [@stevedogers](https://www.ruby-forum.com/u/stevedogers)\
**Post date:** [March 14, 2006, 7:28pm UTC](https://www.ruby-forum.com/t/activerecord-row-by-row-processing/54839/1 "2006-03-14T19:28:14Z")

</div>

I want to use ActiveRecord outside of Rails to do processing on large  
tables. I want to be able to process the rows as they are retrieved  
rather than having to store the results in an intermediate array.

Any suggestions?

Looks like I could hack the adapter to provide an accessor to the  
underlying DBI handle. Anything cleaner than that?

---

<div class="post-metadata">

**Author:** ![stevedogers](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/9dc877/32.png) [@stevedogers](https://www.ruby-forum.com/u/stevedogers)\
**Post date:** [March 14, 2006, 9:28pm UTC](https://www.ruby-forum.com/t/activerecord-row-by-row-processing/54839/2 "2006-03-14T21:28:02Z")

</div>

Check out  
[Peak Obsession](http://wiki.rubyonrails.com/rails/pages/HowToUseActiveRecordOutsideRails),  
[Peak Obsession](http://wiki.rubyonrails.com/rails/pages/HowToPopulateYourDbFromScript)  
and  
[Peak Obsession](http://wiki.rubyonrails.com/rails/pages/HowToUseActiveRecordThroughStandAloneScript)  
for some suggestions that might help.

Regards

Dave M.

---

<div class="post-metadata">

**Author:** ![stevedogers](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/9dc877/32.png) [@stevedogers](https://www.ruby-forum.com/u/stevedogers)\
**Post date:** [March 14, 2006, 10:50pm UTC](https://www.ruby-forum.com/t/activerecord-row-by-row-processing/54839/3 "2006-03-14T22:50:22Z")

</div>

Once I needed to export a big table to CVS file so I did it with this  
soultion:

```
render :text => Proc.new { |response, output|
    # Connect to database and start generation of CSV.
    # First we send a first row, which is sorted field names, then

```

we write  
# a row for each group of same response\_id. Everything is done  
by streaming  
# into the output  
copt = ActiveRecord::Base.configurations[RAILS\_ENV]  
db = Mysql.real\_connect(copt[‘host’], copt[‘username’],  
copt[‘password’], copt[‘database’])  
mysql\_result = db.query(‘SELECT \* FROM table ORDER BY id’)

```
    while r = mysql_result.fetch_row
        # use r[1], r[2], r[3] as columns here
        # output.write(r[1] + "\n")
    end
    db.close
}

```

The above is used with MySQL, for other DB engine you need to use  
different adapter.

–  
Kamil

---

<div class="post-metadata">

**Author:** ![stevedogers](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/9dc877/32.png) [@stevedogers](https://www.ruby-forum.com/u/stevedogers)\
**Post date:** [March 15, 2006, 4:51am UTC](https://www.ruby-forum.com/t/activerecord-row-by-row-processing/54839/4 "2006-03-15T04:51:28Z")

</div>

If you want to do this in ActiveRecord, you wont have much luck. I  
worked on some code to provide functionality like you want but there are  
multiple problems.

class CoolBob \< ActiveRecord::Base  
end

CoolBob.each do |bob|  
// DO something with each record (as it is returned from the adapter)  
end

In this implementation, only 1 CoolBob object is created, and its  
@attributes are updated with each row fetch. The problem here is that  
Ruby being a GC’d language, it doesnt save you anything OTHER than  
object creation time since it doesnt create a new object with each row.  
It doesn’t save on any memory use really. Each @attributes (which is the  
row data as an array) will still use memory until collected. So not much  
different than creating all the objects up front.

Also, the Ruby MySQL adapter (as well as others) doesn’t support  
unbuffered queries which is where you would get your speed increase and  
judicial use of memory from anyways. So in short, what you want to do  
isn’t really possible using AR, but is possible using the method in the  
previous post, even though that still buffers the complete result set  
before being given access to it.

Bob S.  
[http://www.railtie.net/](http://www.railtie.net/)