# Strategy when dealing with LARGE data sets

**URL:** <https://www.ruby-forum.com/t/strategy-when-dealing-with-large-data-sets/66849>\
**Category:** Rails\
**Created:** [August 29, 2006, 8:12pm UTC](https://www.ruby-forum.com/t/strategy-when-dealing-with-large-data-sets/66849 "2006-08-29T20:12:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![JChang](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/b9bd4f/32.png) [@JChang](https://www.ruby-forum.com/u/JChang)\
**Post date:** [August 29, 2006, 8:12pm UTC](https://www.ruby-forum.com/t/strategy-when-dealing-with-large-data-sets/66849/1 "2006-08-29T20:12:09Z")

</div>

I am a new convert to RoR and am loving every minute of it. Absolute  
highest praise goes to everyone who has contributed.

An issue has come up when dealing with tables with large numbers of  
rows and columns. Specifically, i have an issue in an  
ActiveRecord::Migration, where after adding a new column, I would like  
to iterate each row, read an attribute, process that attribute, then  
update the new attribute/column with the result.

however, doing something like this:  
Stuff.find\_all { |r| r[‘new\_attr’] = process r[‘old\_attr’] }

would kill the server returning so many records. I am aware of the  
:select parameter to find() but it would still kill the server with ALL  
records being returned in the array.

I see there is a :limit and :offset parameter to find(), which would  
work. I figured i would ask if there are there any other workable  
solutions to dealing with large data sets, before creating my own.

Thank you!

Jae

---

<div class="post-metadata">

**Author:** ![JChang](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/b9bd4f/32.png) [@JChang](https://www.ruby-forum.com/u/JChang)\
**Post date:** [August 29, 2006, 8:36pm UTC](https://www.ruby-forum.com/t/strategy-when-dealing-with-large-data-sets/66849/2 "2006-08-29T20:36:16Z")

</div>

> Stuff.find\_all { |r| r[‘new\_attr’] = process r[‘old\_attr’] }
> 
> would kill the server returning so many records. I am aware of the  
> :select parameter to find() but it would still kill the server with ALL  
> records being returned in the array.
> 
> I see there is a :limit and :offset parameter to find(), which would  
> work. I figured i would ask if there are there any other workable  
> solutions to dealing with large data sets, before creating my own.

Just saw this on the rss feed yesterday which might work for you…

Plugins - paginating\_find - Agile Web D.  
[http://www.agilewebdevelopment.com/plugins/paginating\_find](http://www.agilewebdevelopment.com/plugins/paginating_find)

Got 15,842 records that you.d like to export to a file? Using the  
standard  
the Rails ActiveRecord::Base#find method will load all 15,842 into  
memory  
all at once and return them all in an array. If your app is running on a  
shared host, or if you.re keeping your app on a memory budget, this is a  
big problem for you. So you could load each record one by one, but  
that.ll  
kill your db server. Wouldn.t it be sweet if #find could return an  
enumerable that would load your records in batches of say 1,500 records?  
Well with my new nifty-jifty paginating\_find plugin, it can.

-philip

---

<div class="post-metadata">

**Author:** ![JChang](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/b9bd4f/32.png) [@JChang](https://www.ruby-forum.com/u/JChang)\
**Post date:** [August 29, 2006, 9:18pm UTC](https://www.ruby-forum.com/t/strategy-when-dealing-with-large-data-sets/66849/3 "2006-08-29T21:18:26Z")

</div>

Thanks Philip… i’m in nirvana. amazing. now i have to find out what  
plugins are.

it’s amazing. a single line install, used it, tested it, it friggin  
works. amazing!

Jae

---

<div class="post-metadata">

**Author:** ![JChang](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/b9bd4f/32.png) [@JChang](https://www.ruby-forum.com/u/JChang)\
**Post date:** [August 30, 2006, 2:47pm UTC](https://www.ruby-forum.com/t/strategy-when-dealing-with-large-data-sets/66849/4 "2006-08-30T14:47:07Z")

</div>

You may find the recently released paginating\_find plugin helpful:  
[http://agilewebdevelopment.com/plugins/paginating\_find](http://agilewebdevelopment.com/plugins/paginating_find)

–  
Building an e-commerce site with Rails?  
[http://www.agilewebdevelopment.com/rails-ecommerce](http://www.agilewebdevelopment.com/rails-ecommerce)

---

<div class="post-metadata">

**Author:** ![JChang](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/b9bd4f/32.png) [@JChang](https://www.ruby-forum.com/u/JChang)\
**Post date:** [September 3, 2006, 11:02pm UTC](https://www.ruby-forum.com/t/strategy-when-dealing-with-large-data-sets/66849/5 "2006-09-03T23:02:14Z")

</div>

Hi Jae. I’m glad to know the plugin is working well for you. Note that  
I recently updated the code so that it supports paging in a manual  
fashion, meaning that you can enumerate only over the current page and  
then step to the next page using the #next\_page! method. This means  
that the plugin can be used really easily for both loading your pages  
and for displaying your pages. If you find yourself using the plugin in  
this fashion, I’d be interested to hear if it works well for you.

-Alex

> **[Welcome cardboardrocket.com - BlueHost.com](http://cardboardrocket.com)**
>
> Bluehost - Top rated web hosting provider - Free 1 click installs For blogs, shopping carts, and more. Get a free domain name, real NON-outsourced 24/7 support, and superior speed. web hosting provider php hosting cheap web hosting, Web hosting,...

JChang wrote:

> Thanks Philip… i’m in nirvana. amazing. now i have to find out what  
> plugins are.
> 
> it’s amazing. a single line install, used it, tested it, it friggin  
> works. amazing!
> 
> Jae