Strategy when dealing with LARGE data sets

Rails
1

I am a new convert to RoR and am loving every minute of it. Absolute
highest praise goes to everyone who has contributed.

An issue has come up when dealing with tables with large numbers of
rows and columns. Specifically, i have an issue in an
ActiveRecord::Migration, where after adding a new column, I would like
to iterate each row, read an attribute, process that attribute, then
update the new attribute/column with the result.

however, doing something like this:
Stuff.find_all { |r| r[‘new_attr’] = process r[‘old_attr’] }

would kill the server returning so many records. I am aware of the
:select parameter to find() but it would still kill the server with ALL
records being returned in the array.

I see there is a :limit and :offset parameter to find(), which would
work. I figured i would ask if there are there any other workable
solutions to dealing with large data sets, before creating my own.

Thank you!

Jae

2

Stuff.find_all { |r| r[‘new_attr’] = process r[‘old_attr’] }

would kill the server returning so many records. I am aware of the
:select parameter to find() but it would still kill the server with ALL
records being returned in the array.

I see there is a :limit and :offset parameter to find(), which would
work. I figured i would ask if there are there any other workable
solutions to dealing with large data sets, before creating my own.

Just saw this on the rss feed yesterday which might work for you…

Plugins - paginating_find - Agile Web D.
http://www.agilewebdevelopment.com/plugins/paginating_find

Got 15,842 records that you.d like to export to a file? Using the
standard
the Rails ActiveRecord::Base#find method will load all 15,842 into
memory
all at once and return them all in an array. If your app is running on a
shared host, or if you.re keeping your app on a memory budget, this is a
big problem for you. So you could load each record one by one, but
that.ll
kill your db server. Wouldn.t it be sweet if #find could return an
enumerable that would load your records in batches of say 1,500 records?
Well with my new nifty-jifty paginating_find plugin, it can.

-philip

3

Thanks Philip… i’m in nirvana. amazing. now i have to find out what
plugins are.

it’s amazing. a single line install, used it, tested it, it friggin
works. amazing!

Jae

4

You may find the recently released paginating_find plugin helpful:
http://agilewebdevelopment.com/plugins/paginating_find

–
Building an e-commerce site with Rails?
http://www.agilewebdevelopment.com/rails-ecommerce

5

Hi Jae. I’m glad to know the plugin is working well for you. Note that
I recently updated the code so that it supports paging in a manual
fashion, meaning that you can enumerate only over the current page and
then step to the next page using the #next_page! method. This means
that the plugin can be used really easily for both loading your pages
and for displaying your pages. If you find yourself using the plugin in
this fashion, I’d be interested to hear if it works well for you.

-Alex

JChang wrote:

Thanks Philip… i’m in nirvana. amazing. now i have to find out what
plugins are.

it’s amazing. a single line install, used it, tested it, it friggin
works. amazing!

Jae