# Dealing with large chunks of data **URL:** \ **Category:** Rails\ **Created:** [November 15, 2006, 1:41am UTC](https://www.ruby-forum.com/t/dealing-with-large-chunks-of-data/75998 "2006-11-15T01:41:33Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![al-evans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/54ee81/32.png) [@al-evans](https://www.ruby-forum.com/u/al-evans)\ **Post date:** [November 15, 2006, 1:41am UTC](https://www.ruby-forum.com/t/dealing-with-large-chunks-of-data/75998/1 "2006-11-15T01:41:33Z")
In a site I’m working on, I have to generate statistics on demand. This requires going through substantially all three tables of the database, without about 1500 records in table A owning about 12,000 records in tables B and C. This is after the site’s been up about three weeks, so I know it will get a lot worse. There’s no issue with memory (dedicated server), and the database operation is fast enough (0.2 second or so), but the number crunching is killing me, presently taking about 16 seconds to iterate through all the data and produce the hash that eventually gets displayed. I’ve gotten to the point where I’m thinking about rewriting the routine in Perl or even C. I’d much prefer to avoid that. I would greatly appreciate suggestions from anybody who’s confronted a similar situation. Thanks, –Al Evans ---
**Author:** ![al-evans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/54ee81/32.png) [@al-evans](https://www.ruby-forum.com/u/al-evans)\ **Post date:** [November 15, 2006, 2:50am UTC](https://www.ruby-forum.com/t/dealing-with-large-chunks-of-data/75998/2 "2006-11-15T02:50:40Z")
On Nov 14, 2006, at 7:41 PM, Al Evans wrote: > crunching is > > Thanks, > > –Al Evans I’d suspect that the “data” you’re iterating through is in the form of ActiveRecord models, yes? If so, you might get a big win if you can avoid that overhead completely by dropping into SQL and using something like select\_all [1] (to get hashes back rather than model objects). Perhaps a bit of code would let us help you better. -Rob [1] [Parked at Loopia](http://rdoc.caboo.se/doc/classes/ActiveRecord/ConnectionAdapters/) DatabaseStatements.html#M004819 Rob B. [http://agileconsultingllc.com](http://agileconsultingllc.com) removed\_email\_address@domain.invalid ---
**Author:** ![al-evans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/54ee81/32.png) [@al-evans](https://www.ruby-forum.com/u/al-evans)\ **Post date:** [November 15, 2006, 3:06am UTC](https://www.ruby-forum.com/t/dealing-with-large-chunks-of-data/75998/3 "2006-11-15T03:06:41Z")
Al Evans wrote: > data and produce the hash that eventually gets displayed. > > I’ve gotten to the point where I’m thinking about rewriting the routine > in Perl or even C. I’d much prefer to avoid that. > > I would greatly appreciate suggestions from anybody who’s confronted a > similar situation. Can you store precomputed stats in the database, updating them as each piece of data is added? Or can you automatically calculate and save the stats once a day, and update these on demand using only the data added since? – We develop, watch us RoR, in numbers too big to ignore. ---
**Author:** ![al-evans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/54ee81/32.png) [@al-evans](https://www.ruby-forum.com/u/al-evans)\ **Post date:** [November 17, 2006, 11:25pm UTC](https://www.ruby-forum.com/t/dealing-with-large-chunks-of-data/75998/4 "2006-11-17T23:25:50Z")
Calle D. wrote: > On 11/15/06, Al Evans [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > I’ve gotten to the point where I’m thinking about rewriting the routine > > in Perl or even C. I’d much prefer to avoid that. > > Before you do that, try doing as much of the calculating as possible > in SQL. I had a similar problem to yours recently, and brought the > statistics generation time down from 45 seconds to 0.6 by letting the > database do the computation. Thanks! This is what I did. The code is uglier now, but it works almost 10 times as fast. –Al Evans ---
**Author:** ![al-evans](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/54ee81/32.png) [@al-evans](https://www.ruby-forum.com/u/al-evans)\ **Post date:** [November 15, 2006, 7:41am UTC](https://www.ruby-forum.com/t/dealing-with-large-chunks-of-data/75998/5 "2006-11-15T07:41:02Z")
On 11/15/06, Al Evans [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I’ve gotten to the point where I’m thinking about rewriting the routine > in Perl or even C. I’d much prefer to avoid that. ## Before you do that, try doing as much of the calculating as possible in SQL. I had a similar problem to yours recently, and brought the statistics generation time down from 45 seconds to 0.6 by letting the database do the computation. Calle D. -\*- removed\_email\_address@domain.invalid