# Querying a large data set

**URL:** <https://www.ruby-forum.com/t/querying-a-large-data-set/145962>\
**Category:** Rails\
**Created:** [August 15, 2008, 3:45pm UTC](https://www.ruby-forum.com/t/querying-a-large-data-set/145962 "2008-08-15T15:45:42Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![macfizz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/b9bd4f/32.png) [@macfizz](https://www.ruby-forum.com/u/macfizz)\
**Post date:** [August 15, 2008, 3:45pm UTC](https://www.ruby-forum.com/t/querying-a-large-data-set/145962/1 "2008-08-15T15:45:42Z")

</div>

Hi all,

I hope someone might have some sage words about the right approach  
here. I have a database application that works OK locally but is  
reporting an error on my host provider. The error comes up when I try  
to bring back a fairly large dataset (22500 rows, 20 columns). In the  
production log I see this error:

“ActionView::TemplateError (failed to allocate memory) on line #52”

From some initial googling it looks like I am running out of resource  
on my server. Is there a setting that would allocate more resource? I  
am running on a Dreamhost server using mod\_rails (or should it be  
Passenger?).

Actually, I suppose what I should be doing is rewriting my query so  
that MySQL does the heavy lifting and my application just gets back a  
smaller resultset. Given that, is there a recommended approach to  
writing my app so that it is only having to handle small tranches of  
data at any time?

Any guidance much appreciated.

## macfizz.

---

<div class="post-metadata">

**Author:** ![macfizz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/b9bd4f/32.png) [@macfizz](https://www.ruby-forum.com/u/macfizz)\
**Post date:** [August 15, 2008, 6:47pm UTC](https://www.ruby-forum.com/t/querying-a-large-data-set/145962/2 "2008-08-15T18:47:33Z")

</div>

Can you break your dataset up into pages? If so, look into  
will\_paginate plugin. Easy to use and easy to set up.

On Aug 15, 10:50 am, Frederick C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)

---

<div class="post-metadata">

**Author:** ![macfizz](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/b9bd4f/32.png) [@macfizz](https://www.ruby-forum.com/u/macfizz)\
**Post date:** [August 15, 2008, 4:51pm UTC](https://www.ruby-forum.com/t/querying-a-large-data-set/145962/3 "2008-08-15T16:51:29Z")

</div>

On 15 Aug 2008, at 14:44, macfizz wrote:

> From some initial googling it looks like I am running out of resource  
> on my server. Is there a setting that would allocate more resource? I  
> am running on a Dreamhost server using mod\_rails (or should it be  
> Passenger?).

Rails doesn’t throttle your resources. If there is a limit it will be  
imposed by your hosting provider (or just the hard limit on what is  
available in that environment)

> Actually, I suppose what I should be doing is rewriting my query so  
> that MySQL does the heavy lifting and my application just gets back a  
> smaller resultset. Given that, is there a recommended approach to  
> writing my app so that it is only having to handle small tranches of  
> data at any time?

I think it will vary according to what you are trying to do.

Fred