# Find(:all) - lazy/eager

**URL:** <https://www.ruby-forum.com/t/find-all-lazy-eager/105356>\
**Category:** Rails\
**Created:** [May 29, 2007, 8:48am UTC](https://www.ruby-forum.com/t/find-all-lazy-eager/105356 "2007-05-29T08:48:12Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![railskater](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/3ab097/32.png) [@railskater](https://www.ruby-forum.com/u/railskater)\
**Post date:** [May 29, 2007, 8:48am UTC](https://www.ruby-forum.com/t/find-all-lazy-eager/105356/1 "2007-05-29T08:48:12Z")

</div>

Hello experts,

Given some simple model and controller and simple views, what happens if  
I have an Employee table with 100,000+ rows? If my controller calls  
.find(:all), then when do all the objects get loaded? I know there are  
issues with lazy and eager loading.

The reason I ask is because if I load the Employees into a collection  
inside the script/console, then the console echos the collection to the  
screen and you can see all the objects and all the “has\_many” objects.  
At this point, the system loaded everything.

Now, what happens in the controller if my views do not use the  
collection variable?

Does the application detect that I never referenced the collection  
(other than when I specified that the .find(:all) method should be  
used)? Does the application never load the collection?

---

<div class="post-metadata">

**Author:** ![railskater](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/3ab097/32.png) [@railskater](https://www.ruby-forum.com/u/railskater)\
**Post date:** [May 29, 2007, 10:03am UTC](https://www.ruby-forum.com/t/find-all-lazy-eager/105356/2 "2007-05-29T10:03:31Z")

</div>

Philip D. wrote:

> Hello experts,
> 
> Given some simple model and controller and simple views, what happens if  
> I have an Employee table with 100,000+ rows? If my controller calls  
> .find(:all), then when do all the objects get loaded? I know there are  
> issues with lazy and eager loading.
> 
> The reason I ask is because if I load the Employees into a collection  
> inside the script/console, then the console echos the collection to the  
> screen and you can see all the objects and all the “has\_many” objects.  
> At this point, the system loaded everything.
> 
> Now, what happens in the controller if my views do not use the  
> collection variable?
> 
> Does the application detect that I never referenced the collection  
> (other than when I specified that the .find(:all) method should be  
> used)? Does the application never load the collection?

If you assign all of your employees to a variable they will be stored in  
RAM. Until that variable gets reassigned or the page finishes loading it  
will take up your RAM, regardless if you use it or not. If you just call  
find(:all) without assigning it to a variable then it will take up RAM  
as it executes, once it finishes executing it will free itself from RAM.

My advice is to only load things you wish to use, obviously. Also don’t  
load 100,000 rows into a variable because it will kill your ram.

---

<div class="post-metadata">

**Author:** ![railskater](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/3ab097/32.png) [@railskater](https://www.ruby-forum.com/u/railskater)\
**Post date:** [May 29, 2007, 10:23pm UTC](https://www.ruby-forum.com/t/find-all-lazy-eager/105356/3 "2007-05-29T22:23:56Z")

</div>

Ben J. wrote:

> If you assign all of your employees to a variable they will be stored in  
> RAM. Until that variable gets reassigned or the page finishes loading it  
> will take up your RAM, regardless if you use it or not. If you just call  
> find(:all) without assigning it to a variable then it will take up RAM  
> as it executes, once it finishes executing it will free itself from RAM.
> 
> My advice is to only load things you wish to use, obviously. Also don’t  
> load 100,000 rows into a variable because it will kill your ram.

Thanks Ben for the input. This is exactly what is making me question  
the “loading” of rails models. I feel like I have been tricked by the  
Rails community into thinking that the filtering/loading/finding methods  
in Rails are object oriented. (Of course you need to know a little  
something about relational databases.) However, If you want to load a  
subset of objects based on other objects owned by the objects, then you  
can not do it without using SQL or join models or polymorphic code. The  
join models are cool and nice but they are not exactly feeling like  
first rate objects.

I know this is a cumbersome argument but please understand I would like  
to know the trade-offs involved with the different approaches for  
loading when using:

1. use the simple single model dynamic finders
2. use some sql
3. use join models

I was trying to understand the first approach and how to load a subset  
of parent models based on some value in it’s child model. I can’t seem  
to figure out how to do this without resorting to steps 2 or 3.

To put it another way, no matter how clever I think I can be, I still  
end up doing some post collection load processing. It feels like rails  
is tricking us into thinking that it does not always load every row.  
However, when addressing non-simple models, I feel like I am always  
forced to load all rows and then do post-processing.

Interestingly, there seems to be a relationship between the  
filtering/loading styles and the combination of polymorphic and join  
model features. For websites of any serious use, there will be lots of  
parent models which own child models and there will be times when I want  
a subset of parent models based on the child model values. I do not  
think there is a way to address these parent-child type filters without  
using sql or join models or polymorphic type objects.

Thanks,

---

<div class="post-metadata">

**Author:** ![railskater](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/3ab097/32.png) [@railskater](https://www.ruby-forum.com/u/railskater)\
**Post date:** [May 29, 2007, 11:47pm UTC](https://www.ruby-forum.com/t/find-all-lazy-eager/105356/4 "2007-05-29T23:47:18Z")

</div>

Eric A. wrote:  
ActiveRecord is a thin wrapper around the relational model. There are  
other ORMs that operate at a higher level although I find that they  
often don’t give you the low-level access that you occasionally need. I  
think Rails reaches a nice compromise of being a thin wrapper with some  
high-level features (such as dynamic finders and :include), but also  
providing complete low-level access.

* * *

Wow, Eric. You nailed everything in place. That last paragraph you  
wrote is the gem of information I was looking for. I appreciate your  
very generous response. I have read the Agile Rails book. Sadly the  
book fails to re-iterate the point you made. Also, it is easy to forget  
(as I did) that everything eventually boils down to find\_by\_sql.

Thanks for clearing up my head.

Philip

---

<div class="post-metadata">

**Author:** ![railskater](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/3ab097/32.png) [@railskater](https://www.ruby-forum.com/u/railskater)\
**Post date:** [May 29, 2007, 11:06pm UTC](https://www.ruby-forum.com/t/find-all-lazy-eager/105356/5 "2007-05-29T23:06:46Z")

</div>

Philip D. wrote:

> The reason I ask is because if I load the Employees into a collection  
> inside the script/console, then the console echos the collection to the  
> screen and you can see all the objects and all the “has\_many” objects.  
> At this point, the system loaded everything.

I’m not sure I understand what you are doing here. What is the  
“has\_many” objects you are referring to? You have an employee model:

class Employee \< ActiveRecord::Base  
end

If also said that you are trying to search based on a related object (in  
your other email). So I assume you have something like:

class Employee \< ActiveRecord::Base  
has\_many :tasks, :dependent =\> :destroy  
end  
class Task \< ActiveRecord::Base  
belongs\_to :employee  
end

Now you want to load all employees that have high priority tasks. So you  
do:

@lazy\_employees = Employee.find :all, :include =\> ‘tasks’,  
:conditions =\> [‘tasks.priority = ?’, ‘high’]

Now your complaint is that the tasks that meet the conditions are loaded  
with the above code which you feel is a bit wasteful? Am I understanding  
your problem correctly?

If that is the case there are a few things you can do:

1. Make sure it really matters. Does that extra loading of data actually  
cause a performance problems? If not don’t worry about it. The above  
code is simple to understand. If it is an actual performance problem  
then proceed.

2. You could switch from using :include to :joins. This allows you to  
join without loading the actual related objects. So:

@lazy\_employees = Employee.find :all,  
:joins =\> %q{INNER JOIN tasks ON  
tasks.employee\_id = employees.id AND  
tasks.priority = ‘high’}

You are removing the convienance of what :include provides (automatic  
joining and automatic object loading of the related object) but you gain  
more control. Your join is now a inner join instead of an outer join  
which is much more efficient. Also you are not loading those related  
objects.

1. If you want to keep the loading of related objects (for example if  
you use those objects in your code and want to avoid N+1 query problem)  
you can also improve the first attempt using the :select option so only  
the fields you actually use are loaded. For example:

@lazy\_employees = Employee.find :all, :include =\> ‘tasks’,  
:select =\> ‘employees.name, task.name, tasks.due\_on’,  
:conditions =\> [‘tasks.priority = ?’, ‘high’]

This limits you to only being able to print the employee name, task name  
and due date but it saves on the amount of data that has to be loaded.

1. If you don’t care about related objects and only want certain fields  
you can combine step 2 and 3 to produce:

@lazy\_employees = Employee.find :all,  
:select =\> ‘employees.name’,  
:joins =\> %q{INNER JOIN tasks ON  
tasks.employee\_id = employees.id AND  
tasks.priority = ‘high’}

Now you are only loading the data you need, using an INNER JOIN to  
reduce the load in SQL and not loading the related objects that you  
don’t plan to print anything out about. But obviously now it is much  
uglier. If you really need the performance try to push the code down to  
the model level so your controllers and view remain easy to work with.  
See [Buckblog: Skinny Controller, Fat Model](http://weblog.jamisbuck.org/2006/10/18/skinny-controller-fat-model)  
for more info on this process.

In the end everything resolves to find\_by\_sql. Depending on your needs  
you might be able to operate at a very high level like the first  
attempt. Or you may need to operate almost at the SQL-level like the  
last attempt. You always still have the option of operating completely  
at sql if you need absolute control. Determining at what level to  
operate at is determined by a balance of performance and code  
readability. I have found in practice I rarely use actual SQL code. Most  
my my queries look like:

Employee.find\_by\_last\_name(‘Anderson’)  
Employee.find :all, :include =\> :departments,  
:conditions =\> [‘departments.name = ?’, ‘IT’]

But every app will have a few queries that get closer to SQL. I have  
developed around 10 Rails applications thus far and never had a need for  
find\_by\_sql although I don’t rule it out.

ActiveRecord is a thin wrapper around the relational model. There are  
other ORMs that operate at a higher level although I find that they  
often don’t give you the low-level access that you occasionally need. I  
think Rails reaches a nice compromise of being a thin wrapper with some  
high-level features (such as dynamic finders and :include), but also  
providing complete low-level access.

Eric

---

<div class="post-metadata">

**Author:** ![railskater](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/3ab097/32.png) [@railskater](https://www.ruby-forum.com/u/railskater)\
**Post date:** [May 30, 2007, 4:38pm UTC](https://www.ruby-forum.com/t/find-all-lazy-eager/105356/6 "2007-05-30T16:38:16Z")

</div>

Philip D. wrote:

> Wow, Eric. You nailed everything in place. That last paragraph you  
> wrote is the gem of information I was looking for. I appreciate your  
> very generous response. I have read the Agile Rails book. Sadly the  
> book fails to re-iterate the point you made. Also, it is easy to forget  
> (as I did) that everything eventually boils down to find\_by\_sql.

In addition to the article on skinny controller, fat model Jamis also  
has some excellent “under the hood” articles.

[http://weblog.jamisbuck.org/under-the-hood](http://weblog.jamisbuck.org/under-the-hood)

The articles about ActiveRecord::Base.find you might find to be useful.

Eric