Forum: Ruby on Rails RE: What to do with HUGE instance variables in Rails?

Announcement (2017-05-07): is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see and for other Rails- und Ruby-related community platforms.
Ross D. (Guest)
on 2006-05-11 11:44
(Received via mailing list)
> >> Problem is that the DB has about 260,000 lines which
> >>
> >>
> >
> > My first answer is: don't do it.
> > Second: Never ever.
> > Third: are you serious?

Well lets qualify Stefan's comments a bit.  A task processing 260K rows
in a database is not a job to be done during a web request.  If you were
doing a mail merge for example then I'd put that off as a scheduled
task.  Which ever way you go this is going to take some considerable
processing.  You want a web page to respond in a couple of seconds not a
few minutes.

In a web page reading 260K rows into memory at once is bad (long time to
process and consumes LOTs of ram). Reading 260K rows one at a time (find
:first/ find :next) is also bad as it takes a long time to process.

So my first answer is: don't do it in Rails, but you could do it in Ruby
still using ActiveRecord scheduled with cron.

Second: Never do something like this in any Web framework. Even if you
process 1000 rows/sec that's 4 1/2 minutes, who'd wait that long for a
web page?

Third: do it as an offline or overnight batch job.  Perhaps using a web
page to trigger the start of processing and an email notifying it's
completion so you can download from the server.

This topic is locked and can not be replied to.