RE: What to do with HUGE instance variables in Rails?


#1

Problem is that the DB has about 260,000 lines which

My first answer is: don’t do it.
Second: Never ever.
Third: are you serious?

Well lets qualify Stefan’s comments a bit. A task processing 260K rows
in a database is not a job to be done during a web request. If you were
doing a mail merge for example then I’d put that off as a scheduled
task. Which ever way you go this is going to take some considerable
processing. You want a web page to respond in a couple of seconds not a
few minutes.

In a web page reading 260K rows into memory at once is bad (long time to
process and consumes LOTs of ram). Reading 260K rows one at a time (find
:first/ find :next) is also bad as it takes a long time to process.

So my first answer is: don’t do it in Rails, but you could do it in Ruby
still using ActiveRecord scheduled with cron.

Second: Never do something like this in any Web framework. Even if you
process 1000 rows/sec that’s 4 1/2 minutes, who’d wait that long for a
web page?

Third: do it as an offline or overnight batch job. Perhaps using a web
page to trigger the start of processing and an email notifying it’s
completion so you can download from the server.

Ross