On Friday 10 July 2009, gsterndale wrote:
I agree 100% that using FasterCSV and ActiveRecord to pull the data
is much more portable (and elegant). In fact, that’s how I’m doing it
now. However, this is a huge dataset that is causing server timeouts
and hogs memory. I’m investigating csv generation in Postgres as it
takes a fraction of the time and resources because each object isn’t
If your dataset is truly is huge an approach that relies on first
extracting the entire dataset and then serving it is not going to work
reliably. If it doesn’t fail outright due to a timeout, it is still
going to strain your memory considerably.
As far as I can see, there are two ways around this.
If the export doesn’t have to be current to this very instance, set up a
background job that exports it regularly to a file and leave the serving
to the web server.
If that doesn’t meet your needs, consider streaming the data. Have a
look at the send_data (controller) method. It may even be sensible to
handle this in Metal, i.e. a Rack module. I haven’t tried any of this
myself, the following link might be helpful