Is there a limit to the practical size of a fixture file?

Rails
1

Is there a limit on how big the fixtures can be?

Suppose I want to use csv fixtures and load up a table with a few
hundred thousand rows to prepopulate my production DB.

I did an initial test, but I canceled it after 35 minutes (with no
indication of whether it was working or not, and nothing got changed
on the db).

Obviously, this won’t work for regular testing because it’ll make the
tests take too long, but is there a practical limit on how big
fixtures can be?

I’ll continue to play with this, but I was just wondering if anyone
had done this before with fixtures of this size.

–
- Adam

** Expert Technical Project and Business Management
**** System Performance Analysis and Architecture
****** [ http://www.adamfields.com ]

[ Adam Fields (weblog) - - entertaining hundreds of millions of eyeball atoms every day ] … Blog
[ Adam Fields Resume ]… Experience
[ Adam Fields | Flickr ] … Photos
[ http://www.aquicki.com/wiki ]…Wiki

2

On Wed, Aug 23, 2006 at 12:59:23PM -0400, Adam F. wrote:

tests take too long, but is there a practical limit on how big
fixtures can be?

I’ll continue to play with this, but I was just wondering if anyone
had done this before with fixtures of this size.

In case anyone else is interested, I’ve successfully imported a csv
fixture file with 120k records.

It took nearly 2.5 hours with indexes in place (which I realized
halfway through but didn’t want to cancel). I’ve not had a chance
to fully test it without indexes, but even without them, it took well
over an hour to parse the whole csv file into memory before it even
started writing to the db.

–
- Adam

** Expert Technical Project and Business Management
**** System Performance Analysis and Architecture
****** [ http://www.adamfields.com ]

[ Adam Fields (weblog) - - entertaining hundreds of millions of eyeball atoms every day ] … Blog
[ Adam Fields Resume ]… Experience
[ Adam Fields | Flickr ] … Photos
[ http://www.aquicki.com/wiki ]…Wiki