Large file storing in postgres sucks?

Rails
1

Hi,

I’m not sure if this is a postgres issue or not, but I’m using
Postgres 8.1. I have (in my functional tests) the following code:

upload = fixture_file_upload('/files/podcast.mp3', 'audio/mpeg')
post :create, :product => valid_product, :media => { :image =>

upload }

And then in the controller:
blob = Blob.create :data => uploaded_file.read

When /files/podcast.mp3 is 32 megabytes and I run the test, memory
spikes hugely. My laptop quickly began swapping out memory and became
unusable. top showed the memory usage of the Ruby process to be about
400MB and growing.

Is there something stupid that I’m doing?

Joe

2

On Apr 30, 2006, at 6:04 PM, Joe Van D. wrote:

blob = Blob.create :data => uploaded_file.read

When /files/podcast.mp3 is 32 megabytes and I run the test, memory
spikes hugely. My laptop quickly began swapping out memory and became
unusable. top showed the memory usage of the Ruby process to be about
400MB and growing.

Is there something stupid that I’m doing?

Are you using a blob field type rather than the postgresql large
objects?

-Robby

Robby R.
Founder & Executive Director

PLANET ARGON, LLC
Ruby on Rails Development, Consulting & Hosting

www.robbyonrails.com

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4968 [fax]

3

On 4/30/06, Robby R. [email protected] wrote:

upload }

Are you using a blob field type rather than the postgresql large
objects?

Probably. Whatever using the :binary thingy in migrations gives me.

Joe

4

On 4/30/06, Joe Van D. [email protected] wrote:

post :create, :product => valid_product, :media => { :image =>
Is there something stupid that I’m doing?

Are you using a blob field type rather than the postgresql large
objects?

Probably. Whatever using the :binary thingy in migrations gives me.

Even a 10 megabyte file makes the Rails process consume a hell of a
lot of memory. And then postmaster consumes quite a lot as well.

5

On Apr 30, 2006, at 6:48 PM, Joe Van D. wrote:

Even a 10 megabyte file makes the Rails process consume a hell of a
lot of memory. And then postmaster consumes quite a lot as well.

Which postgresql adapter are you using?

postgres or postgres-pr ?

It’d be my guess that postgres-pr isn’t doing its garbage collecting
or something…

-Robby

Robby R.
Founder & Executive Director

PLANET ARGON, LLC
Ruby on Rails Development, Consulting & Hosting

www.robbyonrails.com

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4968 [fax]

6

On 4/30/06, Robby R. [email protected] wrote:

It’d be my guess that postgres-pr isn’t doing its garbage collecting
or something…

postgres. On linux.

7

On Monday 01 May 2006 02:04, Joe Van D. wrote:

When /files/podcast.mp3 is 32 megabytes and I run the test, memory
spikes hugely. My laptop quickly began swapping out memory and became
unusable. top showed the memory usage of the Ruby process to be about
400MB and growing.

Is there something stupid that I’m doing?

Perhaps consider using the file_column extension instead of storing the
files
in the database:

http://www.kanthak.net/opensource/file_column/

Cheers,

~Dave

–

Dave S.
Rent-A-Monkey Website Development

PGP Key: http://www.rentamonkey.com/pgpkey.asc

8

Larry Diehl wrote:

in the database:
to as high as 500mb and stays there.
One thing I have noticed is that manually invoking the garbage collector via
GC.start right after the upload brings memory down to 100mb.

I don’t seem to have received the earlier messages in this thread…

There was some discussion of ActiveRecord’s memory use when handling
BLOBs over a year ago, triggered by this article:

http://www.voxclandestina.com/ruby/rails/2005-05-24/improving-activerecord-part-1

in a thread “Improving ActiveRecord Part 1” started at 8:09pm on 24th
May 2005. I don’t know if things have changed, but the article pointed
out how Rails made multiple copies of large data items on their way to
the database.

That wouldn’t explain why memory use continues to grow, though.

regards

Justin

9

Dave S. <dave@…> writes:

in the database:

http://www.kanthak.net/opensource/file_column/

Cheers,

~Dave

I’m also having this issue with a MYSQL database. Before the upload, my
ruby
process memory is at 19-22mb. When I upload a 37mb file, memory soon
increases
to as high as 500mb and stays there.
One thing I have noticed is that manually invoking the garbage collector
via
GC.start right after the upload brings memory down to 100mb.