Forum: Ruby on Rails large file storing in postgres sucks?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Joe Van D. (Guest)
on 2006-05-03 20:56
(Received via mailing list)
Hi,

I'm not sure if this is a postgres issue or not, but I'm using
Postgres 8.1.  I have (in my functional tests) the following code:

    upload = fixture_file_upload('/files/podcast.mp3', 'audio/mpeg')
    post :create, :product => valid_product, :media => { :image =>
upload }

And then in the controller:
  blob = Blob.create :data => uploaded_file.read

When /files/podcast.mp3 is 32 megabytes and I run the test, memory
spikes hugely.  My laptop quickly began swapping out memory and became
unusable.  top showed the memory usage of the Ruby process to be about
400MB and growing.

Is there something stupid that I'm doing?

Joe
Robby R. (Guest)
on 2006-05-03 20:57
(Received via mailing list)
On Apr 30, 2006, at 6:04 PM, Joe Van D. wrote:

>  blob = Blob.create :data => uploaded_file.read
>
> When /files/podcast.mp3 is 32 megabytes and I run the test, memory
> spikes hugely.  My laptop quickly began swapping out memory and became
> unusable.  top showed the memory usage of the Ruby process to be about
> 400MB and growing.
>
> Is there something stupid that I'm doing?

Are you using a blob field type rather than the postgresql large
objects?

-Robby


Robby R.
Founder & Executive Director

PLANET ARGON, LLC
Ruby on Rails Development, Consulting & Hosting

www.planetargon.com
www.robbyonrails.com

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4968 [fax]
Joe Van D. (Guest)
on 2006-05-03 20:57
(Received via mailing list)
On 4/30/06, Robby R. <removed_email_address@domain.invalid> wrote:
> > upload }
>
> Are you using a blob field type rather than the postgresql large
> objects?

Probably.  Whatever using the :binary thingy in migrations gives me.

Joe
Joe Van D. (Guest)
on 2006-05-03 20:57
(Received via mailing list)
On 4/30/06, Joe Van D. <removed_email_address@domain.invalid> wrote:
> > >    post :create, :product => valid_product, :media => { :image =>
> > > Is there something stupid that I'm doing?
> >
> > Are you using a blob field type rather than the postgresql large
> > objects?
>
> Probably.  Whatever using the :binary thingy in migrations gives me.

Even a 10 megabyte file makes the Rails process consume a hell of a
lot of memory.  And then postmaster consumes quite a lot as well.
Robby R. (Guest)
on 2006-05-03 20:57
(Received via mailing list)
On Apr 30, 2006, at 6:48 PM, Joe Van D. wrote:

> Even a 10 megabyte file makes the Rails process consume a hell of a
> lot of memory.  And then postmaster consumes quite a lot as well.

Which postgresql adapter are you using?

postgres or postgres-pr ?

It'd be my guess that postgres-pr isn't doing its garbage collecting
or something...

-Robby

Robby R.
Founder & Executive Director

PLANET ARGON, LLC
Ruby on Rails Development, Consulting & Hosting

www.planetargon.com
www.robbyonrails.com

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4968 [fax]
Joe Van D. (Guest)
on 2006-05-03 20:57
(Received via mailing list)
On 4/30/06, Robby R. <removed_email_address@domain.invalid> wrote:
> It'd be my guess that postgres-pr isn't doing its garbage collecting
> or something...

postgres.  On linux.
Dave S. (Guest)
on 2006-05-03 20:59
(Received via mailing list)
On Monday 01 May 2006 02:04, Joe Van D. wrote:
> When /files/podcast.mp3 is 32 megabytes and I run the test, memory
> spikes hugely.  My laptop quickly began swapping out memory and became
> unusable.  top showed the memory usage of the Ruby process to be about
> 400MB and growing.
>
> Is there something stupid that I'm doing?

Perhaps consider using the file_column extension instead of storing the
files
in the database:

http://www.kanthak.net/opensource/file_column/

Cheers,

~Dave

--

Dave S.
Rent-A-Monkey Website Development
http://www.rentamonkey.com/

PGP Key: http://www.rentamonkey.com/pgpkey.asc
Larry Diehl (Guest)
on 2006-06-19 06:11
(Received via mailing list)
Dave S. <dave@...> writes:

> in the database:
>
> http://www.kanthak.net/opensource/file_column/
>
> Cheers,
>
> ~Dave
>

I'm also having this issue with a MYSQL database. Before the upload, my
ruby
process memory is at 19-22mb. When I upload a 37mb file, memory soon
increases
to as high as 500mb and stays there.
One thing I have noticed is that manually invoking the garbage collector
via
GC.start right after the upload brings memory down to 100mb.
Justin F. (Guest)
on 2006-06-19 12:06
(Received via mailing list)
Larry Diehl wrote:
>> in the database:
> to as high as 500mb and stays there.
> One thing I have noticed is that manually invoking the garbage collector via
> GC.start right after the upload brings memory down to 100mb.

I don't seem to have received the earlier messages in this thread...

There was some discussion of ActiveRecord's memory use when handling
BLOBs over a year ago, triggered by this article:

http://www.voxclandestina.com/ruby/rails/2005-05-2...

in a thread "Improving ActiveRecord Part 1" started at 8:09pm on 24th
May 2005. I don't know if things have changed, but the article pointed
out how Rails made multiple copies of large data items on their way to
the database.

That wouldn't explain why memory use continues to grow, though.

regards

   Justin
This topic is locked and can not be replied to.