Forum: Ruby on Rails large file storing in postgres sucks?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
38a8230ed3d5c685558b4f0aad3fc74b?d=identicon&s=25 Joe Van Dyk (Guest)
on 2006-05-03 18:56
(Received via mailing list)
Hi,

I'm not sure if this is a postgres issue or not, but I'm using
Postgres 8.1.  I have (in my functional tests) the following code:

    upload = fixture_file_upload('/files/podcast.mp3', 'audio/mpeg')
    post :create, :product => valid_product, :media => { :image =>
upload }

And then in the controller:
  blob = Blob.create :data => uploaded_file.read

When /files/podcast.mp3 is 32 megabytes and I run the test, memory
spikes hugely.  My laptop quickly began swapping out memory and became
unusable.  top showed the memory usage of the Ruby process to be about
400MB and growing.

Is there something stupid that I'm doing?

Joe
B84d42a3a5c343f8fc6ab7d7f47fd3f5?d=identicon&s=25 Robby Russell (Guest)
on 2006-05-03 18:57
(Received via mailing list)
On Apr 30, 2006, at 6:04 PM, Joe Van Dyk wrote:

>  blob = Blob.create :data => uploaded_file.read
>
> When /files/podcast.mp3 is 32 megabytes and I run the test, memory
> spikes hugely.  My laptop quickly began swapping out memory and became
> unusable.  top showed the memory usage of the Ruby process to be about
> 400MB and growing.
>
> Is there something stupid that I'm doing?

Are you using a blob field type rather than the postgresql large
objects?

-Robby


Robby Russell
Founder & Executive Director

PLANET ARGON, LLC
Ruby on Rails Development, Consulting & Hosting

www.planetargon.com
www.robbyonrails.com

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4968 [fax]
38a8230ed3d5c685558b4f0aad3fc74b?d=identicon&s=25 Joe Van Dyk (Guest)
on 2006-05-03 18:57
(Received via mailing list)
On 4/30/06, Robby Russell <robby.lists@planetargon.com> wrote:
> > upload }
>
> Are you using a blob field type rather than the postgresql large
> objects?

Probably.  Whatever using the :binary thingy in migrations gives me.

Joe
38a8230ed3d5c685558b4f0aad3fc74b?d=identicon&s=25 Joe Van Dyk (Guest)
on 2006-05-03 18:57
(Received via mailing list)
On 4/30/06, Joe Van Dyk <joevandyk@gmail.com> wrote:
> > >    post :create, :product => valid_product, :media => { :image =>
> > > Is there something stupid that I'm doing?
> >
> > Are you using a blob field type rather than the postgresql large
> > objects?
>
> Probably.  Whatever using the :binary thingy in migrations gives me.

Even a 10 megabyte file makes the Rails process consume a hell of a
lot of memory.  And then postmaster consumes quite a lot as well.
B84d42a3a5c343f8fc6ab7d7f47fd3f5?d=identicon&s=25 Robby Russell (Guest)
on 2006-05-03 18:57
(Received via mailing list)
On Apr 30, 2006, at 6:48 PM, Joe Van Dyk wrote:

> Even a 10 megabyte file makes the Rails process consume a hell of a
> lot of memory.  And then postmaster consumes quite a lot as well.

Which postgresql adapter are you using?

postgres or postgres-pr ?

It'd be my guess that postgres-pr isn't doing its garbage collecting
or something...

-Robby

Robby Russell
Founder & Executive Director

PLANET ARGON, LLC
Ruby on Rails Development, Consulting & Hosting

www.planetargon.com
www.robbyonrails.com

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4968 [fax]
38a8230ed3d5c685558b4f0aad3fc74b?d=identicon&s=25 Joe Van Dyk (Guest)
on 2006-05-03 18:57
(Received via mailing list)
On 4/30/06, Robby Russell <robby.lists@planetargon.com> wrote:
> It'd be my guess that postgres-pr isn't doing its garbage collecting
> or something...

postgres.  On linux.
Cf7e5e4b5ca573eec214191fac420a2f?d=identicon&s=25 Dave Silvester (Guest)
on 2006-05-03 18:59
(Received via mailing list)
On Monday 01 May 2006 02:04, Joe Van Dyk wrote:
> When /files/podcast.mp3 is 32 megabytes and I run the test, memory
> spikes hugely.  My laptop quickly began swapping out memory and became
> unusable.  top showed the memory usage of the Ruby process to be about
> 400MB and growing.
>
> Is there something stupid that I'm doing?

Perhaps consider using the file_column extension instead of storing the
files
in the database:

http://www.kanthak.net/opensource/file_column/

Cheers,

~Dave

--

Dave Silvester
Rent-A-Monkey Website Development
http://www.rentamonkey.com/

PGP Key: http://www.rentamonkey.com/pgpkey.asc
72003cf388f6f7f95b433de3df9ccd38?d=identicon&s=25 Larry Diehl (Guest)
on 2006-06-19 04:11
(Received via mailing list)
Dave Silvester <dave@...> writes:

> in the database:
>
> http://www.kanthak.net/opensource/file_column/
>
> Cheers,
>
> ~Dave
>

I'm also having this issue with a MYSQL database. Before the upload, my
ruby
process memory is at 19-22mb. When I upload a 37mb file, memory soon
increases
to as high as 500mb and stays there.
One thing I have noticed is that manually invoking the garbage collector
via
GC.start right after the upload brings memory down to 100mb.
Eea7ad39737b0dbf3de38874e0a6c7d8?d=identicon&s=25 Justin Forder (Guest)
on 2006-06-19 10:06
(Received via mailing list)
Larry Diehl wrote:
>> in the database:
> to as high as 500mb and stays there.
> One thing I have noticed is that manually invoking the garbage collector via
> GC.start right after the upload brings memory down to 100mb.

I don't seem to have received the earlier messages in this thread...

There was some discussion of ActiveRecord's memory use when handling
BLOBs over a year ago, triggered by this article:

http://www.voxclandestina.com/ruby/rails/2005-05-2...

in a thread "Improving ActiveRecord Part 1" started at 8:09pm on 24th
May 2005. I don't know if things have changed, but the article pointed
out how Rails made multiple copies of large data items on their way to
the database.

That wouldn't explain why memory use continues to grow, though.

regards

   Justin
This topic is locked and can not be replied to.