# Large file storing in postgres sucks? **URL:** \ **Category:** Rails\ **Created:** [May 3, 2006, 6:56pm UTC](https://www.ruby-forum.com/t/large-file-storing-in-postgres-sucks/59317 "2006-05-03T18:56:59Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![joe-dumoulin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/9dc877/32.png) [@joe-dumoulin](https://www.ruby-forum.com/u/joe-dumoulin)\ **Post date:** [May 3, 2006, 6:56pm UTC](https://www.ruby-forum.com/t/large-file-storing-in-postgres-sucks/59317/1 "2006-05-03T18:56:59Z")
Hi, I’m not sure if this is a postgres issue or not, but I’m using Postgres 8.1. I have (in my functional tests) the following code: ``` upload = fixture_file_upload('/files/podcast.mp3', 'audio/mpeg') post :create, :product => valid_product, :media => { :image => ``` upload } And then in the controller: blob = Blob.create :data =\> uploaded\_file.read When /files/podcast.mp3 is 32 megabytes and I run the test, memory spikes hugely. My laptop quickly began swapping out memory and became unusable. top showed the memory usage of the Ruby process to be about 400MB and growing. Is there something stupid that I’m doing? Joe ---
**Author:** ![joe-dumoulin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/9dc877/32.png) [@joe-dumoulin](https://www.ruby-forum.com/u/joe-dumoulin)\ **Post date:** [May 3, 2006, 6:57pm UTC](https://www.ruby-forum.com/t/large-file-storing-in-postgres-sucks/59317/2 "2006-05-03T18:57:00Z")
On Apr 30, 2006, at 6:04 PM, Joe Van D. wrote: > blob = Blob.create :data =\> uploaded\_file.read > > When /files/podcast.mp3 is 32 megabytes and I run the test, memory > spikes hugely. My laptop quickly began swapping out memory and became > unusable. top showed the memory usage of the Ruby process to be about > 400MB and growing. > > Is there something stupid that I’m doing? Are you using a blob field type rather than the postgresql large objects? -Robby Robby R. Founder & Executive Director PLANET ARGON, LLC Ruby on Rails Development, Consulting & Hosting > **[Planet Argon](https://www.planetargon.com)** > > Planet Argon is a Ruby on Rails development agency specializing in maintenance and support for existing Rails apps. Based in Portland, OR, Planet Argon has been a member of the Rails community since 2004. [www.robbyonrails.com](http://www.robbyonrails.com) +1 503 445 2457 +1 877 55 ARGON [toll free] +1 815 642 4968 [fax] ---
**Author:** ![joe-dumoulin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/9dc877/32.png) [@joe-dumoulin](https://www.ruby-forum.com/u/joe-dumoulin)\ **Post date:** [May 3, 2006, 6:57pm UTC](https://www.ruby-forum.com/t/large-file-storing-in-postgres-sucks/59317/3 "2006-05-03T18:57:01Z")
On 4/30/06, Robby R. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > upload } > > Are you using a blob field type rather than the postgresql large > objects? Probably. Whatever using the :binary thingy in migrations gives me. Joe ---
**Author:** ![joe-dumoulin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/9dc877/32.png) [@joe-dumoulin](https://www.ruby-forum.com/u/joe-dumoulin)\ **Post date:** [May 3, 2006, 6:57pm UTC](https://www.ruby-forum.com/t/large-file-storing-in-postgres-sucks/59317/4 "2006-05-03T18:57:01Z")
On 4/30/06, Joe Van D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > > > post :create, :product =\> valid\_product, :media =\> { :image =\> > > > Is there something stupid that I’m doing? > > > > Are you using a blob field type rather than the postgresql large > > objects? > > Probably. Whatever using the :binary thingy in migrations gives me. Even a 10 megabyte file makes the Rails process consume a hell of a lot of memory. And then postmaster consumes quite a lot as well. ---
**Author:** ![joe-dumoulin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/9dc877/32.png) [@joe-dumoulin](https://www.ruby-forum.com/u/joe-dumoulin)\ **Post date:** [May 3, 2006, 6:57pm UTC](https://www.ruby-forum.com/t/large-file-storing-in-postgres-sucks/59317/5 "2006-05-03T18:57:02Z")
On Apr 30, 2006, at 6:48 PM, Joe Van D. wrote: > Even a 10 megabyte file makes the Rails process consume a hell of a > lot of memory. And then postmaster consumes quite a lot as well. Which postgresql adapter are you using? postgres or postgres-pr ? It’d be my guess that postgres-pr isn’t doing its garbage collecting or something… -Robby Robby R. Founder & Executive Director PLANET ARGON, LLC Ruby on Rails Development, Consulting & Hosting > **[Planet Argon](https://www.planetargon.com)** > > Planet Argon is a Ruby on Rails development agency specializing in maintenance and support for existing Rails apps. Based in Portland, OR, Planet Argon has been a member of the Rails community since 2004. [www.robbyonrails.com](http://www.robbyonrails.com) +1 503 445 2457 +1 877 55 ARGON [toll free] +1 815 642 4968 [fax] ---
**Author:** ![joe-dumoulin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/9dc877/32.png) [@joe-dumoulin](https://www.ruby-forum.com/u/joe-dumoulin)\ **Post date:** [May 3, 2006, 6:57pm UTC](https://www.ruby-forum.com/t/large-file-storing-in-postgres-sucks/59317/6 "2006-05-03T18:57:02Z")
On 4/30/06, Robby R. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > It’d be my guess that postgres-pr isn’t doing its garbage collecting > or something… postgres. On linux. ---
**Author:** ![joe-dumoulin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/9dc877/32.png) [@joe-dumoulin](https://www.ruby-forum.com/u/joe-dumoulin)\ **Post date:** [May 3, 2006, 6:59pm UTC](https://www.ruby-forum.com/t/large-file-storing-in-postgres-sucks/59317/7 "2006-05-03T18:59:17Z")
On Monday 01 May 2006 02:04, Joe Van D. wrote: > When /files/podcast.mp3 is 32 megabytes and I run the test, memory > spikes hugely. My laptop quickly began swapping out memory and became > unusable. top showed the memory usage of the Ruby process to be about > 400MB and growing. > > Is there something stupid that I’m doing? Perhaps consider using the file\_column extension instead of storing the files in the database: [http://www.kanthak.net/opensource/file\_column/](http://www.kanthak.net/opensource/file_column/) Cheers, ~Dave – Dave S. Rent-A-Monkey Website Development > **[Tree Removal Service For Home & Businesses in Salt Lake City, UT](https://www.rentamonkey.com/)** > > Rent A Monkey Tree Service provides the safest, highest-quality tree removal for Salt Lake City, UT homes and businesses. Request a service today! PGP Key: [http://www.rentamonkey.com/pgpkey.asc](http://www.rentamonkey.com/pgpkey.asc) ---
**Author:** ![joe-dumoulin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/9dc877/32.png) [@joe-dumoulin](https://www.ruby-forum.com/u/joe-dumoulin)\ **Post date:** [June 19, 2006, 10:06am UTC](https://www.ruby-forum.com/t/large-file-storing-in-postgres-sucks/59317/8 "2006-06-19T10:06:37Z")
Larry Diehl wrote: > > in the database: > > to as high as 500mb and stays there. > > One thing I have noticed is that manually invoking the garbage collector via > > GC.start right after the upload brings memory down to 100mb. I don’t seem to have received the earlier messages in this thread… There was some discussion of ActiveRecord’s memory use when handling BLOBs over a year ago, triggered by this article: [http://www.voxclandestina.com/ruby/rails/2005-05-24/improving-activerecord-part-1](http://www.voxclandestina.com/ruby/rails/2005-05-24/improving-activerecord-part-1) in a thread “Improving ActiveRecord Part 1” started at 8:09pm on 24th May 2005. I don’t know if things have changed, but the article pointed out how Rails made multiple copies of large data items on their way to the database. That wouldn’t explain why memory use continues to grow, though. regards Justin ---
**Author:** ![joe-dumoulin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/9dc877/32.png) [@joe-dumoulin](https://www.ruby-forum.com/u/joe-dumoulin)\ **Post date:** [June 19, 2006, 4:11am UTC](https://www.ruby-forum.com/t/large-file-storing-in-postgres-sucks/59317/9 "2006-06-19T04:11:22Z")
Dave S. \ writes: > in the database: > > [http://www.kanthak.net/opensource/file\_column/](http://www.kanthak.net/opensource/file_column/) > > Cheers, > > ~Dave I’m also having this issue with a MYSQL database. Before the upload, my ruby process memory is at 19-22mb. When I upload a 37mb file, memory soon increases to as high as 500mb and stays there. One thing I have noticed is that manually invoking the garbage collector via GC.start right after the upload brings memory down to 100mb.