I will soon be using the Amazon S3 interface on a RoR web app that is
about 50% complete at this point. I grabbed the S3 interface from
Amazon and began looking through the code and I found this:
uses Net::HTTP to interface with S3. note that this interface
should only
be used for smaller objects, as it does not stream the data. if you
were
to download a 1gb file, it would require 1gb of memory. also, this
class
creates a new http connection each time. it would be greatly
improved with
some connection pooling.
My jaw kinda dropped open at this point as my web app uploads large
files (up to 200 meg in some cases). Has anyone ran into this issue,
and if so how did you go about getting around this? I’m not sure what
they mean by connection pooling to be honest. I thought that referred
to the database connections.
I would check out s33r… It’s a gem you can install that’s made some
improvements to the Amazon code. The biggest problem I’ve seen so far is
that Net:HTTP has some very small buffer sizes hardcoded, and the s33r
author has written some workarounds for this.
I’m in your situation as well… I’m going to be integrating S3 into a
project soon, and I actually wrote a bunch of extra stuff before I found
s33r. I haven’t used it yet, but I took a look at the code and it seems
that
the author has solved some of the bigger issues.
