Aws/s3 0.1.0 (initial release)

This is the initial release of aws/s3, a ruby library for Amazon’s
Simple Store
Service’s (S3) REST API.

It implements the entire REST API aiming to be simple, comprehensive and
extensible.

Aside from the basics, it supports ACLs (reading and writing),
BitTorrent
downloads and bucket access logging (enabling, configuring, fetching).
Data can
be streamed both up and down.

Installation instructions, a tutorial and API documentation can be found
here:

http://amazon.rubyforge.org

This is the initial release, but it has comprehensive test coverage and
we
are using it in production at 37signals.

Thanks!
marcel

On 11/28/06, Marcel Molina Jr. [email protected] wrote:

This is the initial release of aws/s3, a ruby library for Amazon’s Simple Store
Service’s (S3) REST API…

Wow. So has “Ruby S3 library” become the new “Ruby templating
library”? It seems like we’re getting daily announcements about these.
:wink:

Lyle J. wrote:

On 11/28/06, Marcel Molina Jr. [email protected] wrote:

This is the initial release of aws/s3, a ruby library for Amazon’s
Simple Store
Service’s (S3) REST API…

Wow. So has “Ruby S3 library” become the new “Ruby templating
library”? It seems like we’re getting daily announcements about these.
:wink:

Yeah, I’m wishing I hadn’t announced Net::Amazon::S3 yesterday. I like
aws/s3 more than my own library. :slight_smile:

Let me be the first to say thank you!

We’re using S3 in one of our apps but the support was hacked in so I
look forward to checking this out with the hope it will improve our
setup.

Great stuff!

Marcel,

Looks really cool! I’ve checked out a few of the S3 libs that have been
released recently, and I like structure you’ve come up with…

Quick question… I tried swapping out some S3 code in an app to use
aws/s3,
and for some reason I’m having routing difficulties in my app now. My
namespaced controllers aren’t being loaded, for some reason. Is there
any
obvious reason why this would happen? (and, yes, I know namespaces are
on
the top ten don’t-do things. I may eliminate them, but for now they make
sense.)

Matt

On 11/29/06, Marcel Molina Jr. [email protected] wrote:

downloads and bucket access logging (enabling, configuring, fetching).

Thanks!
marcel

Marcel Molina Jr. [email protected]


Matt W.

Thermal Creative
http://blog.thermalcreative.com

On Wed, Nov 29, 2006 at 11:07:56AM +1300, Matt W. wrote:

Looks really cool! I’ve checked out a few of the S3 libs that have been
released recently, and I like structure you’ve come up with…

Hey Matt, glad you like it so far.

Quick question… I tried swapping out some S3 code in an app to use aws/s3,
and for some reason I’m having routing difficulties in my app now. My
namespaced controllers aren’t being loaded, for some reason. Is there any
obvious reason why this would happen? (and, yes, I know namespaces are on
the top ten don’t-do things. I may eliminate them, but for now they make
sense.)

Nothing comes to mind just from what you are describing. A stack trace
would
likely help, or a quick summary of how you plugged aws/s3 into your
application (via rubygems or by putting it in the vendor/plugins
directory)
as well as what namedspaced controllers you have that aren’t being
loaded.

When it comes to things not being loaded and/or routes Nicholas S.
is the expert. I’m
not too familiar with the subtleties of it these days.

marcel

Marcel Molina Jr. [email protected]

Marcel,

I did notice one thing. Check out this stack trace:

NameError (uninitialized constant Account::ArtistController):

/vendor/rails/activerecord/lib/…/…/activesupport/lib/active_support/dependencies.rb:260:in
`load_missing_constant’

/vendor/rails/activerecord/lib/…/…/activesupport/lib/active_support/dependencies.rb:446:in
const_missing_not_from_s3_library' /vendor/plugins/aws/lib/aws/s3/extensions.rb:111:in const_missing’

/vendor/rails/activerecord/lib/…/…/activesupport/lib/active_support/inflector.rb:162:in
`constantize’

/vendor/rails/activerecord/lib/…/…/activesupport/lib/active_support/core_ext/string/inflections.rb:148:in
constantize' /vendor/rails/actionpack/lib/action_controller/routing.rb:1246:in recognize’
/vendor/rails/railties/lib/dispatcher.rb:40:in dispatch' /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel/rails.rb:78:in process’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel/rails.rb:76:in
process' /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel.rb:587:in process_client’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel.rb:586:in
process_client' /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel.rb:700:in run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel.rb:700:in
run' /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel.rb:687:in run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel/configurator.rb:271:in
run' /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel/configurator.rb:270:in run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/bin/mongrel_rails:125:in
run' /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel/command.rb:211:in run’
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/bin/mongrel_rails:235
/usr/local/bin/mongrel_rails:18

Looks like the S3 library extends ActiveSupport
(const_missing_not_from_s3_library)? I’m not a ruby guru (sigh), so it’s
not
immediately apparent to me how this could affect things, but I suspect
that
this is the culprit…

Matt

On 11/29/06, Matt W. [email protected] wrote:

As soon as I call
trace to show.
On 11/29/06, Marcel Molina Jr. [email protected] wrote:

aws/s3,
would
marcel
Thermal Creative
http://blog.thermalcreative.com


Matt W.

Thermal Creative
http://blog.thermalcreative.com

Thanks Marcel! I’ve been investigating S3 storage for my upcoming app.
How does this library compare with s33r
http://rubyforge.org/projects/s33r/?

Shane
http://shanesbrain.net

Marcel,

I’ve tried it both ways, via RubyGems and via svn:externals…

This is truly bizarre… At this point I’m assuming that I must be doing
something wrong. I’m trying to set the base connection info from
environment.rb so that it’s available within my app, as well as within
Backgroundrb.

As soon as I call

AWS::S3::Base.establish_connection!(
:access_key_id => ‘my_id’,
:secret_access_key => ‘my_key’
)

in environment.rb, all of my controllers that are namespaced (i.e.
Account::IndexController < Account:ActionController) stop being
recognized.
For instance, “account/index” will now render “index”. Any errors that
I’m
getting are just routing recognition errors, so I don’t have much of a
stack
trace to show.

I’d be interested to see if anyone else can duplicate this. It’s just so
weird that I can’t believe that it’s the fault of aws-s3. I’ll keep
looking
at it and see if I can find anything obvious.

Thanks!

Matt

On 11/29/06, Marcel Molina Jr. [email protected] wrote:

and for some reason I’m having routing difficulties in my app now. My
application (via rubygems or by putting it in the vendor/plugins


Matt W.

Thermal Creative
http://blog.thermalcreative.com

On Wed, Nov 29, 2006 at 12:14:20PM +1300, Matt W. wrote:

const_missing_not_from_s3_library' /vendor/plugins/aws/lib/aws/s3/extensions.rb:111:in const_missing’

Looks like the S3 library extends ActiveSupport
(const_missing_not_from_s3_library)? I’m not a ruby guru (sigh), so it’s not
immediately apparent to me how this could affect things, but I suspect that
this is the culprit…

S3 does alias and redefine const_missing but it chains back to the
original
if its particular use case doesn’t apply to the given constant and then
just
does what ActiveSupport would have done anyway, which is reflected in
that
strack trace as you can see eventually the ActiveSupport version gets
called.

Just to sanity check, you could go into lib/aws/s3/extensions.rb and
remove
the definition to const_missing_from_s3_library and the two calls to
alias_method below it. I threw in that const missing recently just to
add a
tiny convenience to the API. What it provides isn’t even documented and
the
library doesn’t depend on it so you can safely take it away. I probably
will
remove it. I was just trying it on to see how it felt. But either way,
my
const_missing shouldn’t be getting in the way since it’s chaining back
to the
original.

marcel

Marcel Molina Jr. [email protected]

On Tue, Nov 28, 2006 at 07:01:51PM -0600, Shane V. wrote:

Thanks Marcel! I’ve been investigating S3 storage for my upcoming app.
How does this library compare with s33r
http://rubyforge.org/projects/s33r/?

Hey Shane,

From looking around at the s33r code for a bit, it appears to be one of
the
more complete S3 libraries I’ve seen. It supports similar functionality
to
aws/s3. The differences are mostly in the way the library is decomposed
and
the API it exposes to you, its client. Sort of like, what’s the
difference
between Ruby and Python? In some ways not a lot. It comes back to what
people
prefer. What they find natural and comfortable.

At a glance s33r seems like a fine library. It would be up to you to
determine which is
a better fit for you. I’d encourage you to try them both out.

marcel

Marcel,

For some whacky reason it has to do with your extension to String:

class String
def underscore
gsub(/([A-Z]+)([A-Z][a-z])/,‘\1_\2’).
gsub(/([a-z\d])([A-Z])/,‘\1_\2’).
downcase
end
end

If I get rid of that, I’m able to get at my controllers.

Basically what I’ve been doing is to load the dev console, test that I
can
access the controllers (i.e. Account::AccountController), then require
the
s3/extensions.rb file manually, and then see what happens when I require
another namespaced controller (i.e. Account::ArtistController). With the
library as it is, once I require the extensions file I can no longer get
at
my controllers, and it fails with load_missing_constant throwing
“uninitialized constant”. If I get rid of the extension to String, it
doesn’t fail anymore and I get get at all of my controllers even after
requiring extensions.

So, it doesn’t appear to be your const_missing_from_s3_library function
(which is very cool, by the way! I’ll be sure to use that. I don’t think
it
made it into the docs, did it?), but is instead the underscore function
on
String.

I believe that it is overriding the underscore extension to string in
ActiveSupport::CoreExtensions::String::Inflections#underscore. Since
Inflector is used to derive the controller file names for inclusion, I
think
this is what’s causing the problem.

Matt

On 11/29/06, Marcel Molina Jr. [email protected] wrote:

/vendor/rails/activerecord/lib/…/…/activesupport/lib/active_support/dependencies.rb:260:in

immediately apparent to me how this could affect things, but I suspect

const_missing shouldn’t be getting in the way since it’s chaining back to
the
original.

marcel

Marcel Molina Jr. [email protected]


Matt W.

Thermal Creative
http://blog.thermalcreative.com

On Wed, Nov 29, 2006 at 02:13:49AM +0000, Marcel Molina Jr. wrote:

Ah, that makes sense. I don’t bundle ActiveSupport with the library, but I do
Thanks for tracking that down. That’s my bad. Sorry about that. I’ll be more
mindful of that kind of thing when I extend built in classes. I tried to keep
my extensions to a minimum.

Thanks again for tracking that down. I’m about to push a 0.1.1 point
release
because I think it’s justifiable. I should have known better :slight_smile:

marcel

Marcel Molina Jr. [email protected]

On Wed, Nov 29, 2006 at 03:06:25PM +1300, Matt W. wrote:

end
Ah, that makes sense. I don’t bundle ActiveSupport with the library, but
I do
define an underscore method which ActiveSupport also defines, except
mine is
simpler and doesn’t take into account all the stuff that ActiveSupport’s
does
because it’s not intended to to do all the constant name manipulation
that
ActiveSupport’s does. It slipped my mind that this would mess up
ActiveSupport’s underscore method. The fix should be easy, I’ll just do
this:

def underscore
# …
end unless public_instance_methods.include? ‘underscore’

Thanks for tracking that down. That’s my bad. Sorry about that. I’ll be
more
mindful of that kind of thing when I extend built in classes. I tried to
keep
my extensions to a minimum.

marcel

Marcel Molina Jr. [email protected]

Marcel,

No worries, mate ;). I’m just glad I was able to track it down… A good
ego-booster for a rubynewby such as myself.

I’ll update my copy when you get that checked in…

Thanks!

Matt

On 11/29/06, Marcel Molina Jr. [email protected] wrote:

gsub(/([a-z\d])([A-Z])/,'\1_\2').

because it’s not intended to to do all the constant name manipulation that
mindful of that kind of thing when I extend built in classes. I tried to
keep
my extensions to a minimum.

marcel

Marcel Molina Jr. [email protected]


Matt W.

Thermal Creative
http://blog.thermalcreative.com

On Wed, Nov 29, 2006 at 03:38:30PM +1300, Matt W. wrote:

performance when sending files.

Ref: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/222971

Yeah, I noticed earlier while reading around in the code that he opened
up
Net::HTTP and redefined that code to allow for customization of the
block
size. Others have also bemoaned unfortunately small default sizes in
Net::HTTP.

Thanks for pointing that out. It seems like a sensible change to make.

marcel

Marcel Molina Jr. [email protected]

Nice release with the S3 Marcel.

Question about the service itself which I have been contemplating. I
want to utilize a service such as S3 seeing as the cost for large
amounts of bandwidth far exceed even the best prices I have seen from
hosting providers. My concern is that I will still have to take on the
initial burden of having to first upload the file to my servers, then
stream it to S3. Is there a way to directly upload to S3 without
having to spend bandwidth on my servers only to relay it to theirs?

Thanks in advance.
Nathaniel.

On 11/28/06, Marcel Molina Jr. [email protected] wrote:

yourself. Perhaps this would be a useful extension? I didn’t see it in there
Thanks for pointing that out. It seems like a sensible change to make.

marcel

Marcel Molina Jr. [email protected]


Nathaniel Steven Henry Brown

Open Source Insight - http://nshb.net
Open Source Development - http://inimit.com
Open Source Training - http://osevents.com

Marcel,

One quick observation about your implementation…

One thing that S33r deals with, and I had dealt with in my previous
hacky S3
implementation, is the small block size (1k, I believe) that Net::HTTP
uses
when streaming requests. Unfortunately it’s hard-coded in so you can’t
set
it yourself. Apparently this causes requests to be extremely CPU-bound,
and
can lead to unreasonable CPU usage when streaming files. S33r has a fix
in
there to override the block size used by Net::HTTP, and allows you to
set it
yourself. Perhaps this would be a useful extension? I didn’t see it in
there
anywhere, but it from what I’ve read it’s a pretty easy way to boost the
performance when sending files.

Ref: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/222971

Matt

On 11/29/06, Matt W. [email protected] wrote:

Matt

def underscore
simpler and doesn’t take into account all the stuff that ActiveSupport’s


Matt W.

Thermal Creative
http://blog.thermalcreative.com


Matt W.

Thermal Creative
http://blog.thermalcreative.com

Nathaniel,

AFAIK, there is not. All files loaded onto S3 have to be authenticated,
for
obvious reasons, so all transmission to S3 should be done server-side.
Otherwise, I suppose you could send them directly to S3 client-side with
Flash or something, but you don’t want to ever be in a position where
your
AWS authorization stuff is available to a client machine.

If anyone else knows better, please enlighten me :wink:

Matt

On 11/29/06, Nathaniel B. [email protected] wrote:

having to spend bandwidth on my servers only to relay it to theirs?
hacky S3

yourself. Perhaps this would be a useful extension? I didn’t see it in
Net::HTTP and redefined that code to allow for customization of the


Matt W.

Thermal Creative
http://blog.thermalcreative.com

Should I be getting this after following the install instructions in the
aws/s3 Rubydocs?

theBox:~ muppet
$ irb -rubygems
irb(main):001:0> require ‘aws/s3’
NameError: uninitialized constant XML
from /usr/local/lib/ruby/gems/1.8/gems/aws-
s3-0.1.1/lib/aws/s3/extensions.rb:119:in
const_missing' from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.1.1/lib/aws/s3.rb:59 from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:inrequire’

    from (irb):1

It looks somewhat like the problems that sprung up in another stack
trace
someone posted. Upon removing const_missing_from_s3_library, I still
get
the following error:

/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.1.1/bin/…/lib/aws/s3.rb:60:NameError:
uninitialized constant XML

I’ve got xml-simple installed, so I don’t know what’s going on. Any
ideas?

TIA.