Forum: NGINX Amazon S3 authentication patch

Hi

I've modified the proxy module to be able to authenticate with Amazon 
S3. It also supports the secure download patch by removing the trailing 
ticket (MD5/timestamp) from the URL.

The rationale behind this is: we're using S3 to offload all our app's 
static files but need some of them to be public (images) and some 
private (downloads for registered users). We're using nginx as a proxy 
to S3 to cache requests and minimize the per-GET cost of S3, so our 
first approach was a security through obscurity one: as the final user 
only sees our URL and not the real bucket's URL, we could make the whole 
bucket public and use the secure download patch just on some locations, 
so that the user couldn't download files from there. But this protection 
was only on our server, if some user guessed our S3 bucket he could 
download everything directly from S3.

With this patch we can make parts of the bucket private, and 
authenticate with S3 so that only our servers are able to download them. 
And it plays nice with the secure download patch, so the files are 
secure both on S3 and through our proxy.

Please find the patch attached. This is my first attempt at hacking more 
than a couple lines into nginx, so I'm sure some code cleanup might 
apply. I particularly don't like using static char arrays (should move 
to ngx_copy and such) and couldn't get ngx_conf_set_flag_slot to work 
for config options.

Config syntax is as follows:

        location /private/ {
                proxy_pass 
http://BUCKET.s3.amazonaws.com/private/;
                proxy_s3_auth on;
                proxy_s3_secure_download on;   # optional if you're also 
using secdownload
                proxy_s3_bucket BUCKET;
                proxy_s3_user S3_USER_ID;
                proxy_s3_pass S3_PASSWORD;
        }

Any comments, suggestions, and code corrections are welcome. :-)

Regards

Hello!

On Thu, Feb 04, 2010 at 09:33:21AM +0100, Vicente Aguilar wrote:

> I've modified the proxy module to be able to authenticate with 
> Amazon S3. It also supports the secure download patch by 
> removing the trailing ticket (MD5/timestamp) from the URL.

I believe better aproach would be to build separate module able to
calculate required hash and then use proxy_set_header with
appropriate variable.

Note well: rigth now this should be possible with embedded perl.
Though I think having embedded perl compiled into nginx just for
this isn't a good idea, and C implementation whould be safer and
faster.

[...]

Maxim Dounin

On Thu, Feb 4, 2010 at 4:10 AM, Maxim Dounin <mdounin@mdounin.ru> wrote:
> appropriate variable.
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://nginx.org/mailman/listinfo/nginx
>

For whatever it's worth, I agree that we should make things as easy to
combine as possible, rather than making monolithic modules that cover
all cases.

Small and light :D

-- Merlin

Hi

>> I believe better aproach would be to build separate module able to
>> calculate required hash and then use proxy_set_header with
>> appropriate variable.
> 
> For whatever it's worth, I agree that we should make things as easy to
> combine as possible, rather than making monolithic modules that cover
> all cases.

I guess you're both right, but I wouldn't know how to do that right now, 
I'm new to nginx internals. :-) I needed this feature ASAP and this 
approach was easier.

I might rework it into a separate module as time permits.

Regards

Vicente,

We've been using this but we're looking at moving to newer versions of
nginx - have you released any patches for any versions beyond 0.7.64?

Thanks,
Nicholas

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,50336,159951#msg-159951

On Feb 05, Vicente Aguilar wrote:
>I guess you're both right, but I wouldn't know how to do that right
>now, I'm new to nginx internals. :-) I needed this feature ASAP and
>this approach was easier.
>
>I might rework it into a separate module as time permits.


Here is a module that provides basic S3 proxying without patching any of
the existing files: https://github.com/anomalizer/ngx_aws_auth