Check for existing remote files

Server NGINX
#1

Hi guys. May be my question will break your nerves but please forgive
me. At first I love Nginx. Nginx is dream come true for poor student
like me :)) which doesn’t have money for sloooooowly web servers for my
projects :)). So, BIG thanks for Igor and all of you guys who contribute
beautiful code for this project. You are awesome!

However, my question ofcource :slight_smile: Can you tell me guys how I can check
for existing remote file.
What i mean:
how can i check http://www.google.com/logo.png or
http://dl.dropbox.com/u/3720261/testdir/file.zip

10x in advance.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,165799,165799#msg-165799

#2

On 1/13/2011 11:40 AM, mignev wrote:

However, my question ofcource :slight_smile: Can you tell me guys how I can check
for existing remote file.
What i mean:
how can i check http://www.google.com/logo.png or
http://dl.dropbox.com/u/3720261/testdir/file.zip

Technically, of course you can build some solution for that, even do it
in your application nginx serves. But think about network delays that
would be raise for each check. It’s really not for fast philosophy.

You have to find workaround. I.e., a script running under the cron every
minute, which checks that www.gooogle.com/logo.png exists, and creates
file “exists” under some location, else removes it. And nginx will check
about this file presence.

#3

Hi.
First thanks for the critique. Such types of solutions are many, but I
don’t want such solution. In my case i have over a million images of
which i don’t know the name or their existence. I want to try for
existence of requested file and if he is found then return to me. If the
requested file is not exist go to my proxy_pass $mybackend; and he will
generate and upload it. WIth this idea i want to skip database layer. I
have some ideas how i can do that but firstly i want to hear your
feedback about that.

10x once again for your advice :slight_smile:

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,165799,165808#msg-165808

#4

2011/1/13 mignev [email protected]:

First thanks for the critique. Such types of solutions are many, but I
don’t want such solution. In my case i have over a million images of
which i don’t know the name or their existence. I want to try for
existence of requested file and if he is found then return to me. If the
requested file is not exist go to my proxy_pass $mybackend; and he will
generate and upload it. WIth this idea i want to skip database layer. I
have some ideas how i can do that but firstly i want to hear your
feedback about that.

server {
…

location ~ ^/remote/(?<remote_url>.+)$ {
proxy_intercept_errors on;
error_page 404 = @mybackend;
proxy_pass http://$remote_url;
}

location @mybackend {
proxy_pass http://mybackend_upstream;
}
}

This shows the general idea about how to do this, but i haven’t tested
so there may be syntax errors.

#5

how can i check http://www.google.com/logo.png or
http://dl.dropbox.com/u/3720261/testdir/file.zip

Well it depends what yout want to do after checking if the file does or
doesnt exist.

If you just want to serve the remote file and fall back to some backup
location/host you can technically use proxy_pass (define 2 or more
upstream
( http://wiki.nginx.org/HttpUpstreamModule ) servers which to check
first
(your own you can define as ‘backup’)) with proxy_next_upstream
http_404;
option ( http://wiki.nginx.org/HttpProxyModule#proxy_next_upstream ).

For something more sophiscated scenario you would need to write either
perl
module or just rewrite the requests to a php script for example.

rr