Caching servers in Local ISPs!

Also sync to a temp folder and move after completion or nginx will attempt
to send partial files.

Oh right. Thanks for quick help and suggestion :). I’ll look into
wanproxy
now.

shahzaib1232 Wrote:

@itpp, i just used your method try_files and it worked flawlessly :).
Following is the testing config :

server {
listen 80;
server_name domain.com;
root /var/www/html/files;

location / {

location ~* (.mp3|.avi|.mp4)$ {

Should i use rsync or lsync for mirroring the files between Origin and
caching server ?

Whatever works for you, I’d prefer rsync since that’s easier to schedule
for
off-peek hours.
Also sync to a temp folder and move after completion or nginx will
attempt
to send partial files.
see also http://wanproxy.org/

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250645#msg-250645

shahzaib1232 Wrote:

be cached not for any other client, does nginx support that ??
You could do this based on some IP ranges or via
https://github.com/flant/nginx-http-rdns

See
http://serverfault.com/questions/380642/nginx-how-to-redirect-users-with-certain-ip-to-special-page
and
http://www.cyberciti.biz/faq/nginx-redirect-backend-traffic-based-upon-client-ip-address/

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250707#msg-250707

Thanks a lot itpp. :slight_smile: I’ll look into it and get back to you.

Thanks again for quick solution :slight_smile:

@itpp I am currenlty proceeding with proxy_cache method just because i
had
to done this in emergency mode due to boss pressure :-|. I have a quick
question, can i make nginx to cache files for specific clients ?

Like, if our caching servers are deployed by only single ISP named
“ptcl”.
So if ip from ptcl client is browsing video, only his requested file
should
be cached not for any other client, does nginx support that ??

I know its kind of funny, but i’ve to complete this task :frowning:

On Thu, Jun 5, 2014 at 12:23 AM, shahzaib shahzaib
[email protected]

Hello itpp,i have been abled to use ngx_http_geo_module. Now the
request
coming from local ISP will first go to the main server (US) and then
main
server will check if the ip is 1.2.3.4 so it’ll direct the request to
the
local caching server and than caching server will check if the file is
cached or it should again get the file from main server and cache it
locally.

When i tested it locally, it worked fine but the file URL in firebug is
coming from MAIN server when it should have come from the local caching
server. I can also see the caching directory size increases when the
matching client via geo module is directed to the local caching server
but
the URL remains the same in firebug.

US config :-

geo $TW {
default 0;
192.168.1.0/24 1;
}

server {
listen 80;
server_name 002.files.com;

limit_rate 600k;

    location / {
        root   /var/www/html/files;
        index index.html index.htm index.php;
 #      autoindex on;

}

location ~ .(mp4|jpeg|jpg)$ {
mp4;
root /var/www/html/files;

             if ($TW) {
                    proxy_pass http://192.168.22.32:80;
            }

            expires 7d;
    valid_referers none blocked  domain.com *.domain.com 

blog.domain.com
*.facebook.com *.twitter.com *.files.com *.pump.net domain.tv
*.domain.tv
domainmedia.tv www.domainmedia.tv embed.domainmedia.tv;
if ($invalid_referer) {
return 403;
}
}
}

Edge config :-

proxy_ignore_headers “Set-Cookie”;
proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=static:100m
loader_threshold=300 loader_files=10 inactive=1d
max_size=300000m;

proxy_temp_path /data/nginx/tmp 1 2;
add_header X-Cache-Status $upstream_cache_status;
add_header Accept-Ranges bytes;
max_ranges 512;

server {

    listen       80;
    server_name  192.168.22.32;
    root /var/www/html/files;
    location ~ \.(mp4|jpeg|jpg)$ {
           root   /var/www/html/files;
            mp4;
            try_files $uri @getfrom_origin;

        }


    location @getfrom_origin {
    proxy_pass http://002.files.com:80;
    proxy_cache_valid 200 302   60m;
    proxy_cache_valid any 1m;
    proxy_cache static;
    proxy_cache_min_uses 1;
    }

Maybe i need to add some variable to get original server ip ?

http://nginx.org/en/docs/http/ngx_http_geo_module.html
http://nginx.org/en/docs/http/ngx_http_geo_module.html

On Fri, Jun 6, 2014 at 8:56 PM, shahzaib shahzaib
[email protected]

Thanks itpp but the issue is still same and still the ip is from the
main
server in inspect element as well as in local-caching nginx access logs,
i
am getting the client ip as main-server’s ip instead of original client
ip
and i am sure that i am doing something wrong. Well i have another
question
now, as our test with CIDR notation worked well with nginx geo module
and
nginx decided to route specific ips to specific server (caching server).

So, the specific subnet coming from our ISP to the main server will be
routed to the local caching server and our ISP will have to tell us each
time to add specific ip prefix in the nginx config to route them towards
their caching server. So the problem is, whenever few hundreds ip
prefixes
are added to their network, they’ll have to provide us those prefixes in
order to enable caching for newly added ips.

We just had a chat with our local ISP and he said that you should use
some
services like BGP to automatically detect if any new ip prefixes are
added
to our network and we’ll not have to tell you each time we add some ip
prefixes to our network.

Could you guide me how could i make this work in our environment. The
basic
architecture of our network is :-

Two static servers (serving mp4,jpg). One server located in US and one
server located in Local ISP.

I hope you can put me on some track as you did in the past and provide
me
some kick start to work with BGP.

shahzaib1232 Wrote:

Thanks itpp but the issue is still same and still the ip is from the
main
server in inspect element as well as in local-caching nginx access
logs, i
am getting the client ip as main-server’s ip instead of original
client ip

It could be the case the traffic you are getting is from the ISP proxy
which
could mean that any traffic is from that ISP only, which makes it easier
to
determine which to serve from local. Ask the ISP from where the traffic
is
coming from, if it is a proxy then proxy=local.

So, the specific subnet coming from our ISP to the main server will be
routed to the local caching server and our ISP will have to tell us
each
time to add specific ip prefix in the nginx config to route them
towards
their caching server. So the problem is, whenever few hundreds ip
prefixes
are added to their network, they’ll have to provide us those prefixes
in
order to enable caching for newly added ips.

See above, if this is not the case look into
https://github.com/flant/nginx-http-rdns if a client has something like
‘p1234.adsl-pool2-auckland.au’ you can redirect based on a part of the
client dns name, your ISP can tell you which DHCP named pools there are.

If you can’t get the client IP of hostname you gonna need to do some
wiresharking to see where the info is, if it is anywhere.
If the ISP is using a proxy to pass clients to your server ask them to
add a
header with the client ip/hostname.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250950#msg-250950

shahzaib1232 Wrote:

Maybe i need to add some variable to get original server ip ?

https://www.google.nl/#q=nginx+geo+remote+ip+address

http://www.cyberciti.biz/faq/nginx-redirect-backend-traffic-based-upon-client-ip-address/

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250895#msg-250895

shahzaib1232 Wrote:

local ISP
so they’ll not have to provide us thousands of prefixes on daily
basis.

Why not use a DNS for the clients? your making things too complicated.

[email protected] -> edge1.streaming.au ISP-1-DNS -> 12.34.56.78
(which
is your edge box)
[email protected] -> edge1.streaming.au ISP-2-DNS -> 99.88.77.66
(which
is your box in the US)

Anyone from ISP-1 will always be directed to the edge systems, anyone
else
to where-ever you point the dns.

ISP’s also use regional DNS servers which allows you more edge systems
closer to the users.

Anyway, BGP see http://bird.network.cz/ (netflix solution)

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250953#msg-250953

Why not use a DNS for the clients?
How i would be sure that request coming from the ISP-1 on the DNS server
and then point it to the local caching server? I mean i can use View
directive of BIND to route specific ips (local ISP clients) to the local
caching server and what if tomorrow the ISP has added more clients to
their
network ? I’ll also have those new ip prefixes to DNS server.

Please correct me if i am wrong.

i don’t think the solution rdns will be suitable for us. I have checked
the
zebra software to make linux a BGP router
http://www.techrepublic.com/article/use-zebra-to-set-up-a-linux-bgp-ospf-router/

Could you tell me if BGP is capable of doing what we want? Because our
local ISP supports this method and i have no idea how to implement it.

Functionality we need, is to auto detect the new ip prefixes from local
ISP
so they’ll not have to provide us thousands of prefixes on daily basis.

Why not use a DNS for the clients?
How i would be sure that request coming from the ISP-1 on the DNS server
?
and then point it to the local caching server? I mean i can use View
directive of BIND to route specific ips (local ISP clients) to the local
caching server and what if tomorrow the ISP has added more clients to
their
network ? I’ll also have to add those new ip prefixes to DNS server.

On Tue, Jun 17, 2014 at 11:45 PM, shahzaib shahzaib
[email protected]

You don’t need to do anything with a dns that is only local to the
clients
served by the ISP.

Suppose I am in Africa;
Question to my ISP: I’d like to go to new-york
ISP: new-york is located in south-Africa

Suppose I am in the US;
Question to my ISP: I’d like to go to new-york
ISP: new-york is located in the US

The DNS is just a pointer, where ever you have an edge server make the
dns
name point to it, when not point the dns to origin.
Every ISP client gets the DNS servers from their ISP, its really simple.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,250957#msg-250957

shahzaib1232 Wrote:

Typo??

geo {
default 0;
10.0.0.0/8 1;
39.23.2.0/24 1;
112.50.192.0/18 1;
}

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,249997,251019#msg-251019

geo {
default 0;
10.0.0.0/8 1;
39.23.2.0/24 1;
112.50.192.0/18 1;
}

Sorry i didn’t write accurately here but it is 10.0.0.0/8 1; in nginx
config, so the problem is not the wrong syntax for geo {}.

ok, but i have no idea why ISP is asking for BGP and matter of fact is,
i’ll have to make BGP work somehow, so local caching server will fetch
the
new subnets from ISP router automatically (and i don’t know how).

Btw, our local ISP provided us with some testing ip prefixes to check
nginx
based caching. i.e
geo {
default 0;
10.0.0.0/8
39.23.2.0/24 1;
112.50.192.0/18 1;
}

Now whenever we add the prefix 112.50.192.0/18 in geo {} , all the
requests
coming from the 39.23.2.0/24 and 10.0.0.0/8 returns 504 gateway error
and
videos failed to stream. To resolve this issue, we have to remove
112.50.192.0/18 1; from geo block.

why not host those file on a professional CDN instead of in-house?
Because 80% of the traffic is from our country and 50% of that traffic
is
from the ISP we’re talking to and this is the reason we deployed the
caching box on this ISP edge.

On 18 Jun 2014 20:45, “shahzaib shahzaib” [email protected] wrote:

why not host those file on a professional CDN instead of in-house?
Because 80% of the traffic is from our country and 50% of that traffic is
from the ISP we’re talking to and this is the reason we deployed the
caching box on this ISP edge.

But, as this now pretty off-topic thread is repeatedly demonstrating,
you
haven’t deployed diddly squat. You’ve just chucked a server in a rack
and
are having to rely on unpaid, debugging-by-email advice from an
pseudonymous mailing list to get it even near functional. Let alone
properly defined and understood.

If your business needs to do this, pay a professional person or
organisation to help you like others have suggested. The alternative,
which
you appear to be ending up with, is a black box of hacks known only to
yourself and potentially understood by no-one, which will SPoF on you,
personally, until you leave that organisation. You don’t want that.
Trust
me.

Just my 2 cents,
Jonathan

Hi,

ok, but i have no idea why ISP is asking for BGP and matter of fact is,
i’ll have to make BGP work somehow, so local caching server will fetch
the new subnets from ISP router automatically (and i don’t know how).

I strongly suggest you hire some consultant who can help you setting
all those things up, because this is clearly a task too complex for
a single mailing list thread and some nginx configurations.

Also, why not host those file on a professional CDN instead of in-house?

https://www.google.com/search?q=mp4+streaming+cdn

Lukas