@RR, i’ve prepared the local environment with the following structure :-
client → nginx (edge) → varnish → backend (Origin)
When i tested this method i.e :-
3 clients requested for test.mp4 (file size is 4mb) → nginx → file
not
existed (proxy_store) → varnish → backend (fetch the file from
origin).
When nginx proxied these three requests subsequently towards the
varnish,
despite of filling 4mb of tmp dir it was filled with 12MB which means
nginx
is proxying all three requests towards the varnish server and creating
tmp
files as long as the file is not downloaded. (The method was failed)
Although On putting varnish in front of nginx solved this issue.
3 clients requested for test.mp4(file size is 4mb) → varnish(proxying
all
requests for mp4,jpg) → nginx.(fetch the file from origin).
This time tmp dir was filled with the size of 4Mb which means varnish
combined those 3 subsequent requests into 1.
Now varnish also has a flaw to send subsequent requests for same file
towards the nginx i.e
1st user requested for file http://edge.files.com/videos/test.mp4.
During
the downloading of first requested file, the second user also requested
the
same file but with random seeking
http://edge.files.com/videos/test.mp4?start=33 . Now as the request uri
is
changed, there are two different requests for the same file in varnish
and
again nginx tmp directory was filled with 8MB instead of 4 which means
nginx downloaded the full file twice. So Random seeking will only work
once
the file is cached locally, otherwise nginx will keep on creating tmp
files
against random seekings.
I have two questions now :-
- If there’s way to prevent duplicate downloads for random seekings
while
the file not downloaded yet ? Note :- We cannot disable mp4 module.
- Should nginx in front of varnish never work as expected or i am doing
something wrong ?
Following are existing varnish in front of nginx configs. Please let me
know if something need to be fixed :-
varnish config :-
backend origin002 {
.host = “127.0.0.1”;
.port = “8080”;
}
backend origin003 {
.host = “127.0.0.1”;
.port = “8080”;
}
sub vcl_recv {
if ( req.http.host == "origin002.files.com" ){
set req.backend_hint = origin002;
} elsif ( req.http.host == "origin003.files.com" ){
set req.backend_hint = origin003;
} elsif ( req.http.host == "origin004.files.com" ){
set req.backend_hint = origin004;
}
}
sub vcl_backend_response {
if (bereq.url ~ “^[^?].(mp4|jpeg|jpg)(?.)?$”){
set beresp.do_stream = true;
return (deliver);
}
set beresp.grace = 1m;
return (deliver);
}
sub vcl_deliver {
}
Nginx config :-
server {
listen 127.0.0.1:8080;
server_name origin002.files.com;
root /var/www/html/tunefiles;
location ~ \.(mp4|jpeg|jpg)$ {
root /var/www/html/tunefiles;
mp4;
error_page 404 = @fetch;
}
location ~ \.(php)$ {
proxy_pass http://origin002.files.com:80;
}
location @fetch {
internal;
proxy_max_temp_file_size 0;
proxy_pass http://content.files.com:80$uri;
proxy_store on;
proxy_store_access user:rw group:rw all:r;
root /var/www/html/tunefiles;
}
}
I can also send the configs which were configured for nginx in front of
varnish (which didn’t resolved my issue).
BTW, i am using malloc storage instead of file in varnish.
Thanks !!
On Wed, Sep 24, 2014 at 6:55 PM, shahzaib shahzaib
[email protected]
