Forum: NGINX Download full mp4 file with proxy_cache or proxy_store!

13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-06-19 08:12
(Received via mailing list)
we're using two servers (one proxy and one backend). Proxy server is
using proxy_cache to cache mp4 files from backend server and working
fine.
When i stream a full video from cache, the header response gives me the
cache-status: HIT but whenever i seek the mp4 file i.e
http://url/test.mp4?start=33 , the Cache-status changes to : MISS . Does
that mean, the proxy server is again downloading the same file after the
33
seconds ?

Can't i use nginx proxy_cache to download whole mp4 file and and than
seek
from it instead of fetching the file again and again ? Does proxy_store
has
this functionality if not proxy_cache ?
63f341734581b167c7b698169bdd2510?d=identicon&s=25 Lukas Tribus (Guest)
on 2014-06-19 17:36
(Received via mailing list)
> proxy_store has this functionality if not proxy_cache ?
You would not have this problem with local files (rsync'ing them to your
server, as was previsouly suggested in the other thread).


What nginx release are you using? You probably need at least 1.5.13 as
per:
http://mailman.nginx.org/pipermail/nginx/2014-June...
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-06-20 21:14
(Received via mailing list)
@Lukas, we're using nginx-1.6 and byte range caching is already enabled
by
default(i guess). Below is the curl request :-

 curl -H Range:bytes=16- -I
http://videos.files.com/files/videos/2014/06/20/14...
HTTP/1.1 206 Partial Content
Server: nginx
Date: Fri, 20 Jun 2014 13:36:05 GMT
Content-Type: video/mp4
Content-Length: 25446010
Connection: keep-alive
Last-Modified: Fri, 20 Jun 2014 11:04:11 GMT
ETag: "53a4152b-184468a"
Expires: Fri, 27 Jun 2014 13:36:05 GMT
Cache-Control: max-age=604800
X-Cache-Status: HIT
Content-Range: bytes 16-25446025/25446026

Could you tell me how can i check using curl that if nginx downloading
the
whole file each time the user seek the video with mp4 psuedo module  i.e
http://url/files/videos/test-360.mp4?start=39.

I am newbie to proxy_cache and much confused about the behavior.

I know rsync is better solution but it cannot cache the videos on fly
instead to have a schedule to rsync file in off-peak hours. We want to
cache only videos which are accessed 10 times and nginx is doing well
with
proxy_cache_min directive.
63f341734581b167c7b698169bdd2510?d=identicon&s=25 Lukas Tribus (Guest)
on 2014-06-21 09:11
(Received via mailing list)
Hi,


> Connection: keep-alive
> Last-Modified: Fri, 20 Jun 2014 11:04:11 GMT
> ETag: "53a4152b-184468a"
> Expires: Fri, 27 Jun 2014 13:36:05 GMT
> Cache-Control: max-age=604800
> X-Cache-Status: HIT
> Content-Range: bytes 16-25446025/25446026
>
> Could you tell me how can i check using curl that if nginx downloading
> the whole file each time the user seek the video with mp4 psuedo
> module  i.e http://url/files/videos/test-360.mp4?start=39.

Check the Content-Length header in the response.


Regards,

Lukas
405f7785199e6fb200f5cd68af04a5f8?d=identicon&s=25 Roman Arutyunyan (Guest)
on 2014-06-23 14:23
(Received via mailing list)
On 19 Jun 2014, at 10:12, shahzaib shahzaib <shahzaib.cb@gmail.com>
wrote:

>
>       we're using two servers (one proxy and one backend). Proxy server is using
proxy_cache to cache mp4 files from backend server and working fine. When i 
stream
a full video from cache, the header response gives me the cache-status: HIT but
whenever i seek the mp4 file i.e http://url/test.mp4?start=33 , the Cache-status
changes to : MISS . Does that mean, the proxy server is again downloading the 
same
file after the 33 seconds ?

Since default proxy_cache_key has $args in it, your second request has a
different cache key, so the
file is downloaded again.

Moreover the mp4 module does not work over proxy cache.  That means even
if you fix the cache key issue
mp4 seeking will not work.  You need to have a local mp4 file to be able
to seek mp4 like that.
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-06-23 14:47
(Received via mailing list)
@Roman thanks for reply,

>> your second request has a different cache key, so the
file is downloaded again.

Means, if a user seeks through the video i.e
http://url/test.mp4?start=99 ,
the whole file again gets downloaded or the partial part of the file
from
99sec to onward gets downloaded ? If the whole file downloaded again
each
time, does nginx support something like, if user seeks through the video
start=99 and the rest of the file gets download instead of the whole
file ?
Is the rsync only solution if not nginx?
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2014-06-23 15:16
(Received via mailing list)
Roman Arutyunyan Wrote:
-------------------------------------------------------
> Moreover the mp4 module does not work over proxy cache.  That means
> even if you fix the cache key issue
> mp4 seeking will not work.  You need to have a local mp4 file to be
> able to seek mp4 like that.

Hmm, what about a hack, if the file is cached keep a link to the cached
file
and its original name, if the next request matches a cached file and its
original name and a seek is requested then pass the cache via its
original
name to allow seeking on the local (but cached) file.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251037,251108#msg-251108
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-06-23 15:28
(Received via mailing list)
>>Hmm, what about a hack, if the file is cached keep a link to the cached
file
and its original name, if the next request matches a cached file and its
original name and a seek is requested then pass the cache via its
original
name to allow seeking on the local (but cached) file.

That means, i should have double storage, one for cached files via
proxy_cache and other for local files via rsync.
405f7785199e6fb200f5cd68af04a5f8?d=identicon&s=25 Roman Arutyunyan (Guest)
on 2014-06-23 16:32
(Received via mailing list)
On 23 Jun 2014, at 16:47, shahzaib shahzaib <shahzaib.cb@gmail.com>
wrote:

> @Roman thanks for reply,
>
> >> your second request has a different cache key, so the
> file is downloaded again.
>
> Means, if a user seeks through the video i.e http://url/test.mp4?start=99 , the
whole file again gets downloaded or the partial part of the file from 99sec to
onward gets downloaded ? If the whole file downloaded again each time, does 
nginx
support something like, if user seeks through the video start=99 and the rest of
the file gets download instead of the whole file ? Is the rsync only solution if
not nginx?

If you have mp4 module enabled at the upstream, then partial mp4 (well,
its not partial actually,
but transformed) is downloaded and cached.  If not then you should
configure a proper proxy_cache_key
to download the whole file once from the upstream and serve the whole
mp4 from the cache.
405f7785199e6fb200f5cd68af04a5f8?d=identicon&s=25 Roman Arutyunyan (Guest)
on 2014-06-23 16:44
(Received via mailing list)
On 23 Jun 2014, at 17:15, itpp2012 <nginx-forum@nginx.us> wrote:

> name to allow seeking on the local (but cached) file.
You can use proxy_store with the mp4 module.

Having a link to a nginx cache file is wrong since cache file has
internal header and
HTTP headers.  Cached mp4 entry is not a valid mp4 meaning you cant play
it directly
without stripping headers.
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-06-23 20:06
(Received via mailing list)
>> You can use proxy_store with the mp4 module.
So, proxy_store is able to download whole mp4 file once and than server
that file locally without fetching each time from the origin if users
seek
through the video ?
13731c628e20ffd83b6114ff2c406272?d=identicon&s=25 shahzaib mushtaq (shahzaib12)
on 2014-07-25 12:24
(Received via mailing list)
mp4 seeking is filling up disk rapidly on edge server using proxy_cache
and
also incoming bandwidth is always higher than outgoing bandwidth
(nload).
Maybe, people are seeking to mp4 files and full videos are getting
download
again and again.

How can i manage the mp4 seeking on edge server ? will proxy_store
resolve
the issue ?  I really need to find the solution.

Btw, nginx version is 1.6

Regards.



On Mon, Jun 23, 2014 at 11:06 PM, shahzaib shahzaib
<shahzaib.cb@gmail.com>
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.