Forum: NGINX nginx + caching of static files

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
D0a98f6304ff122ac9b487ff8275d8bb?d=identicon&s=25 Payam Chychi (Guest)
on 2009-04-07 07:26
(Received via mailing list)
Hi Guys,

I was hoping someone could point out a document where it can teach me
how to setup nginx + caching. I know that traditionally nginx did not
have a native caching and it required you to run ncache or cachemem in
order to perform caching of static content... but does nginx now allow
basic static file caching without the use of any other 3rd app?

Im trying to understand if nginx now has a native caching which would
allow caching of static content. I dont want to get fancy in anyway, I
simply want to allocate say 200mb of storage space to allow caching of
static content. the 200mb should be updated with the most
active/recently seen file request.

reason? im trying to allow nginx to serve the static files from disk
(if it has seen it previously) and not have to go back to the origin
server every single time.

Thanks
Payam
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2009-04-07 07:40
(Received via mailing list)
On Mon, Apr 06, 2009 at 10:14:36PM -0700, Payam Chychi wrote:

> simply want to allocate say 200mb of storage space to allow caching of
> static content. the 200mb should be updated with the most
> active/recently seen file request.
>
> reason? im trying to allow nginx to serve the static files from disk
> (if it has seen it previously) and not have to go back to the origin
> server every single time.

Get the lastest nginx (0.7.50) and use

    proxy_cache_path  /path/to/cache  levels=1:2
                      keys_zone=one:10m
                      inactive=7d  max_size=200m;

    proxy_temp_path   /path/to/temp;  # must be on the same filesystem
                                      # as cache

    server {

        location / {
            proxy_pass              http://backend;

            proxy_cache             one;
            proxy_cache_key         backend$request_uri;
            proxy_cache_valid       200  1h;
            proxy_cache_use_stale   error timeout invalid_header;
        }
D0a98f6304ff122ac9b487ff8275d8bb?d=identicon&s=25 Payam Chychi (Guest)
on 2009-04-07 07:53
(Received via mailing list)
2009/4/6 Igor Sysoev <is@rambler-co.ru>:
>> Im trying to understand if nginx now has a native caching which would
>
>            proxy_pass              http://backend;
> Igor Sysoev
> http://sysoev.ru/en/
>
>
 proxy_cache_path  /path/to/cache  levels=1:2
                     keys_zone=one:10m
                     inactive=7d  max_size=200m;

   proxy_temp_path   /path/to/temp;  # must be on the same filesystem
                                     # as cache

   server {

       location / {
           proxy_pass              http://backend;

           proxy_cache             one;
           proxy_cache_key         backend$request_uri;
           proxy_cache_valid       200  1h;
           proxy_cache_use_stale   error timeout invalid_header;
       }



few questions:
1- what does the keys_zone used for
2- what does the "200 1h" in proxy_cache_valid mean
3- how is the "proxy_cache_use_stale  error timeout invalid_header"
used?


thanks again Igor,
-Payam
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2009-04-07 08:05
(Received via mailing list)
On Mon, Apr 06, 2009 at 10:46:24PM -0700, Payam Chychi wrote:

> >>
> > Get the lastest nginx (0.7.50) and use
> > š š š šlocation / {
> > --
>
>
>
>
> few questions:
> 1- what does the keys_zone used for
> 2- what does the "200 1h" in proxy_cache_valid mean
> 3- how is the "proxy_cache_use_stale  error timeout invalid_header" used?

http://wiki.nginx.org/NginxHttpProxyModule#proxy_cache
D0a98f6304ff122ac9b487ff8275d8bb?d=identicon&s=25 Payam Chychi (Guest)
on 2009-04-07 08:11
(Received via mailing list)
thank you sir =) you are a king!

-Payam

2009/4/6 Igor Sysoev <is@rambler-co.ru>:
D0a98f6304ff122ac9b487ff8275d8bb?d=identicon&s=25 Payam Chychi (Guest)
on 2009-04-07 10:03
(Received via mailing list)
On Mon, Apr 6, 2009 at 11:03 PM, Payam Chychi <pchychi@gmail.com> wrote:
>>> >> Hi Guys,
>>> >> static content. the 200mb should be updated with the most
>>> >                      inactive=7d  max_size=200m;
>>> >            proxy_cache_key         backend$request_uri;
>>> >
>>>            proxy_pass              http://backend;
>>> 1- what does the keys_zone used for
>>
>
>
>
> --
> Payam Tarverdyan Chychi
> Network Security Specialist / Network Engineer
>


Igor,

using the caching info  ... i get a "redirect loop" when trying to
download  domain.com/images/spacer.gif  when the file does not exist
on the origin server ... ive also been able replicate this with YSlow
for firefox

When bypassing the proxy/cache and connecting directly to the origin,
i get the proper 404 error
any ideas?



Thanks
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2009-04-07 10:12
(Received via mailing list)
On Tue, Apr 07, 2009 at 12:55:37AM -0700, Payam Chychi wrote:

> >>> >
> >>> >> simply want to allocate say 200mb of storage space to allow caching of
> >>> > š š š š š š š š š š škeys_zone=one:10m
> >>> > š š š š š šproxy_cache š š š š š š one;
> >>> >
> >>> š š š šlocation / {
> >>> few questions:
> >>
> Igor,
>
> using the caching info  ... i get a "redirect loop" when trying to
> download  domain.com/images/spacer.gif  when the file does not exist
> on the origin server ... ive also been able replicate this with YSlow
> for firefox
>
> When bypassing the proxy/cache and connecting directly to the origin,
> i get the proper 404 error
> any ideas?

Probably you have something like

       error_page  404  /404.html;
       proxy_intercept_errors  on;
       recursive_error_pages   on;

without

       location = /404.html {

so the /404.html is proxied to backend again.

Also you may cache 404 errors too:

       proxy_cache_valid    200 1h;
       proxy_cache_valid    404 5m;
D0a98f6304ff122ac9b487ff8275d8bb?d=identicon&s=25 Payam Chychi (Guest)
on 2009-04-07 11:17
(Received via mailing list)
2009/4/7 Igor Sysoev <is@rambler-co.ru>:
>> >>> 2009/4/6 Igor Sysoev <is@rambler-co.ru>:
>> >>> >> Im trying to understand if nginx now has a native caching which would
>> >>> >
>> >>> >            proxy_pass              http://backend;
>> >>> > Igor Sysoev
>> >>>    server {
>> >>>
>> >> Igor Sysoev
>>
>> any ideas?
>
> http://sysoev.ru/en/
>
>

Right on... that was the problem. fixed with adding exactly what you
said... im now also caching 404

thanks again
-payam
0e9ed31bf8241b80e384061cce657aa5?d=identicon&s=25 pf shineyear (Guest)
on 2009-04-07 11:40
(Received via mailing list)
www.ncache.org
an nginx cache module

maybe this can help u .
Eb30a9fc33f5e898d8129392550e38b3?d=identicon&s=25 伍鹏 (Guest)
on 2009-04-08 05:05
(Received via mailing list)
Dear Igor,

Then, how to use proxy_cache WITH flv streaming?
I tried proxy_cache feature, but it seems that the flv streaming
function does not work with proxy_cache.

Thanks

Best Regard,

Peng Wu
2009-04-08


On Mon, Apr 06, 2009 at 10:14:36PM -0700, Payam Chychi wrote:

> simply want to allocate say 200mb of storage space to allow caching of
> static content. the 200mb should be updated with the most
> active/recently seen file request.
>
> reason? im trying to allow nginx to serve the static files from disk
> (if it has seen it previously) and not have to go back to the origin
> server every single time.

Get the lastest nginx (0.7.50) and use

    proxy_cache_path  /path/to/cache  levels=1:2
                      keys_zone=one:10m
                      inactive=7d  max_size=200m;

    proxy_temp_path   /path/to/temp;  # must be on the same filesystem
                                      # as cache

    server {

        location / {
            proxy_pass              http://backend;

            proxy_cache             one;
            proxy_cache_key         backend$request_uri;
            proxy_cache_valid       200  1h;
            proxy_cache_use_stale   error timeout invalid_header;
        }
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2009-04-08 13:19
(Received via mailing list)
On Wed, Apr 08, 2009 at 10:51:59AM +0800, ???? wrote:

> Dear Igor,
>
> Then, how to use proxy_cache WITH flv streaming?
> I tried proxy_cache feature, but it seems that the flv streaming function does not work 
with proxy_cache.

No, flv is not filter, but handler, thefore it's not possible.
52ec6f2d0206837902876df1c72e906d?d=identicon&s=25 Tomasz Pajor (Guest)
on 2009-04-08 13:57
(Received via mailing list)
shinepf

ncache will only compile with versions up to 0.7.43.
with newer once there is a compile error because of the change in
ngx_http_cache_s structure.

are You going to fix anytime soon?


--
Best Regards
Tomasz Pajor
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2009-04-09 06:33
(Received via mailing list)
On Wed, Apr 08, 2009 at 11:16:30AM +0000, Glen Lumanau wrote:

> How about jpg, css, js or any other files?

No problems with these files. Actually, there are no problems with flv
itself. The issue is when you want to get /some.flv?start=5000.
However, it seems flv handler can be converted easy enough to filter.
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2009-04-09 08:51
(Received via mailing list)
Hello!

On Wed, Apr 08, 2009 at 03:18:18PM +0400, Igor Sysoev wrote:

> On Wed, Apr 08, 2009 at 11:16:30AM +0000, Glen Lumanau wrote:
>
> > How about jpg, css, js or any other files?
>
> No problems with these files. Actually, there are no problems with flv
> itself. The issue is when you want to get /some.flv?start=5000.
> However, it seems flv handler can be converted easy enough to filter.

I wrote generic bytes filter some time ago, and as far as I
understand it may be used for .flv streaming with some trivial
rewrites.  I've never tried though.

http://mdounin.ru/hg/ngx_http_bytes_filter_module/
http://mdounin.ru/hg/ngx_http_bytes_filter_module/...

Maxim Dounin
Ef5effe1ff64e4ab2c0bb98137e99d3c?d=identicon&s=25 Resicow (Guest)
on 2009-04-09 21:35
(Received via mailing list)
Hi Igor (and everyone else),

I had a few quick questions about the new proxy_cache feature, and I
apologize if they have already been answered.

1) What about query_strings? Does proxy_cache ignore or include
query_strings? Is there a way to have it ignore query_strings in the
URL?
2) How scalable is this system? Could it handle 50 million different
URIs for instance?
3) Any plans for purge capabilities somehow, in the event you need the
content flushed right away?

Keep up the great work, thanks for making the world's best server!

Thanks,

John
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2009-04-10 08:52
(Received via mailing list)
On Thu, Apr 09, 2009 at 02:27:15PM -0500, Resicow wrote:

> Hi Igor (and everyone else),
>
> I had a few quick questions about the new proxy_cache feature, and I
> apologize if they have already been answered.
>
> 1) What about query_strings? Does proxy_cache ignore or include
> query_strings? Is there a way to have it ignore query_strings in the URL?

By default qeury strings are included. You may ignore them using
something
like this:

    proxy_cache_key   $host/$uri";

The default proxy_cache_key approximates

    proxy_cache_key  $scheme$proxy_host$uri$is_args$args;

> 2) How scalable is this system? Could it handle 50 million different
> URIs for instance?

Each key takes 128 bytes in shared memory zone. Therefore you need 6.1G
zone:

   key_zone=NAME:6200m;

> 3) Any plans for purge capabilities somehow, in the event you need the
> content flushed right away?

Yes, there are such plans.
Ef5effe1ff64e4ab2c0bb98137e99d3c?d=identicon&s=25 Resi Cow (Guest)
on 2009-04-10 17:21
(Received via mailing list)
Thanks,

That helps me understand how hash tables work.

Question: Is this for proxy_cache:
Bugfix: now nginx takes into account the "X-Accel-Expires", "Expires",
and
"Cache-Control" header lines in a backend response.

Or are files always cached for time specified in proxy_cache_valid?

Purging would be amazing, that way I could set proxy_cache_valid to a
day or
more, and know that if a change was made I could just issue a purge for
that
specific key, or key path.

Any plans for nginx to manage the cache by doing a HEAD request looking
for
changed content automatically on the origin at certain intervals?

Thanks!

John






2009/4/10 Igor Sysoev <is@rambler-co.ru>
2974d09ac2541e892966b762aad84943?d=identicon&s=25 yangguangli (Guest)
on 2009-04-11 03:57
(Received via mailing list)
汉语能看懂不?

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,869,959#msg-959
5640e332954fc0006aea97a155ce0afd?d=identicon&s=25 Igor Sysoev (Guest)
on 2009-04-11 06:00
(Received via mailing list)
On Fri, Apr 10, 2009 at 04:48:25AM -0400, yangguangli wrote:

> ?????????????????????

No.
Ca88eb9481b38f1e4e202bdf34efb676?d=identicon&s=25 Kirill A. Korinskiy (Guest)
on 2009-04-11 06:09
(Received via mailing list)
At Fri, 10 Apr 2009 04:48:25 -0400,
"yangguangli" <nginx-forum@nginx.us> wrote:
>
> 汉语能看懂不?
>

Yes, chines don't understand lot of peoples from this maillist. Please
use english.
2974d09ac2541e892966b762aad84943?d=identicon&s=25 xs23933 (Guest)
on 2009-05-27 03:30
(Received via mailing list)
我能看懂 汉语哈哈哈

I can read chinese!

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,869,2375#msg-2375
This topic is locked and can not be replied to.