Let's build nginx server with caching !?

Nati_A · November 28, 2007, 5:43pm

Hi.

I’m currently using squid for cahcing images (CDN like) from my website
to
serv them from different server’s,
the probelm with squid it’s using lot’s of CPU usage,
can nginx do the same was squid does, storing data localy, so it won’t
need
to communicate with the origin server all the time.

Regards,

Nati_A · November 28, 2007, 7:44pm

Hi,

On Mit 28.11.2007 18:31, Nati A. wrote:

Hi.

I’m currently using squid for cahcing images (CDN like) from my website
to serv them from different server’s, the probelm with squid it’s using
lot’s of CPU usage, can nginx do the same was squid does, storing data
localy, so it won’t need to communicate with the origin server all the
time.

I’am sure you know this but to understand your question right.

Squid is a forwarding proxy
nginx is a webserver, with possibility to forward requests to
proxy-backends

You now want to use the nginx ‘framework’ to build a forwarding proxy as
a separate module like http and mail, is this right?

Yes I think this is a nice idea and a lot of work

Cheers

Aleks

Nati_A · November 28, 2007, 11:18pm

On Nov 28, 2007 7:30 PM, Aleksandar L. [email protected] wrote:

Squid is a forwarding proxy
nginx is a webserver, with possibility to forward requests to
proxy-backends

You now want to use the nginx ‘framework’ to build a forwarding proxy as
a separate module like http and mail, is this right?

i think he wants nginx to cache certain things (ie images), so nginx
would’t forward the request to proxy backend but serve the local copy
(and save bandwidth).

Nati_A · November 29, 2007, 2:05am

isn’t that this

serve static files directly

location ~* ^.+.(jpg|jpeg|gif|png)$ {
root /your/public/folder;
access_log on;
expires 30d;
}

Almir K. [email protected] wrote: On Nov 28, 2007 7:30 PM,
Aleksandar L. wrote:

Squid is a forwarding proxy
nginx is a webserver, with possibility to forward requests to
proxy-backends

You now want to use the nginx ‘framework’ to build a forwarding proxy as
a separate module like http and mail, is this right?

i think he wants nginx to cache certain things (ie images), so nginx
would’t forward the request to proxy backend but serve the local copy
(and save bandwidth).

–
almir

Nati_A · November 29, 2007, 7:36am

On Wed, Nov 28, 2007 at 11:01:43PM +0100, Almir K. wrote:

would’t forward the request to proxy backend but serve the local copy
(and save bandwidth).

Right now I’m doing cache.

Nati_A · November 29, 2007, 7:36am

On Wed, Nov 28, 2007 at 04:58:14PM -0800, Eire A. wrote:

isn’t that this

serve static files directly

location ~* ^.+.(jpg|jpeg|gif|png)$ {
root /your/public/folder;
access_log on;
expires 30d;
}

No.

Nati_A · November 30, 2007, 8:36am

nginx has a mirror module named proxy_store,I think that it will be
a uesful method for your CDN works.

2007/11/29, Nati A. [email protected]:

Nati_A · December 23, 2007, 10:49pm

chen changliang wrote:

nginx has a mirror module named proxy_store,I think that it will be
a uesful method for your CDN works.

2007/11/29, Nati A. [email protected]:

I’ve tried to use the proxy_store feature to cache flv files on the
nginx server,
For some reason most of the files are corrupted, when I compared the
original files to the cached I found the cached corrupted files are
little smaller (some of the more and some less smaller) then the
original.

The connection between the servers is fast and there is no congestion on
the line.
I tried to upgrade to nginx 0.6 (latest) â€“ didn’t help.

When I cache small images everything goes well. The FLV files size
varies from 1mb to 20mb , the small (1mb) files also have the same
problem.

At the end, the player cannot play those files at all, what I do now is
running rsync between the servers, it solves me the problem but actually
its not a good solution because I need new added flv files to work
instantly without waiting for the rsync schedule.

What do you think?

Nati_A · December 24, 2007, 3:35am

Try disabling the ngx_http_flv_module or the proxy_buffering £¿

2007/12/24, Dmitry S. [email protected]:

Nati_A · December 26, 2007, 2:59pm

On Wed, Dec 26, 2007 at 12:58:22PM +0100, Dmitry S. wrote:

Changliang Chen wrote:

Try disabling the ngx_http_flv_module or the proxy_buffering ???

2007/12/24, Dmitry S. [email protected]:

when i disabled the proxy_buffering it stopped the local download of the
cache totally. and the proxy temp dir was also empty.

“proxy_buffering off” does not store response at all.

Nati_A · December 26, 2007, 12:58pm

Changliang Chen wrote:

Try disabling the ngx_http_flv_module or the proxy_buffering ï¿½ï¿½

2007/12/24, Dmitry S. [email protected]:

Hi,
when i disabled the proxy_buffering it stopped the local download of the
cache totally. and the proxy temp dir was also empty.
Here is my configuration file:

bash-2.05b# cat nginx.conf
#user nobody;
worker_processes 1;

error_log logs/error.log;
pid logs/nginx.pid;
events {

2048 old

worker_connections 2048;

use rtsig;

}
http {
include mime.types;
default_type application/octet-stream;

default_type video/x-flv;

access_log off;
sendfile on;
tcp_nopush on;
tcp_nodelay on;

keepalive_timeout 65;

connection_pool_size 2k;

request_pool_size 4k;

output_buffers 10 128k;

#gzip on;
server {
listen 82.166.13.202:80;
server_name jugy.com www.jugy.com;

access_log logs/host.access.log main;

    # Main location
    location / {
        proxy_pass         http://84.94.232.33:80/;
        proxy_redirect off;
        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For

$proxy_add_x_forwarded_for;

        client_max_body_size       10m;
        client_body_buffer_size    128k;

        proxy_connect_timeout      90;
        proxy_send_timeout         90;
        proxy_read_timeout         90;

        proxy_buffer_size          4k;
        proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;
    }

location /flvideo/ {
root /usr/local/nginx/data/www;
open_file_cache_errors off;
error_page 404 = /fetch$uri;

}
location /images/ {
root /usr/local/nginx/data/www;
open_file_cache_errors off;
error_page 404 = /fetch$uri;

}
location /thumb/ {
root /usr/local/nginx/data/www;
open_file_cache_errors off;
error_page 404 = /fetch$uri;

}
location /fetch/ {
internal;
proxy_pass http://84.94.232.33:80/;
proxy_store on;

proxy_buffering off;

    proxy_temp_path         /usr/local/nginx/data/temp;
    alias /usr/local/nginx/data/www/;
    }

}
}

Nati_A · December 26, 2007, 4:29pm

Igor S. wrote:

On Wed, Dec 26, 2007 at 12:58:22PM +0100, Dmitry S. wrote:

Changliang Chen wrote:

Try disabling the ngx_http_flv_module or the proxy_buffering ???

2007/12/24, Dmitry S. [email protected]:

when i disabled the proxy_buffering it stopped the local download of the
cache totally. and the proxy temp dir was also empty.

“proxy_buffering off” does not store response at all.

I commented it out, now i have came back to my problem with the media
files,
I don’t think that the flv_module has something to do with it, I do not
use it in my configuration file.

Most of the files that stored by proxy_store are still
corrupted/incomplete.