Using nginx to cache file serving

Hi
We are using nginx & SATA disks on several of our file servers.However
after
hitting 12MBytes/s of port speed (server nic is a 1gbps), the server
gets
down to it’s knees. SSH becomes very slow/unresponsive and our users
complain about slow speeds and dropped connections. Our
users/downloaders
have very slow speed (like 512kbits/s) so there are lots of users
downloading from each server. I thought the problem was from using SATA
disk
and the bottleneck was the HDD IO speed.So we ordered a new server with
SAS
hdds & RAID0, the performance has been great and our server is serving
about
20MBytes/s, (same provider)
However I’m still not satisfied.
1.Is there any configuration options I’m missing? I’m using the default
config.
2. Both servers rams are fully utilized and the memory is 100% filled,
When
I stop nginx the memory is still used up but why? (there is no other
service
running on the server)
If this is the nginx cache, why it’s not emptied when I exit/kill nginx
processes?
3. My most important question is, Is it possible/wise that I use the SAS
server as a rev-proxy/cache to serve the downloads from the SATA server?
My
idea is when a user connects to the SAS server to download a file, The
SAS
server requests the file from the SATA server(using the internal
connection
in datacenter), caches it on it’s fast disk, then serves the file to my
slow
downloader. This way the SATA HDD doesn’t have to seek to the file
location
each time and since there are lots of downloaders this happens a lot.

Sorry for the long message and thank you for your time
Mojiz

Posted at Nginx Forum:

  1. Depends a lot on your environment. If you are sure you are I/O bound,
    there isn’t too much you can tweak.

  2. How are you measuring this? It’s normal for there to be very little
    “free” memory on Linux due to buffers and filesystem caches. Look at the
    -/+ buffers/cache line for your available memory. If you are actually
    running out of memory and hitting swap, this is likely why performance
    is
    so low.

  3. Take a look at ngx_slowfs_cache, it sounds like it might fit the bill

http://labs.frickle.com/nginx_ngx_slowfs_cache

Rich.

Thanks for the answeres.
about the memory the problem is I’m hitting this on only some of my
servers
but not all, one of my servers has only 3GB memory which 1G is used but
on
another 16GB/16GB is used.
The ngx_slowfs_cache module is usefull when the server has 2 type disks,
SATA and SAS, but I have two servers which I want to do the proxying.
I’m looking at the included proxy module
something like this:

proxy_buffers 10240 128k; #needs 1.2GB memory
proxy_cache_path /dev/sas_disk/cache;
proxy_max_temp_file_size 2048m;

proxy_store is tempting:

proxy_store on;
proxy_temp_path /dev/sas_disk/temp;
root /dev/sas_disk/www;

is this what i’m looking for?

Posted at Nginx Forum: