Image Hosting

Indo_P · October 15, 2010, 9:07am

Hello!

On Thu, Oct 14, 2010 at 01:45:55PM +0000, [email protected] wrote:

It’s almost 400GB striped size.

The question was:

What is stripe size of the RAID ?

See here for details:

Obviously with 73G disks stripe size can’t be more than 73G,
effectively making it simple concatenation:

Using too low stripe size is bad as it forces several disk
accesses for single IO request.

Maxim D.

Indo_P · October 15, 2010, 9:44am

Hi Maxim,

I’m trying your suggestions

Here’s the config that I used right now
worker_processes 10;
worker_rlimit_nofile 50000;
error_log /dev/null;

pid /etc/nginx/logs/nginx.pid;

events {
worker_connections 50000;
use epoll;
}

http {
include mime.types;
default_type application/octet-stream;

    client_header_timeout           1m;
    client_body_timeout             1m;
    client_header_buffer_size       24k;
    large_client_header_buffers     32 256k;
    send_timeout                    1m;
    ignore_invalid_headers          on;
    keepalive_timeout               5;
    output_buffers   1  128k;
    directio  1m;
    server_tokens   off;
    location ~*

^.+.(jpg|jpeg|png|gif|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js|swf)$
{
aio on;
directio 1;
access_log off;
expires max;
}

    location / {
            aio on;
            directio 1;
            try_files $uri $uri/ /index.php?$uri;
    }


    location ~ \.php$ {
            fastcgi_pass   127.0.0.1:9000;
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME $document_root/index.php;
            include        /etc/nginx/conf/fastcgi_params;
    }

From: Maxim D. [email protected]
To: [email protected]
Sent: Thu, October 14, 2010 8:46:57 PM
Subject: Re: Image Hosting

Hello!

On Thu, Oct 14, 2010 at 12:26:59PM +0000, [email protected] wrote:

Actually the image size is arround 100KB each. The server is running in 250Mbps
traffic.

I already described the disk I’m using is scsi 15K RPM in raid 0

Basic tunings you have to apply when serving static which doesn’t
fit into memory are:

If you use sendfile:

Make sure your OS uses appropriate read-ahead for sendfile to
avoid trashing disks with small requests (and seeks). For FreeBSD
8.1+ it should be enough to set read_ahead directive in nginx
config (0.8.18+).

http://sysoev.ru/nginx/docs/http/ngx_http_core_module.html#read_ahead
(in Russian)

Using bigger socket buffers (listen … sndbuf=… in nginx
config) may help too.

If serving large files - make sure you use appropriate
sendfile_max_chunk to avoid blocking nginx worker on disk for too
long.
Consider switching sendfile off if you can’t persuade it to read
large blocks from disk.

If not using sendfile:

Tune output_buffers (again, to avoid trashing disks with small
requests and seeks). Default is 2x32k, which will result in 4
disk requests for 100k file. Changing it to 1x128k would result
in 2x memory usage but 4x less disk requests, this is probably
good thing to do if you are disk-bound.

In both cases:

Using aio may help a lot (“aio sendfile” is only available under
FreeBSD) by adding more concurency to disk load and generally
improving nginx interactivity. Though right now it requires
patches to avoid socket leaks, see here:

http://nginx.org/pipermail/nginx-devel/2010-October/000498.html

Using directio may help to improve disk cache effictiveness by
excluding large files (if you have some) from cache. Though keep
in mind that it disables sendfile so if you generally tune for
sendfile - you may have to apply output_buffers tunings as well.

It’s hard to say anything more than this without knowing lots of
details.

Maxim D.

Indo_P · October 15, 2010, 9:31am

Here’s the output of “iostat -d 60 2”
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 85.58 4947.94 2619.30 593632866 314252346

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 114.41 8395.93 339.01 503840 20344

here’s the output of vmstat
procs -----------memory---------- —swap-- -----io---- -system–
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 1 332 63136 628344 6085040 0 0 617 324 17 27 1 4
88 6

From: Dennis J. [email protected]
To: [email protected]
Sent: Thu, October 14, 2010 7:54:26 PM
Subject: Re: Image Hosting

Ah, sorry I somehow didn’t catch that last line with the actual HW
setup.

If my math skills don’t fail me (and they very well might) the traffic
and
image size data mean that you serve about 300 images per second.
Can you post a few lines of vmstat output and perhaps the output of
“iostat
-d 60 2”?

Also how are the hits distributed across the whole pool of images? Are
these hits truly random or are some images hit significantly more often
than others?

More ram would obviously take some pressure off the disks if they are
really the problem.

Regards,
Dennis

On 10/14/2010 02:26 PM, [email protected] wrote:

Date: Thu, 14 Oct 2010 14:23:26
for pagecache can help here if the cache hit ratio is good but if it is not

 *Subject:* Re: Image Hosting

     we have an image hosting, and of course we're using Nginx!

     Now on we're having problem with disk i/o. With 200GB of images.
     Is there any recommendation for this matter? Nginx configuration or
     Hardware/other software configuration?

     We already used Dual Quad Core 3Ghz, 8 GB RAM, 6x73GB 15k SCSI

server

Indo_P · October 15, 2010, 10:02am

Please re-run the vmstat as “vmstat 10 10”. That should give a better
glimpse at what the system is doing.

Indo_P · October 15, 2010, 9:57am

Have read about it. But not sure… It seems that I haven’t modify the
stripe
size of the raid.

From: Igor S. [email protected]
To: [email protected]
Sent: Thu, October 14, 2010 8:47:00 PM
Subject: Re: Image Hosting

On Thu, Oct 14, 2010 at 01:36:20PM +0000, [email protected] wrote:

The disk are 6x73GB in scsi 15K RPM…

I meant stripe size of the RAID0.
I do not know however, how to see it on Linux.

250Mbps traffic.

nginx Info Page
–
Igor S.
Igor Sysoev

Indo_P · October 15, 2010, 10:10am

the output of iotop
Total DISK READ: 15.67 M/s | Total DISK WRITE: 0.00 B/s

the output of vmstat 10 10
procs -----------memory---------- —swap-- -----io---- -system–
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 1 31736 3518240 486048 2719488 0 1 646 321 70 39 1
4 88 7
0 0 31736 3474616 489792 2763936 0 0 10909 96 12010 2568 1
5 76 17
0 0 31736 3428900 492456 2797516 0 0 11062 152 12099 2546 1
6 78 15
0 1 31736 3382620 494924 2833764 0 0 12728 136 12709 2580 1
7 76 16
0 1 31696 3325956 497404 2872368 4 0 11678 244 12532 2571 2
6 77 15
0 0 31696 3288048 500144 2906464 0 0 9947 178 12187 2539 1
6 80 13
0 1 31684 3250064 503244 2947148 1 0 10628 166 12279 2613 1
7 78 15
0 0 31680 3180800 507244 2995880 0 0 12200 186 12512 2680 1
7 73 19
0 0 31680 3126368 510364 3047976 0 0 12083 149 12641 2738 2
6 70 21
0 1 31680 3068704 513724 3093616 0 0 11916 176 12648 2799 1
7 74 18

From: Dennis J. [email protected]
To: [email protected]
Sent: Thu, October 14, 2010 9:57:26 PM
Subject: Re: Image Hosting

Please re-run the vmstat as “vmstat 10 10”. That should give a better
glimpse at what the system is doing.

On 10/14/2010 04:26 PM, Indo P. wrote:

0 1 332 63136 628344 6085040 0 0 617 324 17 27 1 4 88 6
image size data mean that you serve about 300 images per second.
Regards,

sata disk is going to be a huge problem.
On 10/14/2010 10:33 AM, SplitIce wrote:

Hmm well talking from experience nginx can serve upwards of 5,000 req/s
for
static files if configured right. Ill leave this for Igor or someone more
expericed to reply to.

On Thu, Oct 14, 2010 at 7:28 PM, Indo P.<[email protected]
mailto:[email protected]
<mailto:[email protected] mailto:[email protected]>> wrote:

i tried to use AIO, but it seems my server is getting slower

Indo_P · October 15, 2010, 10:39pm

On Thu, Oct 14, 2010 at 9:26 AM, Indo P. [email protected] wrote:

r b swpd free buff cache si so bi bo in cs us sy id
wa
0 1 332 63136 628344 6085040 0 0 617 324 17 27 1 4 88
6

With 114 transactions (IOPS) per second, and 8396 block reads per
second, that’s about 34 Mbit/s being read from disk (blocks are 512
bytes). Considering you say you’re sending out 250 Mbit/s, I think
you’re getting quite a bit of caching benefit from the OS filesystem
cache, and probably have a disk cache hit ratio above 80%.

Also, 114 IOPS is nothing. A single 15K disk can do 180 IOPS, so six
15K drives in a RAID-0 set should let you do >1000 IOPS.

In short… this doesn’t look like a disk-bound system to me from a
hardware perspective.

What does the upstream network look like? Is it Gigabit though a good
provider? That could very well be the bottleneck. What metric are you
using to determine that it is “getting slower” or that you have a
“problem with disk I/O”?

–
RPM

Indo_P · October 15, 2010, 10:39pm

Hi Ryan,

Actually our ISP is giving us 1Gbit connection. Before, we already
tested trough onother ISP, the connection is up to 900Mbps.
So I assume that’s not a problem with ISP neither.

Indo_P · October 15, 2010, 10:39pm

Hi Dennis,

Sorry I think that word is already mentioned that I have a problem with
disk i/o.

That’s why I’m asking wether I need onother software to be used together
with nginx so I can give a best performance

Indo_P · October 16, 2010, 7:12am

I’ve heard great things about ext4 and xfs.

Sent from my iPod

Indo_P · October 15, 2010, 12:46pm

This looks fairly benign to me. You get lots of random reads so you can
expect the cpu to wait for i/o occasionally.
One remedy is to spread the data across more disks and the other is to
put
more ram into the machine so more data can be cached.

I have a 30TB system using 48 disks and two controllers here where the
I/O
wait is on average 85% yet it delivers the data with fairly low latency.

You never actually mentioned what your real problem is (you just said
“Now
on we’re having problem with disk i/o”.)

Regards,
Dennis

Indo_P · October 16, 2010, 7:12am

This server is already using ext4 filesystem

From: Splitice [email protected]
To: “[email protected]” [email protected]
Sent: Fri, October 15, 2010 12:24:14 PM
Subject: Re: Image Hosting

I’ve heard great things about ext4 and xfs.

Sent from my iPod

Indo_P · October 26, 2010, 11:39pm

Hi

Thread is quite old for reply but maybe this will help

our setup :

we set blockdev of RAID devices to 4096
RAID stripe to 256K
scheduler must be set to “deadline” (on Linux
see /sys/block/sda/queue/scheduler)
xfs used here with these options : noatime,nobarrier
worker_processes should be equals your number of cores, we have 16
disable logging
sendfile_max_chunk 512k;
sendfile off;
tcp_nopush on;
tcp_nodelay on;
output_buffers 1 1m;
keepalive_timeout 0;

With this setup we can do 1.7 GBits/s traffic. We are serving bigger
files so these wont be ideal for your situation but may be helpful. We
have 28TB of data per server.

FYI we do 350 mbytes/s on Treeware/Adaptec SATA RAID. You may try bonnie
++ on this box to see how it can do on plain disk access.

have a nice day

–
Maxime Ducharme
Systems Architect
Techboom Inc

Indo_P · October 16, 2010, 11:28am

But what is your problem exactly with the I/O?
The numbers you posted look fine. What are the symptoms you are seeing?
As Maxim pointed out iowait is fairly irrelevant as long as the requests
get reponded to in a timely manner.

Regards,
Dennis

Indo_P · October 27, 2010, 3:53pm

remount the filesystem with option noatime and nodiratime may help

Posted at Nginx Forum: