# Image Hosting **URL:** \ **Category:** NGINX\ **Created:** [October 14, 2010, 9:37am UTC](https://www.ruby-forum.com/t/image-hosting/197422 "2010-10-14T09:37:10Z")\ **Posts on this page:** 15\ **Page:** 2
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 9:07am UTC](https://www.ruby-forum.com/t/image-hosting/197422/21 "2010-10-15T09:07:38Z")
Hello! On Thu, Oct 14, 2010 at 01:45:55PM +0000, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > It’s almost 400GB striped size. The question was: > > What is stripe size of the RAID ? See here for details: > **[Standard RAID levels | RAID 0](https://en.wikipedia.org/wiki/RAID0#RAID_0)** > > RAID 0 (also known as a stripe set or striped volume) splits ("stripes") data evenly across two or more disks, without parity information, redundancy, or fault tolerance. Since RAID 0 provides no fault tolerance or redundancy, the failure of one drive will cause the entire array to fail, due to data being striped across all disks. This configuration is typically implemented having speed as the intended goal. The probability of the stripe set failing is always higher than that of a single disk, a... > **[Data striping](https://en.wikipedia.org/wiki/Data_striping)** > > In computer data storage, data striping is the technique of segmenting logically sequential data, such as a file, so that consecutive segments are stored on different physical storage devices. Striping is useful when a processing device requests data more quickly than a single storage device can provide it. By spreading segments across multiple devices which can be accessed concurrently, total data throughput is increased. It is also a useful method for balancing I/O load across an array of disks... Obviously with 73G disks stripe size can’t be more than 73G, effectively making it simple concatenation: > **[Non-RAID drive architectures](https://en.wikipedia.org/wiki/Non-RAID_drive_architectures#Concatenation_.28SPAN.2C_BIG.29)** > > The most widespread standard for configuring multiple hard disk drives is RAID (redundant array of inexpensive/independent disks), which comes in a number of standard configurations and non-standard configurations. Non-RAID drive architectures also exist, and are referred to by acronyms with tongue-in-cheek similarity to RAID: JBOD (just a bunch of disks or just a bunch of drives) is an architecture using multiple hard drives exposed as individual devices. Hard drives may be treated independentl... Using too low stripe size is bad as it forces several disk accesses for single IO request. Maxim D. ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 9:44am UTC](https://www.ruby-forum.com/t/image-hosting/197422/22 "2010-10-15T09:44:32Z")
Hi Maxim, I’m trying your suggestions Here’s the config that I used right now worker\_processes 10; worker\_rlimit\_nofile 50000; error\_log /dev/null; pid /etc/nginx/logs/nginx.pid; events { worker\_connections 50000; use epoll; } http { include mime.types; default\_type application/octet-stream; ``` client_header_timeout 1m; client_body_timeout 1m; client_header_buffer_size 24k; large_client_header_buffers 32 256k; send_timeout 1m; ignore_invalid_headers on; keepalive_timeout 5; output_buffers 1 128k; directio 1m; server_tokens off; location ~* ``` ^.+.(jpg|jpeg|png|gif|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js|swf)$ { aio on; directio 1; access\_log off; expires max; } ``` location / { aio on; directio 1; try_files $uri $uri/ /index.php?$uri; } location ~ \.php$ { fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $document_root/index.php; include /etc/nginx/conf/fastcgi_params; } ``` * * * From: Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) To: [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) Sent: Thu, October 14, 2010 8:46:57 PM Subject: Re: Image Hosting Hello! On Thu, Oct 14, 2010 at 12:26:59PM +0000, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Actually the image size is arround 100KB each. The server is running in 250Mbps > traffic. > > I already described the disk I’m using is scsi 15K RPM in raid 0 Basic tunings you have to apply when serving static which doesn’t fit into memory are: If you use sendfile: - Make sure your OS uses appropriate read-ahead for sendfile to avoid trashing disks with small requests (and seeks). For FreeBSD 8.1+ it should be enough to set read\_ahead directive in nginx config (0.8.18+). > **[Модуль ngx\_http\_core\_module](http://nginx.org/ru/docs/http/ngx_http_core_module.html)** (in Russian) Using bigger socket buffers (listen … sndbuf=… in nginx config) may help too. - If serving large files - make sure you use appropriate sendfile\_max\_chunk to avoid blocking nginx worker on disk for too long. - Consider switching sendfile off if you can’t persuade it to read large blocks from disk. If not using sendfile: - Tune output\_buffers (again, to avoid trashing disks with small requests and seeks). Default is 2x32k, which will result in 4 disk requests for 100k file. Changing it to 1x128k would result in 2x memory usage but 4x less disk requests, this is probably good thing to do if you are disk-bound. In both cases: - Using aio may help a lot (“aio sendfile” is only available under FreeBSD) by adding more concurency to disk load and generally improving nginx interactivity. Though right now it requires patches to avoid socket leaks, see here: > **[\[PATCH 0 of 4\] various AIO fixes](http://mailman.nginx.org/pipermail/nginx-devel/2010-October/000498.html)** - Using directio may help to improve disk cache effictiveness by excluding large files (if you have some) from cache. Though keep in mind that it disables sendfile so if you generally tune for sendfile - you may have to apply output\_buffers tunings as well. It’s hard to say anything more than this without knowing lots of details. Maxim D. ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 9:31am UTC](https://www.ruby-forum.com/t/image-hosting/197422/23 "2010-10-15T09:31:25Z")
Here’s the output of “iostat -d 60 2” Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn sda 85.58 4947.94 2619.30 593632866 314252346 Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn sda 114.41 8395.93 339.01 503840 20344 here’s the output of vmstat procs -----------memory---------- —swap-- -----io---- -system– ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 332 63136 628344 6085040 0 0 617 324 17 27 1 4 88 6 * * * From: Dennis J. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) To: [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) Sent: Thu, October 14, 2010 7:54:26 PM Subject: Re: Image Hosting Ah, sorry I somehow didn’t catch that last line with the actual HW setup. If my math skills don’t fail me (and they very well might) the traffic and image size data mean that you serve about 300 images per second. Can you post a few lines of vmstat output and perhaps the output of “iostat -d 60 2”? Also how are the hits distributed across the whole pool of images? Are these hits truly random or are some images hit significantly more often than others? More ram would obviously take some pressure off the disks if they are really the problem. Regards, Dennis On 10/14/2010 02:26 PM, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Date: Thu, 14 Oct 2010 14:23:26 > for pagecache can help here if the cache hit ratio is good but if it is not > > > ``` > > *Subject:* Re: Image Hosting > > > > we have an image hosting, and of course we're using Nginx! > > > > Now on we're having problem with disk i/o. With 200GB of images. > > Is there any recommendation for this matter? Nginx configuration or > > Hardware/other software configuration? > > > > We already used Dual Quad Core 3Ghz, 8 GB RAM, 6x73GB 15k SCSI > > > > ``` server ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 10:02am UTC](https://www.ruby-forum.com/t/image-hosting/197422/24 "2010-10-15T10:02:04Z")
Please re-run the vmstat as “vmstat 10 10”. That should give a better glimpse at what the system is doing. ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 9:57am UTC](https://www.ruby-forum.com/t/image-hosting/197422/25 "2010-10-15T09:57:40Z")
Have read about it. But not sure.. It seems that I haven’t modify the stripe size of the raid. * * * From: Igor S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) To: [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) Sent: Thu, October 14, 2010 8:47:00 PM Subject: Re: Image Hosting On Thu, Oct 14, 2010 at 01:36:20PM +0000, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > The disk are 6x73GB in scsi 15K RPM.. I meant stripe size of the RAID0. I do not know however, how to see it on Linux. > 250Mbps traffic. > > * * * > > [mailman.nginx.org archive](http://nginx.org/mailman/listinfo/nginx) > – > Igor S. > [Igor Sysoev](http://sysoev.ru/en/) ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 10:10am UTC](https://www.ruby-forum.com/t/image-hosting/197422/26 "2010-10-15T10:10:50Z")
the output of iotop Total DISK READ: 15.67 M/s | Total DISK WRITE: 0.00 B/s the output of vmstat 10 10 procs -----------memory---------- —swap-- -----io---- -system– ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 31736 3518240 486048 2719488 0 1 646 321 70 39 1 4 88 7 0 0 31736 3474616 489792 2763936 0 0 10909 96 12010 2568 1 5 76 17 0 0 31736 3428900 492456 2797516 0 0 11062 152 12099 2546 1 6 78 15 0 1 31736 3382620 494924 2833764 0 0 12728 136 12709 2580 1 7 76 16 0 1 31696 3325956 497404 2872368 4 0 11678 244 12532 2571 2 6 77 15 0 0 31696 3288048 500144 2906464 0 0 9947 178 12187 2539 1 6 80 13 0 1 31684 3250064 503244 2947148 1 0 10628 166 12279 2613 1 7 78 15 0 0 31680 3180800 507244 2995880 0 0 12200 186 12512 2680 1 7 73 19 0 0 31680 3126368 510364 3047976 0 0 12083 149 12641 2738 2 6 70 21 0 1 31680 3068704 513724 3093616 0 0 11916 176 12648 2799 1 7 74 18 * * * From: Dennis J. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) To: [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) Sent: Thu, October 14, 2010 9:57:26 PM Subject: Re: Image Hosting Please re-run the vmstat as “vmstat 10 10”. That should give a better glimpse at what the system is doing. On 10/14/2010 04:26 PM, Indo P. wrote: > 0 1 332 63136 628344 6085040 0 0 617 324 17 27 1 4 88 6 > image size data mean that you serve about 300 images per second. > Regards, > > > sata disk is going to be a huge problem. > > On 10/14/2010 10:33 AM, SplitIce wrote: > > > > > Hmm well talking from experience nginx can serve upwards of 5,000 req/s > > > for > > > static files if configured right. Ill leave this for Igor or someone more > > > expericed to reply to. > > > > > > On Thu, Oct 14, 2010 at 7:28 PM, Indo P.\<[removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > > > [mailto:removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > > > \<[mailto:removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) [mailto:removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)\>\> wrote: > > > > > > i tried to use AIO, but it seems my server is getting slower * * * ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 10:39pm UTC](https://www.ruby-forum.com/t/image-hosting/197422/27 "2010-10-15T22:39:56Z")
On Thu, Oct 14, 2010 at 9:26 AM, Indo P. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > r b swpd free buff cache si so bi bo in cs us sy id > wa > 0 1 332 63136 628344 6085040 0 0 617 324 17 27 1 4 88 > 6 With 114 transactions (IOPS) per second, and 8396 block reads per second, that’s about 34 Mbit/s being read from disk (blocks are 512 bytes). Considering you say you’re sending out 250 Mbit/s, I think you’re getting quite a bit of caching benefit from the OS filesystem cache, and probably have a disk cache hit ratio above 80%. Also, 114 IOPS is nothing. A single 15K disk can do 180 IOPS, so six 15K drives in a RAID-0 set should let you do \>1000 IOPS. In short… this doesn’t look like a disk-bound system to me from a hardware perspective. What does the upstream network look like? Is it Gigabit though a good provider? That could very well be the bottleneck. What metric are you using to determine that it is “getting slower” or that you have a “problem with disk I/O”? – RPM ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 10:39pm UTC](https://www.ruby-forum.com/t/image-hosting/197422/28 "2010-10-15T22:39:56Z")
Hi Ryan, Actually our ISP is giving us 1Gbit connection. Before, we already tested trough onother ISP, the connection is up to 900Mbps. So I assume that’s not a problem with ISP neither. ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 10:39pm UTC](https://www.ruby-forum.com/t/image-hosting/197422/29 "2010-10-15T22:39:57Z")
Hi Dennis, Sorry I think that word is already mentioned that I have a problem with disk i/o. That’s why I’m asking wether I need onother software to be used together with nginx so I can give a best performance ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 16, 2010, 7:12am UTC](https://www.ruby-forum.com/t/image-hosting/197422/30 "2010-10-16T07:12:13Z")
I’ve heard great things about ext4 and xfs. Sent from my iPod ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 12:46pm UTC](https://www.ruby-forum.com/t/image-hosting/197422/31 "2010-10-15T12:46:48Z")
This looks fairly benign to me. You get lots of random reads so you can expect the cpu to wait for i/o occasionally. One remedy is to spread the data across more disks and the other is to put more ram into the machine so more data can be cached. I have a 30TB system using 48 disks and two controllers here where the I/O wait is on average 85% yet it delivers the data with fairly low latency. You never actually mentioned what your real problem is (you just said “Now on we’re having problem with disk i/o”.) Regards, Dennis ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 16, 2010, 7:12am UTC](https://www.ruby-forum.com/t/image-hosting/197422/32 "2010-10-16T07:12:13Z")
This server is already using ext4 filesystem * * * From: Splitice [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) To: “removed\_email\_address@domain.invalid” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) Sent: Fri, October 15, 2010 12:24:14 PM Subject: Re: Image Hosting I’ve heard great things about ext4 and xfs. Sent from my iPod ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 26, 2010, 11:39pm UTC](https://www.ruby-forum.com/t/image-hosting/197422/33 "2010-10-26T23:39:31Z")
Hi Thread is quite old for reply but maybe this will help our setup : - we set blockdev of RAID devices to 4096 - RAID stripe to 256K - scheduler must be set to “deadline” (on Linux see /sys/block/sda/queue/scheduler) - xfs used here with these options : noatime,nobarrier - worker\_processes should be equals your number of cores, we have 16 - disable logging - sendfile\_max\_chunk 512k; - sendfile off; - tcp\_nopush on; - tcp\_nodelay on; - output\_buffers 1 1m; - keepalive\_timeout 0; With this setup we can do 1.7 GBits/s traffic. We are serving bigger files so these wont be ideal for your situation but may be helpful. We have 28TB of data per server. FYI we do 350 mbytes/s on Treeware/Adaptec SATA RAID. You may try bonnie ++ on this box to see how it can do on plain disk access. have a nice day – Maxime Ducharme Systems Architect Techboom Inc ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 16, 2010, 11:28am UTC](https://www.ruby-forum.com/t/image-hosting/197422/34 "2010-10-16T11:28:42Z")
But what is your problem exactly with the I/O? The numbers you posted look fine. What are the symptoms you are seeing? As Maxim pointed out iowait is fairly irrelevant as long as the requests get reponded to in a timely manner. Regards, Dennis ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 27, 2010, 3:53pm UTC](https://www.ruby-forum.com/t/image-hosting/197422/35 "2010-10-27T15:53:51Z")
remount the filesystem with option noatime and nodiratime may help Posted at Nginx Forum: > **[Re: Image Hosting](https://forum.nginx.org/read.php?2,140436,144961#msg-144961)** > > hi there, we have an image hosting, and of course we're using Nginx! Now on we're having problem with disk i/o. With 200GB of images. Is there any recommendation for this matter? Nginx configuration or Hardware/other software configuration? We alre [Previous page](https://www.ruby-forum.com/t/image-hosting/197422.md?page=1)