# Image Hosting **URL:** \ **Category:** NGINX\ **Created:** [October 14, 2010, 9:37am UTC](https://www.ruby-forum.com/t/image-hosting/197422 "2010-10-14T09:37:10Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![okkezSS](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/o/85f322/32.png) [@okkezSS](https://www.ruby-forum.com/u/okkezSS)\ **Post date:** [October 14, 2010, 9:37am UTC](https://www.ruby-forum.com/t/image-hosting/197422/1 "2010-10-14T09:37:10Z")
hi there, we have an image hosting, and of course we’re using Nginx! Now on we’re having problem with disk i/o. With 200GB of images. Is there any recommendation for this matter? Nginx configuration or Hardware/other software configuration? We already used Dual Quad Core 3Ghz, 8 GB RAM, 6x73GB 15k SCSI server ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 9:48am UTC](https://www.ruby-forum.com/t/image-hosting/197422/2 "2010-10-14T09:48:26Z")
should be able to use a cache with either nginx+caching or Squid Cache. 2010/10/14 Indo P. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 9:49am UTC](https://www.ruby-forum.com/t/image-hosting/197422/3 "2010-10-14T09:49:04Z")
Try any of these: 1. Compile nginx with AIO support 2. Try directio option 3. Try sendfile option ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 10:29am UTC](https://www.ruby-forum.com/t/image-hosting/197422/4 "2010-10-14T10:29:49Z")
i tried to use AIO, but it seems my server is getting slower * * * From: SplitIce [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) To: removed\_email\_address@domain.invalid Sent: Thu, October 14, 2010 2:47:55 PM Subject: Re: Image Hosting Try any of these: 1. Compile nginx with AIO support 2. Try directio option 3. Try sendfile option On Thu, Oct 14, 2010 at 6:36 PM, Indo P. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: hi there, ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 10:34am UTC](https://www.ruby-forum.com/t/image-hosting/197422/5 "2010-10-14T10:34:51Z")
Hmm well talking from experience nginx can serve upwards of 5,000 req/s for static files if configured right. Ill leave this for Igor or someone more expericed to reply to. ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 9:50am UTC](https://www.ruby-forum.com/t/image-hosting/197422/6 "2010-10-14T09:50:56Z")
If its just serving static files nginx should perform better without caching, unless they are calling PHP for all images (in which case thats the root of their performance problem) ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 11:09am UTC](https://www.ruby-forum.com/t/image-hosting/197422/7 "2010-10-14T11:09:20Z")
1. Install the status module and get more infomation 2. Turn down keep alive, its good but if you are keeping that number of connections open then its bound to hit problems. Also probably send the core part of your nginx config might be some optimisations there. ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 10:49am UTC](https://www.ruby-forum.com/t/image-hosting/197422/8 "2010-10-14T10:49:18Z")
Here’s the output of established connection # netstat -n | grep :80 | grep ESTABLISHED | wc -l 20676 * * * From: SplitIce [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) To: [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) Sent: Thu, October 14, 2010 3:33:45 PM Subject: Re: Image Hosting Hmm well talking from experience nginx can serve upwards of 5,000 req/s for static files if configured right. Ill leave this for Igor or someone more expericed to reply to. On Thu, Oct 14, 2010 at 7:28 PM, Indo P. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: i tried to use AIO, but it seems my server is getting slower > * * * From: SplitIce [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 2:28pm UTC](https://www.ruby-forum.com/t/image-hosting/197422/9 "2010-10-14T14:28:24Z")
Actually the image size is arround 100KB each. The server is running in 250Mbps traffic. I already described the disk I’m using is scsi 15K RPM in raid 0 Sent from my BlackBerry powered by Sinyal Kuat INDOSAT ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 2:24pm UTC](https://www.ruby-forum.com/t/image-hosting/197422/10 "2010-10-14T14:24:39Z")
But if those 5,000 req/s each hit a different 1mb image then you need to read 5gb of data from you storage which if your storage is just a plain old sata disk is going to be a huge problem. Unfortunately Indo P. is not providing nearly enough information to give any sort of advice. If he is lucky then putting lots of ram in the machine for pagecache can help here if the cache hit ratio is good but if it is not then he probably has to distribute the I/O across more spindles and go for a raid with lots of disks. Regards, Dennis ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 7:39pm UTC](https://www.ruby-forum.com/t/image-hosting/197422/11 "2010-10-14T19:39:43Z")
On 10/14/2010 09:36 AM, Indo P. wrote: > hi there, > > we have an image hosting, and of course we’re using Nginx! > > Now on we’re having problem with disk i/o. With 200GB of images. > Is there any recommendation for this matter? Nginx configuration or > Hardware/other software configuration? > > We already used Dual Quad Core 3Ghz, 8 GB RAM, 6x73GB 15k SCSI server Hi there, I have something kinda image hosting, almost 220G (651k files, images). Since most users getting quite random content, there is no way cache most used files in ram. We got best performances with ext4 and bfq io sched, Whole system is gentoo-based etc… so I think you may want try optimize it on OS level. Dual Core Xeon 2,5GHz, 4G ram and standalone raid10 array, two nginx workers. – Piotr. ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 14, 2010, 3:06pm UTC](https://www.ruby-forum.com/t/image-hosting/197422/12 "2010-10-14T15:06:41Z")
On Thu, Oct 14, 2010 at 12:36:05AM -0700, Indo P. wrote: > hi there, > > we have an image hosting, and of course we’re using Nginx! > > Now on we’re having problem with disk i/o. With 200GB of images. > Is there any recommendation for this matter? Nginx configuration or > Hardware/other software configuration? > > We already used Dual Quad Core 3Ghz, 8 GB RAM, 6x73GB 15k SCSI server What OS do you use ? Try to increase number of worker\_proceses, for example, to 20. – Igor S. > **[Igor Sysoev](http://sysoev.ru/en/)** ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 3:12am UTC](https://www.ruby-forum.com/t/image-hosting/197422/13 "2010-10-15T03:12:02Z")
What a great result. Could you please be kind to share your configuration? ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 7:57am UTC](https://www.ruby-forum.com/t/image-hosting/197422/14 "2010-10-15T07:57:35Z")
Ah, sorry I somehow didn’t catch that last line with the actual HW setup. If my math skills don’t fail me (and they very well might) the traffic and image size data mean that you serve about 300 images per second. Can you post a few lines of vmstat output and perhaps the output of “iostat -d 60 2”? Also how are the hits distributed across the whole pool of images? Are these hits truly random or are some images hit significantly more often than others? More ram would obviously take some pressure off the disks if they are really the problem. Regards, Dennis ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 7:45am UTC](https://www.ruby-forum.com/t/image-hosting/197422/15 "2010-10-15T07:45:41Z")
> Actually the image size is arround 100KB each. The server is running in > 250Mbps traffic. > I already described the disk I’m using is scsi 15K RPM in raid 0 So do you see iowait (by running ‘iostat’ or ‘top’) which could mean that the bottleneck is disk system (and then the only way to improve the situation is either by getting more disks or adding memory for caching (either just by pure linux vm file cache or adding some memory only proxies like varnish)). Usually its good to try other servers for comparison like apache or lighttpd - if the default configurations show the same results its not the webserver at fault and therefore nothing wrong with nginx config. But still its too few data (for example nginx version / configuration (maybe too less workers) / some IO metrics (filesystem version (ext, xfs .. ), file atributes (directory structure)) / network load) to give any solution or hints - in short be more detailed about the problem you have. rr ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 8:08am UTC](https://www.ruby-forum.com/t/image-hosting/197422/16 "2010-10-15T08:08:16Z")
On Thu, Oct 14, 2010 at 12:26:59PM +0000, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Actually the image size is arround 100KB each. The server is running in 250Mbps > traffic. > I already described the disk I’m using is scsi 15K RPM in raid 0 What is stripe size of the RAID ? – Igor S. > **[Igor Sysoev](http://sysoev.ru/en/)** ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 8:51am UTC](https://www.ruby-forum.com/t/image-hosting/197422/17 "2010-10-15T08:51:59Z")
On Thu, Oct 14, 2010 at 01:36:20PM +0000, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > The disk are 6x73GB in scsi 15K RPM.. I meant stripe size of the RAID0. I do not know however, how to see it on Linux. > > I already described the disk I’m using is scsi 15K RPM in raid 0 > > [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > > [mailman.nginx.org archive](http://nginx.org/mailman/listinfo/nginx) > > * * * > > nginx mailing list > [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > [mailman.nginx.org archive](http://nginx.org/mailman/listinfo/nginx) – Igor S. > **[Igor Sysoev](http://sysoev.ru/en/)** ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 8:52am UTC](https://www.ruby-forum.com/t/image-hosting/197422/18 "2010-10-15T08:52:37Z")
Hello! On Thu, Oct 14, 2010 at 12:26:59PM +0000, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Actually the image size is arround 100KB each. The server is running in 250Mbps > traffic. > I already described the disk I’m using is scsi 15K RPM in raid 0 Basic tunings you have to apply when serving static which doesn’t fit into memory are: If you use sendfile: - Make sure your OS uses appropriate read-ahead for sendfile to avoid trashing disks with small requests (and seeks). For FreeBSD 8.1+ it should be enough to set read\_ahead directive in nginx config (0.8.18+). > **[Модуль ngx\_http\_core\_module](http://nginx.org/ru/docs/http/ngx_http_core_module.html)** (in Russian) Using bigger socket buffers (listen … sndbuf=… in nginx config) may help too. - If serving large files - make sure you use appropriate sendfile\_max\_chunk to avoid blocking nginx worker on disk for too long. - Consider switching sendfile off if you can’t persuade it to read large blocks from disk. If not using sendfile: - Tune output\_buffers (again, to avoid trashing disks with small requests and seeks). Default is 2x32k, which will result in 4 disk requests for 100k file. Changing it to 1x128k would result in 2x memory usage but 4x less disk requests, this is probably good thing to do if you are disk-bound. In both cases: - Using aio may help a lot (“aio sendfile” is only available under FreeBSD) by adding more concurency to disk load and generally improving nginx interactivity. Though right now it requires patches to avoid socket leaks, see here: > **[\[PATCH 0 of 4\] various AIO fixes](http://mailman.nginx.org/pipermail/nginx-devel/2010-October/000498.html)** - Using directio may help to improve disk cache effictiveness by excluding large files (if you have some) from cache. Though keep in mind that it disables sendfile so if you generally tune for sendfile - you may have to apply output\_buffers tunings as well. It’s hard to say anything more than this without knowing lots of details. Maxim D. ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 9:01am UTC](https://www.ruby-forum.com/t/image-hosting/197422/19 "2010-10-15T09:01:23Z")
Thank’s maxim for the clear explanation. Will try ur suggestion and will let u know. What any information do you need to help this situation? ---
**Author:** ![Indo\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/i/ce7236/32.png) [@Indo\_P](https://www.ruby-forum.com/u/Indo_P)\ **Post date:** [October 15, 2010, 8:50am UTC](https://www.ruby-forum.com/t/image-hosting/197422/20 "2010-10-15T08:50:44Z")
Hi Dennis, For the output of vmstat and iostat I’ll update you soon, because now I’m using my mobile. The hits of the file is almost 90% pool of images. [Next page](https://www.ruby-forum.com/t/image-hosting/197422.md?page=2)