# Using proxy\_store under heavy load

**URL:** <https://www.ruby-forum.com/t/using-proxy-store-under-heavy-load/168427>\
**Category:** NGINX\
**Created:** [May 20, 2009, 6:54pm UTC](https://www.ruby-forum.com/t/using-proxy-store-under-heavy-load/168427 "2009-05-20T18:54:09Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![zepolen](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/z/e68b1a/32.png) [@zepolen](https://www.ruby-forum.com/u/zepolen)\
**Post date:** [May 20, 2009, 6:54pm UTC](https://www.ruby-forum.com/t/using-proxy-store-under-heavy-load/168427/1 "2009-05-20T18:54:09Z")

</div>

I’m using proxy\_store to act as a frontend mirror/cache to amazon s3 for  
my  
sites photos.  
Response times are slow, iotop reports ~10-15MB/s being written to disk  
by  
nginx.  
The website implements a ‘latest’ feature, any new photos will requested  
by  
many users at the same time.  
I’m thinking nginx is probably getting a request for a photo it doesn’t  
already have, and while it is retrieving the file from s3, more requests  
come in for the same file, meaning more round trips and more temp files  
being created.  
Does proxy\_cache handle it differently? (ie, does it know that a url is  
currently being retrieved from the backed, and block other requests for  
that  
url until the file has been retrieved)

---

<div class="post-metadata">

**Author:** ![zepolen](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/z/e68b1a/32.png) [@zepolen](https://www.ruby-forum.com/u/zepolen)\
**Post date:** [May 20, 2009, 8:03pm UTC](https://www.ruby-forum.com/t/using-proxy-store-under-heavy-load/168427/2 "2009-05-20T20:03:53Z")

</div>

On Wed, May 20, 2009 at 07:48:27PM +0300, zepolen wrote:

> Does proxy\_cache handle it differently? (ie, does it know that a url is  
> currently being retrieved from the backed, and block other requests for that  
> url until the file has been retrieved)

No, proxy\_cache does the same.

---

<div class="post-metadata">

**Author:** ![zepolen](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/z/e68b1a/32.png) [@zepolen](https://www.ruby-forum.com/u/zepolen)\
**Post date:** [May 27, 2009, 3:32am UTC](https://www.ruby-forum.com/t/using-proxy-store-under-heavy-load/168427/3 "2009-05-27T03:32:55Z")

</div>

I suspect that the worker is spending a lot of time in the write() to  
the  
disk of the cached file. How many workers are your running ?

Cheers

Dave

zepolen writes:

---

<div class="post-metadata">

**Author:** ![zepolen](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/z/e68b1a/32.png) [@zepolen](https://www.ruby-forum.com/u/zepolen)\
**Post date:** [May 27, 2009, 5:25am UTC](https://www.ruby-forum.com/t/using-proxy-store-under-heavy-load/168427/4 "2009-05-27T05:25:21Z")

</div>

On Wed, May 27, 2009 at 4:23 AM, Dave C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> I suspect that the worker is spending a lot of time in the write() to the disk of the cached file. How many workers are your running ?
> 
> > Response times are slow, iotop reports ~10-15MB/s being written to disk by nginx.  
> > I’m thinking nginx is probably getting a request for a photo it doesn’t already have, and while it is retrieving the file from s3, more requests come in for the same file, meaning more round trips and more temp files being created.

It was a typo in the path where nginx was supposed to find the files  
that had been stored. As a result every single file was being  
retrieved from the backend, worse, it was being written to disk, only  
to be discarded.

It made perfect sense in retrospect as outgoing eth traffic was also  
stuck at about ~15MB/s, and incoming had jumped to the same level,  
unfortunately had to wait for the graphs before I could realise the  
problem.