# Performance issus, please help?

**URL:** <https://www.ruby-forum.com/t/performance-issus-please-help/211198>\
**Category:** NGINX\
**Created:** [August 29, 2011, 8:12am UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198 "2011-08-29T08:12:16Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [August 29, 2011, 8:12am UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/1 "2011-08-29T08:12:16Z")

</div>

Hey All,

I am pretty much a noobie at nginx and i really need some help.

I am using nginx as a reverse proxy server to serve primarily as a load  
balancer. I am using almost only dynamic files. The back end servers are  
apache.

Here are my httperf results:

single apache server (1024 mb): 300 requests per second  
2x 512 mb apache server, 1 nginx server( 1024 mb) :300 requests per  
second  
2x 1024 mb apache server, 1 nginx server( 1024 mb) :300 requests per  
second

It seems that my nginx server is the bottleneck but i cant figure out  
how i can optimize it.

the cpu usage and ram usage on the apache backend server and nginx  
server is minimal, less than 10%.

My goal is to find a great way to scale up and by using a load balancer,  
but it seems that if nginx is limited in requests per second as a single  
apache server, then there is no point…

May i get some help from anyone please?

Posted at Nginx Forum:

> **[performance issus, please help???](https://forum.nginx.org/read.php?2,214412,214412#msg-214412)**
>
> Hey All, I am pretty much a noobie at nginx and i really need some help. I am using nginx as a reverse proxy server to serve primarily as a load balancer. I am using almost only dynamic files. The back end servers are apache. Here are my httperf...

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 29, 2011, 10:03am UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/2 "2011-08-29T10:03:26Z")

</div>

If your page can using cache, you can try proxy\_cache.  
And static file don’t fallback to Apache, direct using nginx to  
process.

Example config:  
proxy\_cache\_path /var/www/cdn\_cache levels=1:2  
keys\_zone=cachedata:128m max\_size=4096m;

upstream backend {  
server 127.0.0.1:9000 weight=3 fail\_timeout=2s;  
}

location ~ .(gif|jpg|png|css|js|txt|xml)$  
{  
root /path/to/web;  
}

location ~ ^/view.php$ {  
proxy\_cache cachedata;  
proxy\_cache\_valid 200 304 60s;  
proxy\_cache\_key $uri$gzip$is\_args$args;  
add\_header X-Cache “$upstream\_cache\_status from $hostname”;  
expires 60s;  
proxy\_set\_header X-REAL-IP $remote\_addr;  
proxy\_set\_header Host $host;  
proxy\_pass [http://backend](http://backend);  
}

location / {  
proxy\_pass [http://backend](http://backend);  
}

Posted at Nginx Forum:

> **[Re: performance issus, please help???](https://forum.nginx.org/read.php?2,214412,214416#msg-214416)**
>
> Hey All, I am pretty much a noobie at nginx and i really need some help. I am using nginx as a reverse proxy server to serve primarily as a load balancer. I am using almost only dynamic files. The back end servers are apache. Here are my httperf...

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 29, 2011, 5:50pm UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/3 "2011-08-29T17:50:26Z")

</div>

Are the Apache and Nginx servers separate physical servers or are they  
VPS?

Have you checked if you’re hitting network bottleneck? If each response  
is 10KB, then 300 rps translate to 3MB/s, or 24mbps, add in all the  
networking overhead and we’re talking about 30mbps. Maybe you’ve hit the  
bandwidth limit.

Posted at Nginx Forum:

> **[Re: performance issus, please help???](https://forum.nginx.org/read.php?2,214412,214449#msg-214449)**
>
> Hey All, I am pretty much a noobie at nginx and i really need some help. I am using nginx as a reverse proxy server to serve primarily as a load balancer. I am using almost only dynamic files. The back end servers are apache. Here are my httperf...

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 29, 2011, 6:49pm UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/4 "2011-08-29T18:49:10Z")

</div>

I am currently using rackspace. I am pretty sure they are VPS.

I doubt it is a network issue only because, if i increase the RAM size  
of the nginx server, the requests per second shoots up.

Posted at Nginx Forum:

> **[Re: performance issus, please help???](https://forum.nginx.org/read.php?2,214412,214451#msg-214451)**
>
> Hey All, I am pretty much a noobie at nginx and i really need some help. I am using nginx as a reverse proxy server to serve primarily as a load balancer. I am using almost only dynamic files. The back end servers are apache. Here are my httperf...

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 29, 2011, 6:52pm UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/5 "2011-08-29T18:52:53Z")

</div>

Also,  
Here are more tests:

4x 512 mb apache server, 1 nginx server( 2048 mb) :700 requests per  
second  
4x 2048 mb apache server, 1 nginx server( 2048 mb) :700 requests per  
second

In conjuction with the data giving before, it shows that nginx is the  
bottle neck. But the cpu and memory usage are very very low. I am using  
“top” and sars to check activity.

Also, using and activity on apache seems to be very low as well.

Posted at Nginx Forum:

> **[Re: performance issus, please help???](https://forum.nginx.org/read.php?2,214412,214452#msg-214452)**
>
> Hey All, I am pretty much a noobie at nginx and i really need some help. I am using nginx as a reverse proxy server to serve primarily as a load balancer. I am using almost only dynamic files. The back end servers are apache. Here are my httperf...

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 29, 2011, 7:20pm UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/6 "2011-08-29T19:20:03Z")

</div>

On 8/29/11, w3elf [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> In conjuction with the data giving before, it shows that nginx is the  
> bottle neck.

> But the cpu and memory usage are very very low.

How is it a bottleneck than? It’s kind of the opposite.

Try to increase number of concurrent requests in your testing tool.  
And don’t forget to check nginx’s error.log after each test.

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 29, 2011, 7:39pm UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/7 "2011-08-29T19:39:12Z")

</div>

On 8/29/11, w3elf [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> I am using Autobench to test the nginx server performance. The numbers i  
> generated above are when the response times are still very low. After  
> than, more requests per second starts to error out.

So, those numbers weren’t even real ones.  
Try to use something different that has a proper concurrency control,  
like ab.

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 29, 2011, 7:25pm UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/8 "2011-08-29T19:25:00Z")

</div>

I am using Autobench to test the nginx server performance. The numbers i  
generated above are when the response times are still very low. After  
than, more requests per second starts to error out.

IE:  
4x 512 mb apache server, 1 nginx server( 2048 mb) :700 requests per  
second  
4x 2048 mb apache server, 1 nginx server( 2048 mb) :700 requests per  
second

the range of testing was from 500 -1000 requests per second. the  
response times and network IO increased steadily until it hit 700, then  
response times increase dramatically and network IO stayed flat at that  
point.

Posted at Nginx Forum:

> **[Re: performance issus, please help???](https://forum.nginx.org/read.php?2,214412,214454#msg-214454)**
>
> Hey All, I am pretty much a noobie at nginx and i really need some help. I am using nginx as a reverse proxy server to serve primarily as a load balancer. I am using almost only dynamic files. The back end servers are apache. Here are my httperf...

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 29, 2011, 7:43pm UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/9 "2011-08-29T19:43:31Z")

</div>

what is ab?

Posted at Nginx Forum:

> **[Re: performance issus, please help???](https://forum.nginx.org/read.php?2,214412,214456#msg-214456)**
>
> Hey All, I am pretty much a noobie at nginx and i really need some help. I am using nginx as a reverse proxy server to serve primarily as a load balancer. I am using almost only dynamic files. The back end servers are apache. Here are my httperf...

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 29, 2011, 7:48pm UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/10 "2011-08-29T19:48:27Z")

</div>

On 8/29/11, w3elf [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> what is ab?

Apache benchmark tool, it comes with apache I believe or at least it  
did a few years ago.

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 29, 2011, 8:17pm UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/11 "2011-08-29T20:17:31Z")

</div>

Thanks for the suggestion.  
I ran some tests and used top to monitor my nginx server and apache  
backend server. During this whole test, i still had about 60mb free on  
my nginx server and 80mb free (85mb before test) on my apache backend  
server. I can clearly see that the free ram was slowly decreasing on the  
apache server as it was running. So i can see it working (aside from the  
new process spawns taking about 2.7% mem usage).

But why isnt it using up almost all the free ram?

here is my output:

#ab -n 30000 -c 50 [http://xxx.xxx.xxx.xxx/index.html](http://xxx.xxx.xxx.xxx/index.html)

This is ApacheBench, Version 2.3 \<$Revision: 655654 $\>  
Copyright 1996 Adam Twiss, Zeus Technology Ltd,  
[http://www.zeustech.net/](http://www.zeustech.net/)  
Licensed to The Apache Software Foundation, [http://www.apache.org/](http://www.apache.org/)

Benchmarking 50.56.117.62 (be patient)  
Completed 3000 requests  
Completed 6000 requests  
Completed 9000 requests  
Completed 12000 requests  
Completed 15000 requests  
Completed 18000 requests  
Completed 21000 requests  
Completed 24000 requests  
Completed 27000 requests  
Completed 30000 requests  
Finished 30000 requests

Server Software: nginx/1.0.5  
Server Hostname: 50.56.117.62  
Server Port: 80

Document Path: /index.html  
Document Length: 423 bytes

Concurrency Level: 50  
Time taken for tests: 45.253 seconds  
Complete requests: 30000  
Failed requests: 15000  
(Connect: 0, Receive: 0, Length: 15000, Exceptions: 0)  
Write errors: 0  
Total transferred: 20655000 bytes  
HTML transferred: 12675000 bytes  
Requests per second: 662.94 [#/sec] (mean)  
Time per request: 75.422 [ms] (mean)  
Time per request: 1.508 [ms] (mean, across all concurrent  
requests)  
Transfer rate: 445.73 [Kbytes/sec] received

Connection Times (ms)  
min mean[+/-sd] median max  
Connect: 0 13 151.9 1 9610  
Processing: 1 61 206.3 38 7767  
Waiting: 1 58 203.6 37 7767  
Total: 1 73 254.0 40 9613

Percentage of the requests served within a certain time (ms)  
50% 40  
66% 40  
75% 40  
80% 40  
90% 76  
95% 280  
98% 759  
99% 761  
100% 9613 (longest request)

Posted at Nginx Forum:

> **[Re: performance issus, please help???](https://forum.nginx.org/read.php?2,214412,214459#msg-214459)**
>
> Hey All, I am pretty much a noobie at nginx and i really need some help. I am using nginx as a reverse proxy server to serve primarily as a load balancer. I am using almost only dynamic files. The back end servers are apache. Here are my httperf...

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 30, 2011, 10:21am UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/12 "2011-08-30T10:21:16Z")

</div>

Hi,

I think it is client problem (bandwidth or cpu). This is my short test  
on your server:

Xeon CPU and 10 Mbit/s line

ab -n 10000 -c 40 [http://50.56.117.62/index.html](http://50.56.117.62/index.html)

This is ApacheBench, Version 2.3 \<$Revision: 655654 $\>  
Copyright 1996 Adam Twiss, Zeus Technology Ltd,  
[http://www.zeustech.net/](http://www.zeustech.net/)  
Licensed to The Apache Software Foundation, [http://www.apache.org/](http://www.apache.org/)

Benchmarking 50.56.117.62 (be patient)  
Completed 1000 requests  
Completed 2000 requests  
Completed 3000 requests  
Completed 4000 requests  
Completed 5000 requests  
Completed 6000 requests  
Completed 7000 requests  
Completed 8000 requests  
Completed 9000 requests  
Completed 10000 requests  
Finished 10000 requests

Server Software: nginx/1.0.5  
Server Hostname: 50.56.117.62  
Server Port: 80

Document Path: /index.html  
Document Length: 423 bytes

Concurrency Level: 40  
Time taken for tests: 8.322 seconds  
Complete requests: 10000  
Failed requests: 0  
Write errors: 0  
Total transferred: 8440000 bytes  
HTML transferred: 4230000 bytes  
Requests per second: 1201.57 [#/sec] (mean)  
Time per request: 33.290 [ms] (mean)  
Time per request: 0.832 [ms] (mean, across all concurrent  
requests)  
Transfer rate: 990.36 [Kbytes/sec] received

Connection Times (ms)  
min mean[+/-sd] median max  
Connect: 0 0 0.2 0 4  
Processing: 5 33 5.3 30 79  
Waiting: 5 33 5.1 30 79  
Total: 6 33 5.3 30 80

Percentage of the requests served within a certain time (ms)  
50% 30  
66% 36  
75% 38  
80% 38  
90% 40  
95% 41  
98% 41  
99% 42  
100% 80 (longest request)

Posted at Nginx Forum:

> **[Re: performance issus, please help???](https://forum.nginx.org/read.php?2,214412,214485#msg-214485)**
>
> Hey All, I am pretty much a noobie at nginx and i really need some help. I am using nginx as a reverse proxy server to serve primarily as a load balancer. I am using almost only dynamic files. The back end servers are apache. Here are my httperf...

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 30, 2011, 10:59am UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/13 "2011-08-30T10:59:40Z")

</div>

Perhaps this may help.

[http://web.eecs.utk.edu/~xwang/papers/rtss08.pdf](http://web.eecs.utk.edu/~xwang/papers/rtss08.pdf)

Sent from my iPhonea

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 30, 2011, 10:46am UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/14 "2011-08-30T10:46:40Z")

</div>

What metric are you concerned with?

Sent from my iPhone

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 31, 2011, 1:54am UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/15 "2011-08-31T01:54:16Z")

</div>

Hmm That makes sense, but in my case, i cant figure out why i am  
experiencing this issue:

IE, 1gb nginx server handles 300 requests per second but a 2gb nginx  
server handles 700 requests per second. Everything else is identical in  
the. The weird is that the ram usage on these tests is very low.

So my question is, why would increasing the total system ram affect the  
server performance if total ram usage is very low in the first place???

Posted at Nginx Forum:

> **[Re: performance issus, please help???](https://forum.nginx.org/read.php?2,214412,214531#msg-214531)**
>
> Hey All, I am pretty much a noobie at nginx and i really need some help. I am using nginx as a reverse proxy server to serve primarily as a load balancer. I am using almost only dynamic files. The back end servers are apache. Here are my httperf...

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 31, 2011, 2:56am UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/16 "2011-08-31T02:56:00Z")

</div>

What do your system tuning look like? Show an output of ulimit and  
sysctl please

On 8/30/11, w3elf [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Posted at Nginx Forum:  
> [http://forum.nginx.org/read.php?2,214412,214531#msg-214531](http://forum.nginx.org/read.php?2,214412,214531#msg-214531)
> 
> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)

–  
Sent from my mobile device

Payam Tarverdyan Chychi  
Network Security Specialist / Network Engineer

---

<div class="post-metadata">

**Author:** ![w3elf](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/7cd45c/32.png) [@w3elf](https://www.ruby-forum.com/u/w3elf)\
**Post date:** [August 30, 2011, 1:32pm UTC](https://www.ruby-forum.com/t/performance-issus-please-help/211198/17 "2011-08-30T13:32:58Z")

</div>

## highpointe Wrote:

> What metric are you concerned with?

None. I just wanted to point out, that results of performance tests  
heavily depend on your location and client computer. Without any special  
settings I could easily top w3elfs results.

When doing performance tests, always check:

- is the client-power sufficient? (packet rate, tcp settings etc.)
- can your connection handle the intended traffic? (not only capacity,  
but in terms of latency and packet rate → problems with link saturation  
etc.)
- which route to the server is used?
- are the results different, when using another client/location?

Always check these double, if you are not testing with direct  
connection.

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,214412,214493#msg-214493](http://forum.nginx.org/read.php?2,214412,214493#msg-214493)