Performance issus, please help?

Server NGINX
1

Hey All,

I am pretty much a noobie at nginx and i really need some help.

I am using nginx as a reverse proxy server to serve primarily as a load
balancer. I am using almost only dynamic files. The back end servers are
apache.

Here are my httperf results:

single apache server (1024 mb): 300 requests per second
2x 512 mb apache server, 1 nginx server( 1024 mb) :300 requests per
second
2x 1024 mb apache server, 1 nginx server( 1024 mb) :300 requests per
second

It seems that my nginx server is the bottleneck but i cant figure out
how i can optimize it.

the cpu usage and ram usage on the apache backend server and nginx
server is minimal, less than 10%.

My goal is to find a great way to scale up and by using a load balancer,
but it seems that if nginx is limited in requests per second as a single
apache server, then there is no point…

May i get some help from anyone please?

Posted at Nginx Forum:

2

If your page can using cache, you can try proxy_cache.
And static file don’t fallback to Apache, direct using nginx to
process.

Example config:
proxy_cache_path /var/www/cdn_cache levels=1:2
keys_zone=cachedata:128m max_size=4096m;

upstream backend {
server 127.0.0.1:9000 weight=3 fail_timeout=2s;
}

location ~ .(gif|jpg|png|css|js|txt|xml)$
{
root /path/to/web;
}

location ~ ^/view.php$ {
proxy_cache cachedata;
proxy_cache_valid 200 304 60s;
proxy_cache_key $uri$gzip$is_args$args;
add_header X-Cache “$upstream_cache_status from $hostname”;
expires 60s;
proxy_set_header X-REAL-IP $remote_addr;
proxy_set_header Host $host;
proxy_pass http://backend;
}

location / {
proxy_pass http://backend;
}

Posted at Nginx Forum:

3

Are the Apache and Nginx servers separate physical servers or are they
VPS?

Have you checked if you’re hitting network bottleneck? If each response
is 10KB, then 300 rps translate to 3MB/s, or 24mbps, add in all the
networking overhead and we’re talking about 30mbps. Maybe you’ve hit the
bandwidth limit.

Posted at Nginx Forum:

4

I am currently using rackspace. I am pretty sure they are VPS.

I doubt it is a network issue only because, if i increase the RAM size
of the nginx server, the requests per second shoots up.

Posted at Nginx Forum:

5

Also,
Here are more tests:

4x 512 mb apache server, 1 nginx server( 2048 mb) :700 requests per
second
4x 2048 mb apache server, 1 nginx server( 2048 mb) :700 requests per
second

In conjuction with the data giving before, it shows that nginx is the
bottle neck. But the cpu and memory usage are very very low. I am using
“top” and sars to check activity.

Also, using and activity on apache seems to be very low as well.

Posted at Nginx Forum:

6

On 8/29/11, w3elf [email protected] wrote:

In conjuction with the data giving before, it shows that nginx is the
bottle neck.

But the cpu and memory usage are very very low.

How is it a bottleneck than? It’s kind of the opposite.

Try to increase number of concurrent requests in your testing tool.
And don’t forget to check nginx’s error.log after each test.

7

On 8/29/11, w3elf [email protected] wrote:

I am using Autobench to test the nginx server performance. The numbers i
generated above are when the response times are still very low. After
than, more requests per second starts to error out.

So, those numbers weren’t even real ones.
Try to use something different that has a proper concurrency control,
like ab.

8

I am using Autobench to test the nginx server performance. The numbers i
generated above are when the response times are still very low. After
than, more requests per second starts to error out.

IE:
4x 512 mb apache server, 1 nginx server( 2048 mb) :700 requests per
second
4x 2048 mb apache server, 1 nginx server( 2048 mb) :700 requests per
second

the range of testing was from 500 -1000 requests per second. the
response times and network IO increased steadily until it hit 700, then
response times increase dramatically and network IO stayed flat at that
point.

Posted at Nginx Forum:

9

what is ab?

Posted at Nginx Forum:

10

On 8/29/11, w3elf [email protected] wrote:

what is ab?

Apache benchmark tool, it comes with apache I believe or at least it
did a few years ago.

11

Thanks for the suggestion.
I ran some tests and used top to monitor my nginx server and apache
backend server. During this whole test, i still had about 60mb free on
my nginx server and 80mb free (85mb before test) on my apache backend
server. I can clearly see that the free ram was slowly decreasing on the
apache server as it was running. So i can see it working (aside from the
new process spawns taking about 2.7% mem usage).

But why isnt it using up almost all the free ram?

here is my output:

#ab -n 30000 -c 50 http://xxx.xxx.xxx.xxx/index.html

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 50.56.117.62 (be patient)
Completed 3000 requests
Completed 6000 requests
Completed 9000 requests
Completed 12000 requests
Completed 15000 requests
Completed 18000 requests
Completed 21000 requests
Completed 24000 requests
Completed 27000 requests
Completed 30000 requests
Finished 30000 requests

Server Software: nginx/1.0.5
Server Hostname: 50.56.117.62
Server Port: 80

Document Path: /index.html
Document Length: 423 bytes

Concurrency Level: 50
Time taken for tests: 45.253 seconds
Complete requests: 30000
Failed requests: 15000
(Connect: 0, Receive: 0, Length: 15000, Exceptions: 0)
Write errors: 0
Total transferred: 20655000 bytes
HTML transferred: 12675000 bytes
Requests per second: 662.94 [#/sec] (mean)
Time per request: 75.422 [ms] (mean)
Time per request: 1.508 [ms] (mean, across all concurrent
requests)
Transfer rate: 445.73 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 13 151.9 1 9610
Processing: 1 61 206.3 38 7767
Waiting: 1 58 203.6 37 7767
Total: 1 73 254.0 40 9613

Percentage of the requests served within a certain time (ms)
50% 40
66% 40
75% 40
80% 40
90% 76
95% 280
98% 759
99% 761
100% 9613 (longest request)

Posted at Nginx Forum:

12

Hi,

I think it is client problem (bandwidth or cpu). This is my short test
on your server:

Xeon CPU and 10 Mbit/s line

ab -n 10000 -c 40 http://50.56.117.62/index.html

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 50.56.117.62 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software: nginx/1.0.5
Server Hostname: 50.56.117.62
Server Port: 80

Document Path: /index.html
Document Length: 423 bytes

Concurrency Level: 40
Time taken for tests: 8.322 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 8440000 bytes
HTML transferred: 4230000 bytes
Requests per second: 1201.57 [#/sec] (mean)
Time per request: 33.290 [ms] (mean)
Time per request: 0.832 [ms] (mean, across all concurrent
requests)
Transfer rate: 990.36 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 4
Processing: 5 33 5.3 30 79
Waiting: 5 33 5.1 30 79
Total: 6 33 5.3 30 80

Percentage of the requests served within a certain time (ms)
50% 30
66% 36
75% 38
80% 38
90% 40
95% 41
98% 41
99% 42
100% 80 (longest request)

Posted at Nginx Forum:

13

Perhaps this may help.

http://web.eecs.utk.edu/~xwang/papers/rtss08.pdf

Sent from my iPhonea

14

What metric are you concerned with?

Sent from my iPhone

15

Hmm That makes sense, but in my case, i cant figure out why i am
experiencing this issue:

IE, 1gb nginx server handles 300 requests per second but a 2gb nginx
server handles 700 requests per second. Everything else is identical in
the. The weird is that the ram usage on these tests is very low.

So my question is, why would increasing the total system ram affect the
server performance if total ram usage is very low in the first place???

Posted at Nginx Forum:

16

What do your system tuning look like? Show an output of ulimit and
sysctl please

On 8/30/11, w3elf [email protected] wrote:

Posted at Nginx Forum:
Re: performance issus, please help???

nginx mailing list
[email protected]
nginx Info Page

–
Sent from my mobile device

Payam Tarverdyan Chychi
Network Security Specialist / Network Engineer

17

highpointe Wrote:

What metric are you concerned with?

None. I just wanted to point out, that results of performance tests
heavily depend on your location and client computer. Without any special
settings I could easily top w3elfs results.

When doing performance tests, always check:

  • is the client-power sufficient? (packet rate, tcp settings etc.)
  • can your connection handle the intended traffic? (not only capacity,
    but in terms of latency and packet rate → problems with link saturation
    etc.)
  • which route to the server is used?
  • are the results different, when using another client/location?

Always check these double, if you are not testing with direct
connection.

Posted at Nginx Forum: