Nginx ridiculously slow with PHP+SSL combined

So, I’m having an issue with SSL and fastcgi_pass,
specifically to PHP.

PHP over standard http gets me over 6,000 requests
per second, without stressing the CPU. It’s limited
by somaxconn and maximum php children.

Static SSL also easily passes 6,000 requests per
second without stressing the CPU. It’s limited
by raw bandwidth. Setting up a proxy on localhost
and forcing it through proxy_pass did not significantly
slow this test down.

Calling the same php file over SSL, however, causes
nginx to fully utilize all cores - php takes nothing -
and only permits 400 requests/second. Taking CPU
usage into account, this is roughly fifty times slower
than the above two tests. Disabling gzip provides
a ~10% boost in speed. Likewise, I got the same
result through proxy_pass.

My server is a quad-core nehalem (E5520 - 2.27 ghz)
with 12 gigs of RAM, running Debian Squeeze.

I’ve also done these tests on a Wheezy server,
on the hopes that the issue had been resolved
in 1.1.12, but no such luck. I hadn’t tweaked
its settings to handle this sort of load, so
the base php and ssl requests/second are
much less dramatic than the production server,
but the PHP-over-SSL results are the same.

This seems like a rather enormous discrepancy,
but I have no idea where to look. Does anyone have
an idea of what might be going on?

Siege reports the same limitation for php over SSL,
but because it doesn’t support keepalive it blows
through sockets too fast to meaningfully test
my servers for standard php or ssl calls.

apachebench logs follow.

ab -k -c 900 -n 90000 http://amaterasu/test.php
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking amaterasu (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests

Server Software: nginx
Server Hostname: amaterasu
Server Port: 80

Document Path: /test.php
Document Length: 24 bytes

Concurrency Level: 900
Time taken for tests: 14.159 seconds
Complete requests: 90000
Failed requests: 5754
(Connect: 0, Receive: 0, Length: 5754, Exceptions: 0)
Write errors: 0
Keep-Alive requests: 0
Total transferred: 14577071 bytes
HTML transferred: 2157071 bytes
Requests per second: 6356.29 [#/sec] (mean)
Time per request: 141.592 [ms] (mean)
Time per request: 0.157 [ms] (mean, across all concurrent
requests)
Transfer rate: 1005.38 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 58 61 3.4 61 122
Processing: 59 80 83.6 70 3084
Waiting: 59 78 83.4 69 3083
Total: 118 141 83.8 131 3145

Percentage of the requests served within a certain time (ms)
50% 131
66% 139
75% 146
80% 151
90% 168
95% 185
98% 200
99% 210
100% 3145 (longest request)


ab -k -c 900 -n 90000 https://amaterasu/robots.txt
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking amaterasu (be patient)
Completed 9000 requests
Completed 18000 requests
Completed 27000 requests
Completed 36000 requests
Completed 45000 requests
Completed 54000 requests
Completed 63000 requests
Completed 72000 requests
Completed 81000 requests
Completed 90000 requests
Finished 90000 requests

Server Software: nginx
Server Hostname: amaterasu
Server Port: 443
SSL/TLS Protocol: TLSv1/SSLv3,RC4-SHA,2048,128

Document Path: /robots.txt
Document Length: 1063 bytes

Concurrency Level: 900
Time taken for tests: 13.838 seconds
Complete requests: 90000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 89708
Total transferred: 118881532 bytes
HTML transferred: 95670000 bytes
Requests per second: 6503.96 [#/sec] (mean)
Time per request: 138.377 [ms] (mean)
Time per request: 0.154 [ms] (mean, across all concurrent
requests)
Transfer rate: 8389.77 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 20 222.2 0 3959
Processing: 59 117 177.0 107 3496
Waiting: 59 117 177.0 107 3496
Total: 59 137 351.4 107 4083

Percentage of the requests served within a certain time (ms)
50% 107
66% 110
75% 112
80% 113
90% 117
95% 120
98% 125
99% 1590
100% 4083 (longest request)


ab -k -c 900 -n 9000 https://amaterasu/test.php
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking amaterasu (be patient)
Completed 900 requests
Completed 1800 requests
Completed 2700 requests
Completed 3600 requests
Completed 4500 requests
Completed 5400 requests
Completed 6300 requests
Completed 7200 requests
Completed 8100 requests
Completed 9000 requests
Finished 9000 requests

Server Software: nginx
Server Hostname: amaterasu
Server Port: 443
SSL/TLS Protocol: TLSv1/SSLv3,RC4-SHA,2048,128

Document Path: /test.php
Document Length: 24 bytes

Concurrency Level: 900
Time taken for tests: 22.271 seconds
Complete requests: 9000
Failed requests: 445
(Connect: 0, Receive: 0, Length: 445, Exceptions: 0)
Write errors: 0
Keep-Alive requests: 0
Total transferred: 1457620 bytes
HTML transferred: 215620 bytes
Requests per second: 404.11 [#/sec] (mean)
Time per request: 2227.101 [ms] (mean)
Time per request: 2.475 [ms] (mean, across all concurrent
requests)
Transfer rate: 63.92 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 181 1033 682.6 885 3609
Processing: 59 1169 655.3 1103 3552
Waiting: 59 1157 661.4 1094 3552
Total: 266 2202 796.6 2223 4715

Percentage of the requests served within a certain time (ms)
50% 2223
66% 2369
75% 2474
80% 2720
90% 3140
95% 3775
98% 3982
99% 4005
100% 4715 (longest request)

Posted at Nginx Forum:

26 января 2012, 19:31 от “Xeriar” [email protected]:

This seems like a rather enormous discrepancy,
but I have no idea where to look. Does anyone have
an idea of what might be going on?

Take a look at this and you will probably notice
what is going on:

Transfer rate: 8389.77 [Kbytes/sec] received

ab -k -c 900 -n 9000 https://amaterasu/test.php
Complete requests: 9000
Failed requests: 445
Keep-Alive requests: 0
Transfer rate: 63.92 [Kbytes/sec] received

The robots.txt via SSL test shows 89708 of 90000
Keep-Alive requests, which means the same SSL
connection was reused 99.67 % of the time, which
is why you got such good test results.

But the test.php via SSL test shows 0 out of 9000
Keep-Alive requests, which means that a new SSL
connection was established for each and every
successful request. Considering how CPU intensive
SSL is, your test results are really not that bad.

Max

I am a moron.

This is embarassing, you’d think that ‘siege
reported the same results…doesn’t use
keepalive’ would have lit a bulb somewhere.

Setting a valid Content-Length gives
a much nicer picture, and it still only uses
half the cpu:

Server Software: nginx
Server Hostname: amaterasu
Server Port: 443
SSL/TLS Protocol: TLSv1/SSLv3,RC4-SHA,2048,128

Document Path: /test.php
Document Length: 13 bytes

Concurrency Level: 900
Time taken for tests: 13.894 seconds
Complete requests: 90000
Failed requests: 5
(Connect: 0, Receive: 0, Length: 5, Exceptions: 0)
Write errors: 0
Keep-Alive requests: 89372
Total transferred: 29680613 bytes
HTML transferred: 1169948 bytes
Requests per second: 6477.66 [#/sec] (mean)
Time per request: 138.939 [ms] (mean)
Time per request: 0.154 [ms] (mean, across all concurrent
requests)
Transfer rate: 2086.16 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 15 146.6 0 2346
Processing: 62 123 110.9 102 3125
Waiting: 0 123 108.5 102 1679
Total: 62 138 237.3 102 3125

Percentage of the requests served within a certain time (ms)
50% 102
66% 113
75% 122
80% 129
90% 155
95% 193
98% 413
99% 1204
100% 3125 (longest request)


Thank you for pointing out my stupidity, I’ll go hide
in shame now. : )

Max Wrote:

Max


nginx mailing list
[email protected]
nginx Info Page

Posted at Nginx Forum: