Reverse ssl proxy - speed & jitter

I am setting up a nginx reverse ssl proxy - I have a machine I can use
2 E5-2650 CPU’s and lots of RAM. I have nginx-1.6.0 + openssl-1.0.1h
installed. I have taken into consideration most optimization suggestions
there and incorporated them. I will attach a copy of my config file.

(optimizing first connection experience is good) With my testing just
handshake + connection setup with 2K cert it is taking 3.5ms on average.
see spikes in this time every 40 or so handshakes. I would like the 90+
percentile of the handshakes to not have any jitter/variance.

testing method:
for i in {1…1000};
httperf --hog --server localhost --port 443 --ssl --uri /nginx_ping
–ssl-no-reuse --num-calls 1 --num-conns 1 --rate 1 | egrep "Connection
time [ms]: |Reply time [ms]: " | awk {‘print $5’} | xargs | tr -s
" "
", " >> test.log;

-if you think this methodology is not right - do let me know. I have
at the tcpdumps and made sure a full handshake is happening and then a
request is issued

gives me: request-time, connect_time, response_time
request_time = connect_time(ssl handshake + connection setup) +

  1. I want to debug why there is jitter in the handshake time - i want
    90th, 95th, 99th, 99.9th percentiles to also be around 3.5ms.
  2. I want to see if i can make nginx any faster to do handshake. what is
    fastest you guys think this can happen
  3. how can i profile nginx and proceed to make this faster

all comments are welcome!


not sure how to attach config:
config details:
5 workers, worker_priority -10, timer_resolution 200ms,
to separates cores on cpu2, error_log to dev/null, use epoll,
2000, multi_accept on, accept_mutex off, sendfile on, tcp_nopush on,
tcp_nodelay on, file caches, keepalive_timeout 5000, keepalive_requests
100000, reset_timedout_connection on, client_body_timeout 10,
2, gzip, server_tokens off, postpone_output 0. upstream: keep alive 180,
proxy_buffering off, client_body_buffer_size 512K,
large_client_header_buffers 4 64k, client_max_body_size 0. server:
443 ssl, access_log off, ssl_buffer_size 8k, ssl_session_timeout 10m,
ssl_protocols SSLv3 TLSv1, ssl_ciphers RC4-MD5,
on, ssl_session_cache shared:SSL:10m. location /nginx_ping - return

Posted at Nginx Forum:

Full Config:

#user nobody;

This number should be, at maximum, the number of CPU cores on your


(since nginx doesn’t benefit from more than one worker per CPU.)

worker_processes 5;

#give worker processes the priority (nice) you need/wish, it calls
worker_priority -10;

#decrease number gettimeofday() syscalls. By default gettimeofday() is
called after each return from

kevent(), epoll, /dev/poll, select(), poll().

timer_resolution 200ms;

#trying to set CPU affinity
worker_cpu_affinity 10001 10010 10011 10100 10101;

#error_log LOGFILE [debug_core | debug_alloc | debug_mutex | debug_event
debug_http | debug_imap], debug, crit, emerg;
error_log /dev/null emerg;

pid var/state/;

Number of file descriptors used for Nginx. This is set in the OS with

‘ulimit -n 200000’ or using /etc/security/limits.conf
#worker_rlimit_nofile 60000;

workers_conns * 2

events {
use epoll;
worker_connections 20000;
# Accept as many connections as possible, after nginx gets
about a new connection.
# May flood worker_connections, if that option is set too low.
multi_accept on;
accept_mutex off;

http {
default_type application/octet-stream;

log_format  main  '[$time_local] - [$time_iso8601] - [$request_time] 

[$upstream_response_time] - $remote_addr ’

’ “$http_x_forwarded_for” - $remote_user -
[$request_time] ’
'$status - $request_length - $body_bytes_sent -
“$http_referer” - ’
'“$http_user_agent” - $uri - $request_method - ’
‘$ssl_protocol - $ssl_cipher’;

sendfile        on;

# Tcp_nopush causes nginx to attempt to send its HTTP response head 

one packet,
# instead of using partial frames. This is useful for prepending
before calling sendfile,
# or for throughput optimization.
tcp_nopush on;

#the TCP_NODELAY option. The option is enabled only when a 

connection is
transitioned into the keep-alive state.
tcp_nodelay on;

# Caches information about open FDs, freqently accessed files.
# Changing this setting, in my environment, brought performance up 

560k req/sec, to 904k req/sec.
# I recommend using some varient of these options, though not the
specific values listed below.
open_file_cache max=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
open_log_file_cache max=100000 inactive=2m valid=10m min_uses=2;

keepalive_timeout  5000;

# Number of requests which can be made over a keep-alive connection.
# Review and change it to a more suitable value if required.
keepalive_requests 100000;

# allow the server to close the connection after a client stops

responding. Frees up socket-associated memory.
reset_timedout_connection on;

# send the client a "request timed out" if the body is not loaded by

this time. Default 60.
client_body_timeout 10;

# If the client stops reading data, free up the stale client 

after this much time. Default 60.
send_timeout 2;

# Compression. Reduces the amount of data that needs to be 

over the network
gzip on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml text/javascript
application/x-javascript application/xml;
gzip_disable “MSIE [1-6].”;

client_body_temp_path var/state/nginx/client_body_temp 1 2;
proxy_temp_path var/state/nginx/proxy_temp 1 2;
fastcgi_temp_path var/state/nginx/fastcgi_temp 1 2;
uwsgi_temp_path var/state/nginx/uwsgi_temp 1 2;
scgi_temp_path var/state/nginx/scgi_temp_path 1 2;

server_tokens off;
postpone_output 0;

upstream downstream_service {
    keepalive 180;

# Turn off proxy buffering
proxy_buffering off;
proxy_buffer_size 128K;
proxy_busy_buffers_size 128K;
proxy_buffers 64 4K;

client_body_buffer_size 512K;
large_client_header_buffers 4 64k;

limit_conn_zone $server_name zone=perserver1:32k;

# Allow arbitrary size client posts
client_max_body_size 0;

# HTTPS Server config
server {
    listen       443 ssl;



# Buffer log writes to speed up IO, or disable them altogether
    access_log off;  # turn off for better performance

    ssl_certificate      /dev/shm/;
    ssl_certificate_key  /dev/shm/;

Do not overflow the SSL send buffer (causes extra round trips)

ssl_buffer_size 8k;

    ssl_session_timeout  10m;

    ssl_protocols  SSLv3 TLSv1;

ssl_ciphers RC4-MD5;

    ssl_prefer_server_ciphers   on;
    ssl_session_cache   shared:SSL:10m;

    set $host_header $host;
    if ($http_host != "")
        set $host_header $http_host;

    location / {
        proxy_pass          http://downstream_service/;
        proxy_http_version  1.1;
        proxy_set_header    Connection "";
        proxy_set_header    Host $host_header;
        proxy_set_header    X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For

limit_conn perserver1 180;

    # Nginx health check only to verify server is up and running
    location /nginx_ping {
        return 200;


Posted at Nginx Forum:

On Wednesday 23 July 2014 14:00:36 newnovice wrote:

Full Config:


You may get better results if you remove most of “optimizations” (as you
called that).

Please don’t collect bad advices. You shouldn’t use any directive
full understanding what it’s for according to official documentation
(nginx documentation), and what real problem you’re trying to

Those spikes can be a result of “multi_accept on”.

error_log /dev/null emerg;

Everyone who added this line to config should be fired.

send_timeout 2;

Mobile clients will blame you for this setting.

wbr, Valentin V. Bartenev

What is the fastest SSL connection setup time that anyone can achieve?
also how do i reduce the jitter/variance…

so what settings should I take out & what shud i have - do you have an
example of an optimal config?

the logs are turned off to see first if this is even a viable option, i
turn up debug and check stuff if needed.

Posted at Nginx Forum:

this is for a very fast internal only API driven service and not serving
webpages/static files/multimedia…

Posted at Nginx Forum: