SSL handshake errors when configured as a reverse proxy

Recently I tried setting up a basic nginx reverse proxy in production on
Ubuntu 14.04 using their default supported version of nginx 1.4.6.

Basic config as follows:

server {

listen 127.0.0.1:443;

server_name myhost.ca;

ssl on;

ssl_certificate /etc/nginx/certs/cert.chained.with.intermediates.crt

ssl_certificate_key /etc/nginx/certs/cert.key

ssl_dhparam /etc/nginx/certs/dhparams.pem;

ssl_session_timeout 5m;

ssl_session_cache shared:test_cache:5m;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_ciphers

'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:

ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:

!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA’;

ssl_prefer_server_ciphers on;

access_log /var/log/nginx/proxy.access.log;

error_log /var/log/nginx/proxy.error.log;

location / {

    proxy_buffering         off;

    proxy_set_header        Host $host;

    proxy_set_header        X-Real-IP $remote_addr;

    proxy_set_header        X-Forwarded-For 

$proxy_add_x_forwarded_for;

    proxy_set_header        X-Forwarded-Proto $scheme;

    proxy_pass https://10.1.10.1;

}

}

Config worked pretty good in testing, but when we put it in production,
we
quickly started seeing intermittent handshake failures.

The handshakes were being rejected by the server with errors like this
in
the error log:

2016/02/16 13:30:18 [info] 6470#0: *6349 SSL_do_handshake() failed (SSL:
error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad
record
mac) while SSL handshaking, client: x.x.x.x, server: x.x.x.x:443

And sometimes from the client like this:

2016/02/16 13:27:51 [info] 6468#0: *6226 peer closed connection in SSL
handshake while SSL handshaking, client: x.x.x.x, server: x.x.x.x:443

Upon further investigation we discovered that the same clients were more
or
less “randomly” effected by this handshake error on an intermittent
basis,
so one request might work but then the next would fail.

Initially we didn’t have any ssl_session_cache enabled, but adding that
shared cache setting above had no effect on the random handshake errors.

Thought it might be an openssl issue, so we tried updating from ubuntu’s
default version of 1.0.1f to 1.0.2f from source, but that had no impact
on
the clients receiving the handshake error.

Subsequently, we switched the reverse proxy on the same system, with the
same configuration (i.e. supported protocols and ciphers) from nginx to
apache, and the intermittent handshake errors have gone away.

I’d still like to know what was wrong with our nginx setup to be causing
this in the first place, anyone have any ideas?

Hi Josh,

When you installed the newer OpenSSL did you recompile NGINX to use the
newer version? If not then it may still have been using the older
OpenSSL with this bug in it. It is likely to be pinned to a specific
version. You can check by running “ldd” on your NGINX binary.

Kind Regards
Andrew

On 18/02/16 23:07, Josh Jaques wrote:

 ssl_session_cache shared:test_cache:5m;

!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA’;

 }

error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad

I’d still like to know what was wrong with our nginx setup to be causing
this in the first place, anyone have any ideas?


nginx mailing list
[email protected]
http://mailman.nginx.org/mailman/listinfo/nginx


Andrew H. (LinuxJedi)
Technical Product Manager, NGINX Inc.

Hi Josh,

There are bugs in OpenSSL 1.0.1e that could trigger this which is why I
asked. The two other things I would suggest trying are:

  1. Look again at your cipher list, missing important ones out can
    trigger this error, especially with ssl_prefer_server_ciphers set.
    Judging by the quick skim looking at them they don’t look correct for
    the ssl_protocols you have chosen. Using the defaults should be fine for
    most cases.

  2. Upgrading to a supported version of NGINX, there have been many SSL
    related bug fixes since then (although I don’t think any match your
    specific case). Our own apt repositories found on nginx.org have more
    current versions in them.

Kind Regards
Andrew

On 19/02/16 18:13, Josh Jaques wrote:

When I switched to Apache, I reverted the system to the package manger
nginx mailing list
[email protected]
http://mailman.nginx.org/mailman/listinfo/nginx


Andrew H. (LinuxJedi)
Technical Product Manager, NGINX Inc.

Hi Andrew,

To clarify the setup earlier,

I continued to use the Ubuntu compiled version of NGINX from apt-get.

The specific procedure I used to change the lib that NGINX would load
was by replacing the libssl.so.1.0.0 and libcrypto.so.1.0.0 files in the
path referenced by ldd for the NGINX binary with ones compiled from
source.

When I switched to Apache, I reverted the system to the package manger
versions of libssl and openssl. It continues to be in production without
producing any handshake errors.

So from my end there doesn’t seem to be any evidence to support OpenSSL
version ever being an issue, because Apache works fine using the same
version of OpenSSL that we initially experienced the problem with in
NGINX.

The Apache version I am running is also the default from apt-get.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs