# Intermittent SSL Handshake Errors **URL:** \ **Category:** NGINX\ **Created:** [January 31, 2015, 7:06pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027 "2015-01-31T19:06:11Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![addis\_a](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/c4cdca/32.png) [@addis\_a](https://www.ruby-forum.com/u/addis_a)\ **Post date:** [January 31, 2015, 7:06pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/1 "2015-01-31T19:06:11Z")
Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL handshake errors. Users reported that these appeared as “ssl\_error\_no\_cypher\_overlap” in the browser. Most of our reports have come from Firefox users, although we have seen reports from Safari and stock Android browser users as well. In our nginx error logs, we began to see consistent errors across all three servers. They started at around the same time and no recent modifications were made to hardware or software: … 2015/01/13 12:22:59 [crit] 11871#0: _140260577 SSL\_do\_handshake() failed (SSL: error:1408A0D7:SSL routines:SSL3\_GET\_CLIENT\_HELLO:required cipher missing) while SSL handshaking, client: .._._, server: 0.0.0.0:443 2015/01/13 12:23:09 [crit] 11874#0: 140266246 SSL\_do\_handshake() failed (SSL: error:1408A0D7:SSL routines:SSL3\_GET\_CLIENT\_HELLO:required cipher missing) while SSL handshaking, client: ..._, server: 0.0.0.0:443 2015/01/13 12:23:54 [crit] 11862#0: _140293705 SSL\_do\_handshake() failed (SSL: error:1408A0D7:SSL routines:SSL3\_GET\_CLIENT\_HELLO:required cipher missing) while SSL handshaking, client: .._._, server: 0.0.0.0:443 2015/01/13 12:23:54 [crit] 11862#0: 140293708 SSL\_do\_handshake() failed (SSL: error:1408A0D7:SSL routines:SSL3\_GET\_CLIENT\_HELLO:required cipher missing) while SSL handshaking, client: ..._, server: 0.0.0.0:443 2015/01/13 12:25:18 [crit] 11870#0: _140342155 SSL\_do\_handshake() failed (SSL: error:1408A0D7:SSL routines:SSL3\_GET\_CLIENT\_HELLO:required cipher missing) while SSL handshaking, client: .._.\*, server: 0.0.0.0:443 … Suspecting that this may be related to our SSL configuration in nginx and a recent update to a major browser, I decided to get us up to date. Previously we were on CentOS5 and could only use an older version of OpenSSL with the latest security patches. This meant we could only support TLSv1.0 and a few of the secure recommended ciphers. After upgrading to CentOS6 and implementing Mozilla’s recommended configurations for TLSv1.0, TLSv1.1, and TLSv1.2 support, I am confident that we are following best practices for SSL browser compatibility and security. Unfortunately this did not fix the issue. Users began to report a new error in their browser: “ssl\_error\_inappropriate\_fallback\_alert”, and this is currently reflected in our nginx error logs across all three servers: … 2015/01/31 03:24:33 [crit] 30658#0: _57298755 SSL\_do\_handshake() failed (SSL: error:140A1175:SSL routines:SSL\_BYTES\_TO\_CIPHER\_LIST:inappropriate fallback) while SSL handshaking, client: .._._, server: 0.0.0.0:443 2015/01/31 03:24:35 [crit] 30661#0: 57299105 SSL\_do\_handshake() failed (SSL: error:140A1175:SSL routines:SSL\_BYTES\_TO\_CIPHER\_LIST:inappropriate fallback) while SSL handshaking, client: ..._, server: 0.0.0.0:443 2015/01/31 03:24:41 [crit] 30657#0: _57300774 SSL\_do\_handshake() failed (SSL: error:140A1175:SSL routines:SSL\_BYTES\_TO\_CIPHER\_LIST:inappropriate fallback) while SSL handshaking, client: .._._, server: 0.0.0.0:443 2015/01/31 03:24:41 [crit] 30657#0: 57300783 SSL\_do\_handshake() failed (SSL: error:140A1175:SSL routines:SSL\_BYTES\_TO\_CIPHER\_LIST:inappropriate fallback) while SSL handshaking, client: ..._, server: 0.0.0.0:443 2015/01/31 03:24:41 [crit] 30661#0: _57300785 SSL\_do\_handshake() failed (SSL: error:140A1175:SSL routines:SSL\_BYTES\_TO\_CIPHER\_LIST:inappropriate fallback) while SSL handshaking, client: .._.\*, server: 0.0.0.0:443 … Thinking that I had ruled out a faulty SSL stack or nginx configuration, I focused on monitoring the network connections on these servers. ESTABLISHED connections are currently at 13k and TIME\_WAIT is at 94k on one server, if that gives any indication to the type of connections we are dealing with. The other two have very similar stats. This is typical for peak hours of traffic. I tried tuning kernel params: lowering tcp\_fin\_timeout, increasing tcp\_max\_syn\_backlog, increasing the range of ip\_local\_port\_range, turning on tcp\_tw\_reuse, and other popular tuning practices. Nothing has helped so far and more users continue to contact us about issues using our site. I’ve exhausted my ideas and I’m not quite sure what’s gone wrong. I would be extremely appreciative of any guidance list members could provide. Below are more technical details about our installation and configuration of nginx. nginx -V output: nginx version: nginx/1.6.2 built by gcc 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx –conf-path=/etc/nginx/nginx.conf –error-log-path=/var/log/nginx/error.log –http-log-path=/var/log/nginx/access.log –pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock –http-client-body-temp-path=/var/cache/nginx/client\_temp –http-proxy-temp-path=/var/cache/nginx/proxy\_temp –http-fastcgi-temp-path=/var/cache/nginx/fastcgi\_temp –http-uwsgi-temp-path=/var/cache/nginx/uwsgi\_temp –http-scgi-temp-path=/var/cache/nginx/scgi\_temp --user=nginx –group=nginx --with-http\_ssl\_module --with-http\_realip\_module –with-http\_addition\_module --with-http\_sub\_module –with-http\_dav\_module --with-http\_flv\_module --with-http\_mp4\_module –with-http\_gunzip\_module --with-http\_gzip\_static\_module –with-http\_random\_index\_module --with-http\_secure\_link\_module –with-http\_stub\_status\_module --with-http\_auth\_request\_module –with-mail --with-mail\_ssl\_module --with-file-aio --with-ipv6 –with-http\_spdy\_module --with-cc-opt=‘-O2 -g -pipe -Wp,-D\_FORTIFY\_SOURCE=2 -fexceptions -fstack-protector –param=ssp-buffer-size=4 -m64 -mtune=generic’ nginx config files: — /etc/nginx/nginx.conf — user nginx; worker\_processes 12; error\_log /var/log/nginx/error.log; pid /var/run/nginx.pid; events { worker\_connections 50000; } http { include /etc/nginx/mime.types; default\_type application/octet-stream; log\_format with\_cookie '$remote\_addr - $remote\_user [$time\_local] ’ '“$request” $status $body\_bytes\_sent ’ ‘“$http\_referer” “$http\_user\_agent” “$cookie\_FL”’; access\_log /var/log/nginx/access.log; sendfile on; tcp\_nopush on; tcp\_nodelay on; keepalive\_timeout 65; gzip on; gzip\_http\_version 1.0; gzip\_comp\_level 2; gzip\_proxied any; gzip\_types text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript application/json; gzip\_vary on; server\_names\_hash\_bucket\_size 64; set\_real\_ip\_from _._._._; real\_ip\_header X-Forwarded-For; include /etc/nginx/upstreams.conf; include /etc/nginx/sites-enabled/\*; } — /etc/nginx/sites-enabled/fl-ssl.conf — server { root /var/www/fl/current/public; listen 443; ssl on; ssl\_certificate /etc/nginx/ssl/wildcard.fl.pem; ssl\_certificate\_key /etc/nginx/ssl/wildcard.fl.key; ssl\_session\_timeout 5m; ssl\_session\_cache shared:SSL:50m; ssl\_protocols TLSv1 TLSv1.1 TLSv1.2; ssl\_ciphers ‘ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA’; ssl\_prefer\_server\_ciphers on; server\_name \*\*\*\*\*\*\*\*\*\*.com; access\_log /var/log/nginx/fl.ssl.access.log with\_cookie; client\_max\_body\_size 400M; index index.html index.htm; if (-f $document\_root/system/maintenance.html) { return 503; } # Google Analytics if ($request\_filename ~\* ga.js$) { rewrite .\* [http://www.google-analytics.com/ga.js](http://www.google-analytics.com/ga.js) permanent; break; } if ($request\_filename ~\* /adgear.js/current/adgear\_standard.js) { rewrite .\* http://\*\*\*\*\*\*\*\*\*\*.com/adgear/adgear\_standard.js permanent; break; } if ($request\_filename ~\* /adgear.js/current/adgear.js) { rewrite .\* http://\*\*\*\*\*\*\*\*\*\*.com/adgear/adgear\_standard.js permanent; break; } if ($request\_filename ~\* \_\_utm.gif$) { rewrite .\* [http://www.google-analytics.com/\_\_utm.gif](http://www.google-analytics.com/__utm.gif) permanent; break; } if ($host ~\* “www”) { rewrite ^(._)$ http://_\*\*\*\*\*\*\*\*.com$1 permanent; break; } location / { location ~\* .(eot|ttf|woff)$ { add\_header Access-Control-Allow-Origin \*; } ``` if ($request_uri ~* ".(ico|css|js|gif|jpe?g|png)\?[0-9]+$") { expires max; break; } # needed to forward user's IP address to rails proxy_set_header X-Real-IP $remote_addr; # needed for HTTPS proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X-FORWARDED_PROTO https; proxy_redirect off; proxy_max_temp_file_size 0; if ($request_uri ~* /polling) { proxy_pass http://ssl_polling_upstream; break; } if ($request_uri = /upload) { proxy_pass http://rest_stop_upstream; break; } if ($request_uri = /crossdomain.xml) { proxy_pass http://rest_stop_upstream; break; } if (-f $request_filename/index.html) { rewrite (.*) $1/index.html break; } # Rails 3 is for old testing stuff... We don't need this anymore #if ($http_cookie ~ "rails3=true") { # set $request_type '3'; #} if ($request_uri ~* /polling) { set $request_type '${request_type}P'; } if ($request_type = '3P') { proxy_pass http://rails3_upstream; break; } if ($request_uri ~* /polling) { set $request_type '${request_type}P'; } if ($request_type = '3P') { proxy_pass http://rails3_upstream; break; } if ($request_type = 'P') { proxy_pass http://ssl_polling_upstream; break; } if (!-f $request_filename) { set $request_type '${request_type}D'; } if ($request_type = 'D') { proxy_pass http://ssl_fl_upstream; break; } if ($request_type = '3D') { proxy_pass http://rails3_upstream; break; } ``` } error\_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [January 31, 2015, 8:02pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/2 "2015-01-31T20:02:57Z")
> … > 2015/01/13 12:22:59 [crit] 11871#0: _140260577 SSL\_do\_handshake() > failed (SSL: error:1408A0D7:SSL > routines:SSL3\_GET\_CLIENT\_HELLO:required cipher missing) while SSL > handshaking, client: .._.\*, server: 0.0.0.0:443 According to the openssl code, this occurs when a client attempts to resume a session that had made use of previously-enabled ciphers. If you’re changing your allowed ciphers frequently this could be why, otherwise a full cycle of nginx to empty out the session cache seems like it should resolve this. ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [January 31, 2015, 9:25pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/3 "2015-01-31T21:25:49Z")
Hi Le 31 janv. 2015 20:02, “Richard S.” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) a écrit : > > … > > 2015/01/13 12:22:59 [crit] 11871#0: _140260577 SSL\_do\_handshake() > > failed (SSL: error:1408A0D7:SSL > > routines:SSL3\_GET\_CLIENT\_HELLO:required cipher missing) while SSL > > handshaking, client: .._.\*, server: 0.0.0.0:443 > > According to the openssl code, this occurs when a client attempts to > resume a session that had made use of previously-enabled ciphers. If > you’re > changing your allowed ciphers frequently this could be why, otherwise a > full cycle of nginx to empty out the session cache seems like it should > resolve this. Reading Richard reply, maybe the client try to resume the session on a different server? (If you can check the logs to see where were the client before the error) ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [February 2, 2015, 8:57pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/4 "2015-02-02T20:57:27Z")
Prior to this issue starting, we had not changed our ciphers in several months. I have tried changing them once since. We have also tried restarting nginx several times on each server to clear the cache, but it has not helped. Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,256406#msg-256406)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [February 2, 2015, 9:26pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/5 "2015-02-02T21:26:52Z")
My first question is do these I have been fighting a similar issue with SSL handshake issues for the past few days. After reboots and upgrades for GHOST, we started seeing errors like this in our error logs constantly: \*579 SSL\_do\_handshake() failed (SSL: error:140A1175:SSL routines:SSL\_BYTES\_TO\_CIPHER\_LIST:inappropriate fallback) while SSL handshaking, in conjunction with an elevated error rate in client requests to nginx in the initial connection phase. I’m not completely sure if the two issues are correlated to be honest, I’m still in the troubleshooting process. I am on a Debian Wheezy system and it started happening with the libssl package 1.0.1e-2+deb7u13 and continues with u14. As soon as I rolled back libssl to u12 and restarted nginx, the logging of errors goes away. I then tested ssl to make sure we weren’t vulnerable to POODLE or Heartbleed, and it’s all clear. I would recommend trying to go back a few versions in libssl, restarting nginx and see if that helps, making sure you’re not leaving yourself open to the major vulnerabilities. Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,256407#msg-256407)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [February 3, 2015, 8:05pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/6 "2015-02-03T20:05:24Z")
Eric, Did you try to downgrade your libssl to the previous version I mentioned earlier? Would love to hear if your issues go away. Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,256428#msg-256428)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [February 3, 2015, 7:19pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/7 "2015-02-03T19:19:15Z")
I just finished running an experiment that has shed some light on the issue. It has not yet been solved though. I setup another nginx server with the same configuration with an upstream app that always responds with HTTP 200. I included JS on each page load in production to make a single request to this server. I ran tcpdump on the test server and what I found was very interesting. Client connections producing the above “inappropriate fallback” on the test server all appear to do some form of the following: (Client and Server successfully complete 3-way handshake) Client: Client Hello TLSv1.2 Server: RST Client: ACK Server: RST (Client and Server successfully complete 3-way handshake) Client: Client Hello TLSv1.1 Server: RST Client: ACK Server: RST (Client and Server successfully complete 3-way handshake) Client: Client Hello TLSv1.0 Server: Encrypted Alert (Content Type: Alert (21)) (Client sends RST, which the server acknowledges, and the connection ends) I don’t know what the alert is, but I can only assume it’s related to TLS\_FALLBACK\_SCSV since the client closes the connection right after. What’s interesting here is that there is little consistency to these RSTs. Sometimes a client downgrades to TSLv1.1 before getting the Encrypted Alert (Content Type: Alert(21)). Sometimes a client tries the same version over and over again, each time getting an RST from the server, and eventually gives up. Later many of these IP addresses are observed establishing successful connections. Am I correct to assume Nginx is sending these RST packets? Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,256427#msg-256427)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [February 3, 2015, 9:41pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/8 "2015-02-03T21:41:35Z")
> (Client and Server successfully complete 3-way handshake) > Client: Client Hello TLSv1.0 > Server: Encrypted Alert (Content Type: Alert (21)) > (Client sends RST, which the server acknowledges, and the connection ends) Can you reliably reproduce this with specific client software or networks? Can you upload a pcap file this failed handshake somewhere for further inspection? ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [February 4, 2015, 3:43am UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/9 "2015-02-04T03:43:52Z")
The errors went away, and now the only errors I see in our logs relating to SSL are handshake timeouts when I turn debug logs on. Now that I think about it, though, isn’t this to be expected? The errors immediately went away as soon as I downgraded far enough back to a version of OpenSSL that didn’t support TLS\_FALLBACK\_SCSV. That doesn’t address why the connections are getting reset and clients are downgrading in the first place, though. Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,256434#msg-256434)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [February 6, 2015, 7:50pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/10 "2015-02-06T19:50:33Z")
We’ve been unable to reproduce it with any one browser or IP address. It really is very intermittent. Fortunately, I believe we’ve gotten to the bottom of this. It looks like our data center switched us over to anti-DDoS route. This means all of our traffic has been passing through hardware that performs heavy packet filtering. The packet loss was causing a lot of confusion for both server and clients. The TLS version fallback that some browsers do upon an unsuccessful handshake made it all the more confusing, since these errors get logged as SSL errors in nginx logs. Posted at Nginx Forum: > **[Re: RE: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,256492#msg-256492)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [February 4, 2015, 3:48am UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/11 "2015-02-04T03:48:48Z")
You are absolutely correct, but I figured you would want a working environment while we work with nginx/openssl on figuring out how to fix this bug. Knowing that it worked for you also increases my own comfort that the issue is mitigated on my side and I won’t have performance issues at my next peak time. Thank you so much for the pcap stuff, I’m sure the information you will provide to Lukas will be invaluable! Way to lead the charge! Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,256435#msg-256435)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [February 7, 2015, 12:30am UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/12 "2015-02-07T00:30:48Z")
> We’ve been unable to reproduce it with any one browser or IP address. It > really is very intermittent. Fortunately, I believe we’ve gotten to the > bottom of this. It looks like our data center switched us over to anti-DDoS > route. This means all of our traffic has been passing through hardware that > performs heavy packet filtering. The packet loss was causing a lot of > confusion for both server and clients. The TLS version fallback that some > browsers do upon an unsuccessful handshake made it all the more confusing, > since these errors get logged as SSL errors in nginx logs. So a MITM security device basically did a TLS downgrade attack here, which the new fallback extension successfully prevented. Thats a good thing, it means it works. ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [March 20, 2015, 7:17pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/13 "2015-03-20T19:17:19Z")
I had to start looking at this issue again now that yet another openssl security issue. Now that I know I can go back to a working setup just by downgrading SSL, I am able to gather more information. This morning, I updated the libssl libraries and restarted nginx, and the errors started flooding back. This time, I took a packet capture to see what was happening and what I could correlate. I run a set of servers that handle API requests from a mobile phone application, and every single client that produced this error was running iOS. In the packet capture, we offer the same cipher that the clients always use without a problem, but for some reason, some of our iPhone clients have issues (not all.) I have been unable to discern a pattern, but it’s always iPhones and doesn’t seem to have anything to do with the device model or the OS version. I haven’t found a single Android instance of the IP’s that show up in our error logs, and we have slightly more Android devices than iOS devices. We get the Client Hello which has a list of 37 potential ciphers for TLS 1.2. We send the server hello and offer the normal cipher. The client, instead of continuing on, immediately sends a FIN, ACK. It then tries to connect again over TLS 1.0, gives the client hello, we send the ACK and almost immediately, WE send a FIN, ACK to the client. Since it’s an API and there are multiple requests being made from the client, not every one will fail. Some negotiate SSL just fine, others do not. I’m still digging through the packet captures to try and figure out any other patterns. As soon as I downgrade libssl, everything works fine. Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,257499#msg-257499)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [March 21, 2015, 3:54pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/14 "2015-03-21T15:54:44Z")
Hello! On Fri, Mar 20, 2015 at 02:15:42PM -0400, tempspace wrote: > In the packet capture, we offer the same cipher that the clients always use > connect again over TLS 1.0, gives the client hello, we send the ACK and > almost immediately, WE send a FIN, ACK to the client. So it looks like th fallback prevention part looks like it should - the inappropriate fallback is prevented. The question now is why fallback happens at all, that is - why the client sends a FIN. It might be some specific cipher which causes the problem - you may try switching ssl\_prefer\_server\_ciphers to off (the default) to see if it helps, and/or playing with ciphers supported (again, default will be a good starting point). – Maxim D. > **[nginx](http://nginx.org/)** ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [March 20, 2015, 6:58pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/15 "2015-03-20T18:58:46Z")
I am seeing similar error as well. It is showing up for lot of people and am not sure why it is happening and if actually the clients facing the error are actually able to browse through the website or not. Can someone please help me understanding that is it safe to downgrade to the earlier version of libssl? and does it solve the problem of client unable to connect (if that happens) in this case? Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,257497#msg-257497)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [March 21, 2015, 4:52pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/16 "2015-03-21T16:52:23Z")
Maxim, I have been playing with the ciphers as well, and it doesn’t appear to be cipher related. It happens for every cipher I’ve tried. I tried with turning off the prefer on the server, and it uses the same cipher with the prefer on. I then turned prefer server ciphers back on, and tailed our access logs which show which cipher was used for the communication. I then went through cipher by cipher, disabled the cipher in our config and restarted nginx each time. None of them had any difference, we’re still seeing lots of fallbacks exclusively from our iOS clients. I tried the following ciphers to no avail: ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA DHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,257522#msg-257522)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [March 21, 2015, 5:00pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/17 "2015-03-21T17:00:16Z")
I should specify that I agree with what is happening. We have clients that are falling back under normal conditions, and the latest libssl that implemented fallback prevention for TLS is stopping. I have downgraded our libssl and I’m looking in my logs, and I see plenty of iOS 8 devices that auto-negotiate to TLS 1.2 that end up with a TLS 1.0 session. When the new libssl is installed, these connections get blocked. Is there a way to turn off the fallback prevention for TLS on the server side while we try to figure out what’s happening? Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,257524#msg-257524)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [March 22, 2015, 2:14am UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/18 "2015-03-22T02:14:12Z")
Hello! On Sat, Mar 21, 2015 at 11:59:17AM -0400, tempspace wrote: > I should specify that I agree with what is happening. We have clients that > are falling back under normal conditions, and the latest libssl that > implemented fallback prevention for TLS is stopping. I have downgraded our > libssl and I’m looking in my logs, and I see plenty of iOS 8 devices that > auto-negotiate to TLS 1.2 that end up with a TLS 1.0 session. When the new > libssl is installed, these connections get blocked. > > Is there a way to turn off the fallback prevention for TLS on the server > side while we try to figure out what’s happening? Looking though OpenSSL code - I don’t think it’s possible without OpenSSL code changes. Changes will be trivial though. – Maxim D. > **[nginx](http://nginx.org/)** ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [March 26, 2015, 7:43pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/19 "2015-03-26T19:43:10Z")
That surely helps. So as of now the only way to resolve the issue is going back to u12 version of libssl? Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,257705#msg-257705)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... ---
**Author:** ![erubin](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@erubin](https://www.ruby-forum.com/u/erubin)\ **Post date:** [May 8, 2015, 4:50pm UTC](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027/20 "2015-05-08T16:50:21Z")
First off, thanks to all who contributed to this thread. I must admit I did not understand much of it, however as someone plagued by this bug (we have a bunch of cherrypy REST servers talking to iOS and Android clients and have seen a lot of those fallback errors), I must admit I’m a bit of a loss on how to proceed here with regards to the future. Yes, I have downgraded my libssl to deb7u12, however I wonder if the openssl team or debian or anyone capable of fixing this issue for good in future openssl releases is aware of what we found here. How to proceed? Especially in light of a new debian release (not sure whether I can downgrade to deb7u12 on jessie…). Best regards, Michael Lauer. Posted at Nginx Forum: > **[Re: Intermittent SSL Handshake Errors](https://forum.nginx.org/read.php?2,256373,258752#msg-258752)** > > Hi, We are using round-robin DNS to distribute requests to three servers all running identically configured nginx. Connections then go upstream to HAProxy and then to our Rails app. About two weeks ago, users began to experience intermittent SSL... [Next page](https://www.ruby-forum.com/t/intermittent-ssl-handshake-errors/234027.md?page=2)