502 Bad Gateway/while SSL handshaking to upstream

I have been fighting with this one for some time, would appreciate
some assistance.

in my test environment on ec2, I have:

[ nginx → haproxy ] → [ apache w/ ajp → tomcat ] → [ mysql cluster ]

nginx and haproxy on the same machine, apache and tomcat on the same
machine - and the mysql cluster has 2-4 sql nodes+data nodes.

I have been testing with a single GET request, which exercises all of
the above (ex. ab -n 200 -c 200). the net of the problem is that 95%
of the time, 1/3 to 1/2 of my connections result in a 502 returned by
nginx. I’ve yet to have the full 200 completed requests.

the nginx error.log contains both types of errors below, but there is
a direct relation to the failures - if 93/200 connections result in a
502, I will have exactly 93 SSL handshake errors - so I’ve narrowed
the problem down I believe.

a)
2010/04/23 07:49:43 [error] 18430#0: *364 SSL_do_handshake() failed
(SSL: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or
bad record mac) while SSL handshaking to upstream, client: 174.1.X.X,
server: api.example.com, request: “GET /1.0/getName HTTP/1.0”,
upstream: “https://127.0.0.1:8003/1.0/getName”, host:
api.example.com

b)
2010/04/23 07:49:41 [info] 18430#0: *260 peer closed connection in SSL
handshake while SSL handshaking to upstream, client: 174.10.X.X,
server: api.example.com, request: “GET /1.0/getName HTTP/1.0”,
upstream: “https://127.0.0.1:8003/1.0/getName”, host:
api.example.com

unfortunately, it’s unclear why this is happening. nginx config:

server {
listen 0.0.0.0:443;
server_name api.example.com;

  location ~ /* {
    proxy_pass         https://api_443;
  }

  access_log            logs/api.mobioid.com-access.log main;

  ssl                   on;
  keepalive_timeout     70;
  ssl_certificate       /u/nginx/ssl/api.example.com.crt;
  ssl_certificate_key   /u/nginx/ssl/api.example.com.key;

}

upstream api_443   { server 127.0.0.1:8003; }

and haproxy config on port 8003:

listen api_443 127.0.0.1:8003
mode tcp
option ssl-hello-chk
server s1.example.com 10.242.X.X:443 weight 1 minconn 3
maxconn 125 check inter 20000

api.example.com is a trusted cert, s1.example.com/apache is a
self-signed cert.

a previous email thread by David Taveras last month, asked about
essentially the same problem:

Intermittent SSL errors - SSL_do_handshake() failed (SSL: error:1408C095:SSL routines:SSL3_GET_FINISHED:digest check failed) while SSL handshaking to upstream Reverse proxy mode.. HELP

and my config is also similar. unfortunately no one had responded to his
query.

I’m hopeful the community will be able to lend a hand here!

teege

and I’m running the following versions, meant to include this
originally:

$ nginx/sbin/nginx -v
nginx version: nginx/0.7.65
$ haproxy/bin/haproxy -v
HA-Proxy version 1.4.2 2010/03/17
$ apache/bin/httpd -v
Server version: Apache/2.2.14 (Unix)
$ tomcat/bin/version.sh
Server version: Apache Tomcat/6.0.20

teege