SSL Strangeness

Hi,

I am running 4 mongrels behind nginx with two servers blocks (one ssl
enabled). I have a public facing app that 20-30 users go through each
day (non ssl) and administrative SSL controllers in the app that
people at my office use. I am noticing that by the end of day after a
lot of use of the SSL version of the app, that some requests are
hanging for certain people in the office (Mac OSX and Windows Firefox
2.0.0.5-10). It very rarely hangs on my machine (Linux), but it
happens every now and then. I don’t think the mongrels are dying
because usually when these people encounter the hangs we can go to
other computers (Linux Windows OSX) in the office and hammer on the
SSL portion of site with no problems. Also, the non SSL version of
the site never has any hanging AFAIK.

My server settings look like this (which is mainly Ezra’s Default conf
for mongrels):

server {
server_name www.mydomain.com;
listen 443;

ssl on;
ssl_certificate /var/keys/www.mydomain.com.pem;
ssl_certificate_key /var/keys/www.mydomain.com.np.pem;

client_max_body_size 50M;

doc root

root /var/www/www.mydomain.com/current/public;

vhost specific access log

access_log /var/log/nginx/ssl.www.mydomain.com.access.log main;

this rewrites all the requests to the maintenance.html

page if it exists in the doc root. This is for capistrano’s

disable web task

if (-f $document_root/system/maintenance.html) {
rewrite ^(.*)$ /system/maintenance.html last;
break;
}

location / {

needed to forward user’s IP address to rails

proxy_set_header  X-Real-IP  $remote_addr;

needed for HTTPS

proxy_set_header X-FORWARDED_PROTO https;
proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_max_temp_file_size 0;
proxy_redirect false;# If the file exists as a static file serve

it directly without

running all the other rewite tests on it

if (-f $request_filename) {
  break;
}

check for index.html for directory index

if its there on the filesystem then rewite

the url to add /index.html to the end of it

and then break to send it to the next config rules.

if (-f $request_filename/index.html) {
  rewrite (.*) $1/index.html break;
}

this is the meat of the rails page caching config

it adds .html to the end of the url and then checks

the filesystem for that file. If it exists, then we

rewite the url to have explicit .html on the end

and then send it on its way to the next config rule.

if there is no file on the fs then it sets all the

necessary headers and proxies to our upstream mongrels

if (-f $request_filename.html) {
  rewrite (.*) $1.html break;
}

if (!-f $request_filename) {
  proxy_pass http://mongrel;
  break;
}

}

error_page 500 502 503 504 /500.html;
error_page 413 /413.html;
location = /500.html {
root /var/www/www.mydomain.com/current/public;
}
}

One thing I did happen to notice as well, is if I do performance
testing on SSL with httperf, at first it is decently fast, but on many
sequential runs it consistently degrades until httperf can barely do
any requests. However, even while I do an httperf I can use the SSL
version of the site with different computers. It just hangs on some,
usually towards the end of the day.

Any ideas?

Thanks,
Curtis

forgot to mention that I am running nginx 0.5.33 with the following
use flags on Gentoo:
USE=“pcre perl ssl zlib -debug -fastcgi -flv -imap -status -sub -webdav”

On Wed, Nov 28, 2007 at 06:53:04PM -0800, Curtis Spencer wrote:

SSL portion of site with no problems. Also, the non SSL version of
the site never has any hanging AFAIK.

One thing I did happen to notice as well, is if I do performance
testing on SSL with httperf, at first it is decently fast, but on many
sequential runs it consistently degrades until httperf can barely do
any requests. However, even while I do an httperf I can use the SSL
version of the site with different computers. It just hangs on some,
usually towards the end of the day.

How many nginx worker processes do you use ?
Does nginx eat CPU while the test ?
Could you set

 ssl_session_cache    shared:SSL:10m;

and run httperf again ?

I have been able to recreate the httperf issue on my dev server and on
my production server: both running 0.5.33 and 4 mongrels behind the
nginx.
On the dev server I have 2 work processes with 1024 worker connections
and 6 x 1024 on the production

nginx eats almost no cpu whatsoever during the test, which lends me to
believe it is partially a client issue when interacting with ssl.

httperf is able to perform fast for the first few ssl tests, but then
on subsequent it still goes down to 0.1 req/s. I restart nginx on the
server box and the slowness persists with httperf.

Also, using the ssl_session_cache didn’t make a difference on either
machine.

Any other ideas?

Also here is some info about my SSL setup. One of the pems is a self
signed cert and the other is a verisign, both created with openssl.

$ openssl engine -t
(dynamic) Dynamic engine loading support
[ unavailable ]
$ openssl ciphers
DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:AES256-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DES-CBC3-SHA:DES-CBC3-MD5:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES128-SHA:IDEA-CBC-SHA:IDEA-CBC-MD5:RC2-CBC-MD5:RC4-SHA:RC4-MD5:RC4-MD5:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA:DES-CBC-MD5:EXP-EDH-RSA-DES-CBC-SHA:EXP-EDH-DSS-DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:EXP-RC2-CBC-MD5:EXP-RC4-MD5:EXP-RC4-MD5

-Curtis

On Don 29.11.2007 13:17, Curtis Spencer wrote:

I have been able to recreate the httperf issue on my dev server and on
my production server: both running 0.5.33 and 4 mongrels behind the
nginx.
[snipp]
Any other ideas?

can you try another tool, just to check if this is not a tool issue :wink:

ab
curl-loader
http_load
.
.
.

Cheers

Aleks

My only worry is that the original problem stemmed from firefox’s
hanging on ssl requests, and it seemed that the problem was being
duplicated in some respect with httperf. Does anyone know of any
issues with SSL hanging within firefox? Are there any settings in
nginx that would force a new SSL handshake to maybe fix things. I
am really out of my domain when it comes to SSL.

Ok, so here is an update.

  1. I tried running that test with ab, and I didn’t get the httperf
    issue where the SSL requests started taking forever, so probably means
    it is an httperf tool issue with --ssl option. Strange…

  2. I still encounter the issue where SSL requests hang indefinitely
    for some firefox users in my office. I dug a little deeper and I found
    that people around the internet are having issues with Mozilla Firefox
    2.0 and having the Use TLS 1.0 set to checked in the preferences and
    negotiating SSL connections with secure servers. Everyone in my
    office who was having the problem was using Mozilla Firefox 2.0, so I
    had them all disable the TLS 1.0 settings. I am going to watch and
    see what happens over the next few days.

This brings up the issue. Has anyone encountered this TLS issue as
well, and is there a server setting I can set on nginx to prevent
Firefox from even trying to use TLS 1.0 (if this is even the problem)?

Thanks,
Curtis

On Mon, Dec 10, 2007 at 03:12:54PM -0800, Curtis Spencer wrote:

Ok, so here is an update.

  1. I tried running that test with ab, and I didn’t get the httperf
    issue where the SSL requests started taking forever, so probably means
    it is an httperf tool issue with --ssl option. Strange…

Could you how you run httperf ? I will try to reproduce it in my
environment.

well, and is there a server setting I can set on nginx to prevent
Firefox from even trying to use TLS 1.0 (if this is even the problem)?

You may only disable TLSv1 at all:

ssl_protocols SSLv2 SSLv3;

The no way to find out a browser before SSL handshake will be done.
This is the same case as it was with name-based virtual hosts.

Hi,

On Dec 11, 2007 12:00 AM, Igor S. [email protected] wrote:

Could you how you run httperf ? I will try to reproduce it in my
environment.

httperf --server=www.mydomain.com --server-name=www.mydomain.com
–uri=/public/index --ssl --num-conns=10 --num-calls 10

I can run this a few times at a decent speed, but the more I do it it
just degrades until it will take about 10 minutes to finish. Even on
just an index page. I ran it using httperf-0.8 compiled Sep 8 2006
without DEBUG without TIME_SYSCALLS.

Let me know if you need me to try some more examples with it.

This brings up the issue. Has anyone encountered this TLS issue as
well, and is there a server setting I can set on nginx to prevent
Firefox from even trying to use TLS 1.0 (if this is even the problem)?

OK, so even after disabling TLS on the firefox that has the issue,
there is still the slowdown for the people in the office today. I
will give the server change a try and see what happens. Should I also
set the ssl_prefer_server_ciphers configuration setting as well? Are
there any other browser issues that may be causing this?

You may only disable TLSv1 at all:

ssl_protocols SSLv2 SSLv3;

The no way to find out a browser before SSL handshake will be done.
This is the same case as it was with name-based virtual hosts.

What is the downside to this?

Thanks,
Curtis

On Tue, Dec 11, 2007 at 12:16:01PM -0800, Curtis Spencer wrote:

I can run this a few times at a decent speed, but the more I do it it
just degrades until it will take about 10 minutes to finish. Even on
just an index page. I ran it using httperf-0.8 compiled Sep 8 2006
without DEBUG without TIME_SYSCALLS.

Let me know if you need me to try some more examples with it.

OK, I will try.

well, and is there a server setting I can set on nginx to prevent
Firefox from even trying to use TLS 1.0 (if this is even the problem)?

OK, so even after disabling TLS on the firefox that has the issue,
there is still the slowdown for the people in the office today. I
will give the server change a try and see what happens. Should I also
set the ssl_prefer_server_ciphers configuration setting as well? Are

ssl_prefer_server_ciphers was created to force client to use ciphers
that can be accelerated by hardware SSL engine.

If these directives are set:

ssl_prefer_server_ciphers on;
ssl_ciphers AES128-SHA:DES-CBC3-SHA:!EXPORT56:RC4+RSA:+SSLv2:+EXP;

and Soekris vpn1401 is used, then MSIE and Opera will use DES-CBC3-SHA
and Firefox and Konqueror will use AES128-SHA.

there any other browser issues that may be causing this?

I do not know.

You may only disable TLSv1 at all:

ssl_protocols SSLv2 SSLv3;

The no way to find out a browser before SSL handshake will be done.
This is the same case as it was with name-based virtual hosts.

What is the downside to this?

TLSv1 is almost the same as SSLv3.