How to enable OCSP stapling when default server is self-signed?

Server NGINX
1

My web server is intentionally set up to only support virtual hosts and
TLS
SNI. I know that the latter eliminates some ancient web browsers but I
don’t care about those browsers.

I want to enable OCSP stapling and it seems to be configured correctly
in my
test vhost (everything else about SSL already works fine - I get an A on
the
Qualys SSL Labs test) and there are no errors or warnings but “openssl
s_client” always returns:

“OCSP response: no response sent”

Yes, I ran the s_client command multiple times to account for the nginx
responder delay. I was testing OCSP stapling on just one of my domains.
Then I read that the ‘default_server’ SSL server also has to have OCSP
stapling enabled for vhost OCSP stapling to work:

This is a huge problem if I want to enable OCSP for my vhosts because my
‘default_server’ certificate is self-signed (intentional) and running
‘configtest’ with ‘ssl_stapling’ options on the default server, of
course,
results in a warning:

“nginx: [warn] “ssl_stapling” ignored, issuer certificate not found”

Which indicates that it isn’t enabled on the default server and
subsequent
s_client tests (after reloading the config, which, of course, issued the
same warning a second time) on the test vhost confirm that there was
still
no OCSP stapling. It was a long-shot in the first place.

So how do I enable OCSP stapling for my vhosts when the default server
cert
is self-signed? This seems like a potential bug in the nginx SSL
module.

Other useful info: Running nginx 1.6.2 (Stable) built from source. My
‘resolver 127.0.0.1’ line in my config points at a local BIND9 server
that
‘dig myvhostdomain.com @localhost’ confirms is working just fine - so it
isn’t a DNS resolver issue as far as I can tell. The error logs are
quiet
other than the warning I got when I added the OCSP stapling options to
‘default_server’.

Posted at Nginx Forum:

2

Maxim D. Wrote:

I want to enable OCSP stapling and it seems to be configured
nginx
This is a huge problem if I want to enable OCSP for my vhosts
subsequent
switching off ssl_stapling for the server{} block in question.
As I explained, I enabled it as a long-shot. I was expecting to get a
warning and I did. I have, of course, disabled it for the default
server
section.

So how do I enable OCSP stapling for my vhosts when the default
server cert
is self-signed? This seems like a potential bug in the nginx SSL
module.

Just enable ssl_stapling in appropriate server{} blocks.

As far as I can tell, I’m already doing that:

Posted at Nginx Forum:

3

Hello!

On Sun, Apr 05, 2015 at 11:26:19PM -0400, bughunter wrote:

Yes, I ran the s_client command multiple times to account for the nginx
responder delay. I was testing OCSP stapling on just one of my domains.
Then I read that the ‘default_server’ SSL server also has to have OCSP
stapling enabled for vhost OCSP stapling to work:

nginx TLS / SSL configuration options for konklone.com · GitHub

There is no such a requirement.

no OCSP stapling. It was a long-shot in the first place.
This warning indicates that you’ve tried to enable OCSP Stapling
for a server with a certificate whose issuer certificate cannot be
found, therefore the “ssl_stapling” directive was ignored for the
server. To avoid seeing the warning on each start, consider
switching off ssl_stapling for the server{} block in question.

So how do I enable OCSP stapling for my vhosts when the default server cert
is self-signed? This seems like a potential bug in the nginx SSL module.

Just enable ssl_stapling in appropriate server{} blocks.

–
Maxim D.
http://nginx.org/

4

Maxim D. Wrote:

module.
makes you think that it doesn’t work?
Running the ‘openssl s_client’ command only returns “OCSP response: no
response sent” as evidenced here (I’ve replaced the actual domain with
“mydomain.org” in the command):

openssl s_client -servername mydomain.org -connect mydomain.org:443

-tls1
-tlsextdebug -status
CONNECTED(00000003)
TLS server extension “server name” (id=0), len=0
TLS server extension “renegotiation info” (id=65281), len=1
0001 - <SPACES/NULS>
TLS server extension “EC point formats” (id=11), len=4
0000 - 03 00 01 02 …
TLS server extension “session ticket” (id=35), len=0
TLS server extension “heartbeat” (id=15), len=1
0000 - 01 .
OCSP response: no response sent
…

Also, the Qualys SSL labs test indicates OCSP support in the certificate
but
no OCSP stapling for the server.

    ssl_certificate

/var/www/mydomain.org/mydomain.org.chain.pem;

That contains the signed certificate, intermediate CA cert, and root CA
cert
(in that order). PEM format.

    ssl_certificate_key

/var/www/mydomain.org/mydomain.org.key.pem;

That contains the private key. PEM format.

    ssl_trusted_certificate    /var/www/root.certs.pem;

That contains the intermediate CA cert and root CA cert (in that order).
PEM format.

And the OCSP responder itself is working fine because Firefox is working
fine (for the moment) and I can also ping the OCSP responder and access
the
OCSP responder directly using the URL in the certificate from the server
that nginx sits on. The CA’s OCSP responder went down for a few hours a
couple of days ago, which caused my browser (Firefox) to freak out and
deny
access to my own website. At that point I went about figuring out
setting
up OCSP stapling to prevent the issue from reoccurring in the future.
The
certificate has the v3 OCSP extension in it and it points at a valid
location. There aren’t any errors in the nginx logs about attempts to
retrieve OCSP responses and failing. There are no errors, warnings, or
notices during startup of nginx. I’ve reloaded and restarted nginx many
times, rebooted the whole system one time, and run the “openssl
s_client”
command a bunch of times after each “long-shot” configuration adjustment
(and reverted shortly after back to the config you saw in the pastebin).

Posted at Nginx Forum:

5

Hello!

On Wed, Apr 08, 2015 at 02:30:12AM -0400, bughunter wrote:

is self-signed? This seems like a potential bug in the nginx
overcomplicated. Nevertheless, it should work assuming correct
TLS server extension “server name” (id=0), len=0
TLS server extension “renegotiation info” (id=65281), len=1
0001 - <SPACES/NULS>
TLS server extension “EC point formats” (id=11), len=4
0000 - 03 00 01 02 …
TLS server extension “session ticket” (id=35), len=0
TLS server extension “heartbeat” (id=15), len=1
0000 - 01 .
OCSP response: no response sent
…

Note that a connection with a Sertificate Status Request will only
return a status if it is already loaded. If there is no OCSP
status available in the worker process, nginx will return no OCSP
status, but will initiate a request to OCSP responder. That is,
it may take a while before OCSP status will be available - even if
everything works fine.

[…]

And the OCSP responder itself is working fine because Firefox is working
fine (for the moment) and I can also ping the OCSP responder and access the
OCSP responder directly using the URL in the certificate from the server

Note that this isn’t really indicate anything: there are two forms of
OCSP requests, POST and GET. And Firefox uses POST, while nginx
uses GET. Given the fact that the responder was completely broken
just a few days ago - it’s quite possible that it’s still broken
for GETs in some cases.

(and reverted shortly after back to the config you saw in the pastebin).
I would recommend the following:

  • test a trivial config with a single server{} block with the
    certificate and “ssl_stapling on”, nothing more; this should
    rule out problems related to OCSP response verification, as well
    as well as problems related to default vs. non-default server
    you’ve claimed.

  • try using debugging log to see what happens on low level in
    nginx (see A debugging log), and
    tcpdump to see what happens on the wire between nginx and OCSP
    responder.

–
Maxim D.
http://nginx.org/

6

Yes, I ran the s_client command multiple times to account for the nginx
responder delay. I was testing OCSP stapling on just one of my domains.
Then I read that the ‘default_server’ SSL server also has to have OCSP
stapling enabled for vhost OCSP stapling to work:

nginx TLS / SSL configuration options for konklone.com · GitHub

There is no such a requirement.

I have the same problem here.

openssl s_client -servername ${WEBSITE} -connect ${WEBSITE}:443 -tls1
-tlsextdebug -status|grep OCSP

Always returns the following on all virtual hosts no matter on how many
times I try:
OCSP response: no response sent

But as soon that I disable my self-signed default host and restart
Nginx, I
get a successfull repsonse on the second request on all CA signed hosts:
OCSP Response Status: successful (0x0)

Posted at Nginx Forum:

7

Hello!

On Tue, Apr 07, 2015 at 12:26:23AM -0400, bughunter wrote:

[…]

So how do I enable OCSP stapling for my vhosts when the default
server cert
is self-signed? This seems like a potential bug in the nginx SSL
module.

Just enable ssl_stapling in appropriate server{} blocks.

As far as I can tell, I’m already doing that:

### nginx.conf# Generic startup file.user www-data www-data;worker_process - Pastebin.com

The configuration you are testing with seems to be
overcomplicated. Nevertheless, it should work assuming correct
certificates are supplied and OCSP responder works fine. What
makes you think that it doesn’t work?

–
Maxim D.
http://nginx.org/

8

Finally had some time to construct an extremely basic server
configuration
with a default HTTP and HTTPS server and test it. I’m working on a
production server, so there are quite a few requests every second and
therefore the downtime had to be scheduled into a tiny window of
opportunity. I also temporarily compiled and enabled a debug build for
a
few minutes (the log file went nuts). I had ssl_stapling on and no
verification. There was still no OCSP stapling response data or
anything
related to OCSP in the debug logs.

Based on numroo’s earlier response and since I was also able to fiddle
around with the config in production, I decided to temporarly disable
the
default SSL server with the self-signed cert. After reloading the
config,
bam! Instantly OCSP stapling started working as expected (even with
verification turned on). Re-enabling the default SSL server with the
self-signed cert caused OCSP to stop working again.

Posted at Nginx Forum:

9

Hello!

On Sun, Apr 12, 2015 at 12:21:19PM -0400, numroo wrote:

openssl s_client -servername ${WEBSITE} -connect ${WEBSITE}:443 -tls1
-tlsextdebug -status|grep OCSP

Always returns the following on all virtual hosts no matter on how many
times I try:
OCSP response: no response sent

But as soon that I disable my self-signed default host and restart Nginx, I
get a successfull repsonse on the second request on all CA signed hosts:
OCSP Response Status: successful (0x0)

As previously suggested, tests with trivial config and debugging
log may help to find out what goes wrong.

–
Maxim D.
http://nginx.org/

10

Hello!

On Thu, May 07, 2015 at 11:54:21AM -0400, 173279834462 wrote:

[…]

virtual server. If intermediate certificates should be specified in addition
remove the intermediate, we do break the chain and the openssl test
complains loudly.

This depends on how your certificate is issued. If your
certificate is issued directly by root CA certificate, then you don’t
need any extra certs here. If there are some intermediate certs,
then you’ll have to put them also.

When this directive was introduced, almost all certificates were
issued directly by roots. No in most cases intermediate
certificates are additionally required. Either way, this doesn’t
actually change things: think of it as “SSL certificate and
certificate chain” if you want some better mnemonic.

ssl_trusted_certificate → public_ca_certificate

In so doing, the configuration would finally be unambiguous.

Some most obvious issues:

  • the “ssl_” prefix, common one for all ngx_http_ssl_module
    directives, is lost;

  • the term “private certificate” is just wrong, there is no such
    thing.

So no, thanks, doesn’t looks like an improvement for me.

problem 2

If it is true that FF uses POST to read, by default, then this explains
the original problem with OCSP, and the fact that nginx is well configured
and openssl and other browsers do work as expected. Google and other search
engines show that Firefox has been affected by this OCSP problem for a long
time. Perhaps they could start using GET like everybody else?

Unless you are CA and running your own OCSP server, you shouldn’t
care. If you do - you probably already know that not everybody
uses GET for OCSP requests, and most notable exception is OpenSSL
itself. Actually, there are more interoperability problems with
GET OCSP requests than with POST ones, and that’s probably why
security.OCSP.GET.enabled is set to “false” by default. Also note
that it’s a wrong list to suggest changes to Firefox.

–
Maxim D.
http://nginx.org/

11

This depends on how your certificate is issued. If your certificate is
issued directly by root CA certificate, then you don’t need any extra
certs
here. If there are some intermediate certs, then you’ll have to put them
also.
When this directive was introduced, almost all certificates were issued
directly by roots. No in most cases intermediate certificates are
additionally required. Either way, this doesn’t actually change things:
think of it as “SSL certificate and certificate chain” if you want some
better mnemonic.

The fact remains that “ssl_certificate” is singular, and its description
is
less than clear.
So, thank you for the explanation, because it completes the original
description.

Certificate chains are way longer than 2 (leaf + ca) nowadays. CRL
checks
can encompass 20+ nodes.
It is for this reason, the lenght of the chain, that I still remain of
the
opinion that “ssl_certificate” ought to
be limited to the leaf’s own public certificate. The intermediates ought
to
be bundled on a separate file.

Labels…

ssl_certificate_key -----> ssl_private_certificate[…cough…]_key
ssl_certificate 1/2 ------> ssl_public_certificate
ssl_certificate 2/2 -------> ssl_public_intermediate_certificates
ssl_trusted_certificate → ssl_public_ca_certificate

I hate the first two, and definitely prefer the original.
The third could simply be “ssl_intermediates”, and the fourth “ssl_ca”.
Whatever, I think they will stay as they are anyway.

security.OCSP.GET.enabled is set to “false” by default

In my FF it set to “false” too, and flipping it does not make any
difference,
so my local problem is neither with GET nor with POST.

It turns out that the problem is “security.ssl.enable_ocsp_stapling”,
which
is
“true” by default. If I disable it, then FF loads the web sites. If I
re-enable it,
then FF complains again:

Secure Connection Failed
An error occurred during a connection to madreacqua.org.
Invalid OCSP signing certificate in OCSP response.
(Error code: sec_error_ocsp_invalid_signing_cert)

The page you are trying to view cannot be shown because the authenticity
of the received data could not be verified.
Please contact the website owners to inform them of this problem.

If FF is correct, then nginx is returning a bad certificate, and we are
back
to square one.

Is it the bundle of certificates? No, because I have verified the chain
from
nginx,
both by hand and automatically with openssl and libressl.

It is GET instead of POST again? No, it is not, because FF “fails” in
both
cases.

Posted at Nginx Forum:

12

Hello!

On Thu, May 07, 2015 at 02:28:12PM -0400, 173279834462 wrote:

[…]

The page you are trying to view cannot be shown because the authenticity
of the received data could not be verified.
Please contact the website owners to inform them of this problem.

If FF is correct, then nginx is returning a bad certificate, and we are back
to square one.

The “Invalid OCSP signing certificate in OCSP response” likely
means that an OCSP response returned by nginx is signed by an
invalid certificate, at least that’s what written. Unless you’ve
forced nginx to return something invalid using the
ssl_stapling_file directive, it is probably due to a behaviour of
your CA. Ask your CA for more information.

Trivial workaround on nginx side is to switch off ssl_stapling.

–
Maxim D.
http://nginx.org/

13

Note that this isn’t really indicate anything: there are two forms of OCSP
requests, POST and GET. And Firefox uses POST, while nginx uses GET.
Given
the fact that the responder was completely broken just a few days ago -
it’s
quite possible that it’s still broken for GETs in some cases.

To comply with local security policy, we disabled POST globally on all
public-facing servers.
This has the advantage of killing web 2.0 and all of its vulnerabilities
with one simple rule, emphasis on killing web 2.0.
Yes, the sites are read-only, and we just love it that way.

For each vhost,
“ssl_certificate_key” includes the vhost’s private key,
“ssl_certificate” includes the vhosts’s public key (leaf) AND the
intermediate key of the Issuer,
“ssl_trusted_certificate” includes the certificate chain in full (leaf +
intermediate + root CA),
all in PEM format.

The openssl test works as expected:

vhost=“”; echo Q | openssl s_client -CAfile
/path/to/your/local/trust/store/ca-bundle.pem -tls1 -tlsextdebug -status
-connect $vhost:443 -servername $vhost 2>&1 | less

There are two problems.

problem 1

nginx’s “ssl_certificate” (note the singular) is truly a bundle of the
certificate and the intermediate.
In fact, if we remove the intermediate, we break the chain.

The description for “ssl_certificate” is also misleading.

"Specifies a file with the certificate in the PEM format for the given
virtual server. If intermediate certificates should be specified in
addition
to a primary certificate, they should be specified in the same file in
the
following order: the primary certificate comes first, then the
intermediate
certificates. A secret key in the PEM format may be placed in the same
file.
"
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_certificate

Although the above sentence “If intermediate certificates should be
specified” suggests that one may omit the intermediate certificate, in
reality you can only do this if you are the CA. I do not wish to sound
opinionated here, because I am making an effort to stick to the facts:
if we
remove the intermediate, we do break the chain and the openssl test
complains loudly.

Therefore, if your own facts correspond to the above, then the solution
is
to edit nginx’s source to limit “ssl_certificate” to the leaf’s public
key
only, and correct the description accordingly. The intermediate(s) can
be
bundled in a separate file.

It would be easier on the eyes to re-write the keywords as well:

ssl_certificate_key -----> private_certificate
ssl_certificate 1/2 ------> public_certificate
ssl_certificate 2/2 -------> public_intermediate_certificates
ssl_trusted_certificate → public_ca_certificate

In so doing, the configuration would finally be unambiguous.

problem 2

If it is true that FF uses POST to read, by default, then this
explains
the original problem with OCSP, and the fact that nginx is well
configured
and openssl and other browsers do work as expected. Google and other
search
engines show that Firefox has been affected by this OCSP problem for a
long
time. Perhaps they could start using GET like everybody else?

Posted at Nginx Forum:

14

173279834462 Wrote:

Yes, the sites are read-only, and we just love it that way.

certificate and the intermediate.
Module ngx_http_ssl_module
solution is to edit nginx’s source to limit “ssl_certificate” to the
In so doing, the configuration would finally be unambiguous.

problem 2

If it is true that FF uses POST to read, by default, then this
explains the original problem with OCSP, and the fact that nginx is
well configured and openssl and other browsers do work as expected.
Google and other search engines show that Firefox has been affected by
this OCSP problem for a long time. Perhaps they could start using GET
like everybody else?

Umm…please don’t hijack threads. Your issue(s) are not related to the
main thread and are even partially off-topic for nginx. Hijacking
threads
is distracting for those who run threaded clients.

My issue regarding OCSP stapling still remains unresolved.

Posted at Nginx Forum: