No CORS Workaround - SSL Proxy

We run a API web service and have two web sites that access the web
service via AJAX. The web sites are accessed via HTTPS and, for security
reasons, we need to have the API web service also accessed by HTTPS.
Due to the need to support the IE9 browser, which does not properly
support CORS, we are unable to have the web applications on our web
servers configured to access the API web service through a different
hostname than the hostnames of the two web sites. Consequently, we
trick IE9 into thinking the origin host (web site) and destination host
(API service) are on the same host and proxy requests from the web sites
to the web service via proxy_pass. Unfortunately, since the API web
service must be accessed by HTTPS, nginx has to establish an SSL session
with the API web service, because we cannot proxy to HTTP. Our config
looks something like this for simplicity I only show one of the web
sites nginx config.

server {
   listen       443;
   server_name  app.example.com;     // this is the web application
   server_tokens off;

   ssl                  on;
   ssl_certificate      cert.pem;
   ssl_certificate_key  cert.key;

   ssl_session_timeout  5m;

   ssl_protocols  SSLv3 TLSv1 TLSv1.1 TLSv1.2;
   ssl_ciphers  HIGH:!aNULL:!MD5;
   ssl_prefer_server_ciphers   on;

// this URL pattern is interpreted as meaning: forward the request to
the web service running on another host
location /svc/api/ {
proxy_pass https://svc.example.com/api/; // this is
the web service running on another host
proxy_set_header Host svc.example.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}

Location / {
// normal web site access here
}

}

This works fine. However, every once in a while (say, every week or
so), traffic to https://app.example.com/svc/api/xxxx returns gateway 502
errors. The API service (located at https://svc.example.com/api) is
working fine and is accessible directly. However, through the proxy
setup (above), nginx will not pass traffic. Simply restarting nginx
gets it working again for another week or so, only to have it get into
the same state again some random interval later.

Does anyone have any ideas what might be causing nginx to fail to proxy
traffic when no changes to the configuration have been made and the
backend service is functioning normally?

Since I anticipate some will want to tell me that proxying to HTTPS is a
bad idea, please realize we do not have the luxury of talking to the
backend service (which lives on the Internet and is accessed by multiple
parties) via HTTP. Also, yes, I realize that the proxy_set_header stuff
probably has no useful effect with HTTPS proxying.

Thanks much in advance. Eric

Hello!

On Fri, Jun 20, 2014 at 07:32:36PM +0000, Eric Swenson wrote:

[…]

to proxy traffic when no changes to the configuration have been
made and the backend service is functioning normally?

First of all, it may be a good idea to take a look into error log.


Maxim D.
http://nginx.org/

Hello Maxim,

On 6/20/14, 3:46 PM, “Maxim D.” [email protected] wrote:

directly. However, through the proxy setup (above), nginx will
not pass traffic. Simply restarting nginx gets it working again
for another week or so, only to have it get into the same state
again some random interval later.

Does anyone have any ideas what might be causing nginx to fail
to proxy traffic when no changes to the configuration have been
made and the backend service is functioning normally?

First of all, it may be a good idea to take a look into error log.

The only messages in the error log were ones for hours before and for
hours after the problem, of the form:

2014/06/17 17:14:16 [error] 28165#0: *1508 no user/password was provided
for basic authentication, client: 11.12.13.14, server:
app.test.example.com, request: “GET / HTTP/1.1”, host:
app.test.example.com"

There was nothing related to the 502 errors around the time of the first
request that failed with a 502 error code. The access log shows quite a
few requests failing with a 502 status code from that point forward, up
until the time that I restarted nginx.

Eric

Hello!

On Fri, Jun 20, 2014 at 11:29:39PM +0000, Eric Swenson wrote:

This works fine. However, every once in a while (say, every
made and the backend service is functioning normally?

There was nothing related to the 502 errors around the time of the first
request that failed with a 502 error code. The access log shows quite a
few requests failing with a 502 status code from that point forward, up
until the time that I restarted nginx.

If there is nothing in error logs, and you are getting 502 errors,
then there are two options:

  1. The 502 errors are returned by your backend, not generated by
    nginx.

  2. You did something wrong while configuring error logs and/or you
    are looking into a wrong log.

In this particular case, I would suggest the latter.


Maxim D.
http://nginx.org/

Hello Maxim,

On 6/22/14, 7:32 AM, “Maxim D.” [email protected] wrote:

If there is nothing in error logs, and you are getting 502 errors,
then there are two options:

  1. The 502 errors are returned by your backend, not generated by
    nginx.

  2. You did something wrong while configuring error logs and/or you
    are looking into a wrong log.

In this particular case, I would suggest the latter.

Ive verified that my error log are configured fine I do get errors
reported in my configured error log but nothing at the time that nginx
returns 502 errors to the client.

Ive checked the upstream servers logs, even when configured with debug
logging, and never see any requests making it to the upstream server
when
nginx returns a 502 to the client.

If the issue were with the upstream server, why is it that simply
restarting nginx causes everything to proceed normally. I never have to
touch the upstream server (which, by the way is serving other requests
successfully from other proxies at the same time as the nginx proxy that
returns 502s is doing do.

Eric

Hello!

On Tue, Jul 01, 2014 at 05:58:33PM +0000, Eric Swenson wrote:

In this particular case, I would suggest the latter.

I¹ve verified that my error log are configured fine ‹ I do get errors
reported in my configured error log ‹ but nothing at the time that nginx
returns 502 errors to the client.

As nginx has lots of options to control error logging (logging
level based filtering, as well as per-server and per-location
error logs), it’s not enough to check that some errors are logged
to make sure logging is configured properly.

Simplest way to configure logs properly is to comment out all
error_log directives, and add error_log at global level, with
desired logging level. In this case, logging at “error” level
should be enough, i.e., the following should do the trick:

error_log /path/to/log error;

(At global level, i.e., at the top of your nginx.conf file. And
don’t forget to comment out all other error_log directives.)

I¹ve checked the upstream server¹s logs, even when configured with debug
logging, and never see any requests making it to the upstream server when
nginx returns a 502 to the client.

If the issue were with the upstream server, why is it that simply
restarting nginx causes everything to proceed normally. I never have to
touch the upstream server (which, by the way is serving other requests
successfully from other proxies at the same time as the nginx proxy that
returns 502s is doing do.

The fact that reastarting nginx fixes things indicates that the
problem is likely caused by connections already established by
nginx to the upstream server in question. And restarting nginx
fixes things by closing these connections.


Maxim D.
http://nginx.org/

Hello!

On Wed, Jul 02, 2014 at 02:37:17PM +0400, Maxim D. wrote:

nginx.
As nginx has lots of options to control error logging (logging

(At global level, i.e., at the top of your nginx.conf file. And
don’t forget to comment out all other error_log directives.)

Correction: in case of proxy to https, there is one case when the
error is reported at “info” level (“peer closed connection in SSL
handshake”), so “info” level logging is needed to see it.

(This looks like a bug and should be fixed, while talking to
upstream servers proper logging level for SSL handshake errors is
“error”.)

The fact that reastarting nginx fixes things indicates that the
problem is likely caused by connections already established by
nginx to the upstream server in question. And restarting nginx
fixes things by closing these connections.

If the problem is indeed during SSL handshake, it may be caused by
a cached session peer starts to dislike for some reason.
Switching off “proxy_ssl_session_reuse” could help. See also this
answer, which may be related:

http://mailman.nginx.org/pipermail/nginx/2014-July/044329.html


Maxim D.
http://nginx.org/