Multiple Upstream Servers Result in Error 310 (net::ERR_TOO_MANY_REDIRECTS)

I have an Nginx reverse-proxy sitting in front of two JBoss servers. If
I attempt to add the second JBoss server to the upstream directive, I
receive an Error 310 (net::ERR_TOO_MANY_REDIRECTS) from Chrome, IE just
displays an “Internet Explorer cannot display this webpage” error. If I
comment the second upstream server out a kill -HUP the master process,
everything works properly.

I am attempting to use Nginx to rewrite all HTTP requests to HTTPS and
upon first contact with our domain, the JBoss application redirects you
to https://my.domain.com/home.

Here is my default config file:

upstream jboss_dev_servers {
server 10.0.3.15:8080;
server 10.0.3.16:8080;
}

server {
listen 10.0.3.28:80;
server_name my.domain.com;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_next_upstream error timeout invalid_header;
rewrite ^ https://$server_name$request_uri? permanent;
proxy_pass http://jboss_dev_servers;
}
}

server {
listen 10.0.3.28:443 default ssl;
ssl on;
ssl_certificate /srv/ssl/combined.crt;
ssl_certificate_key /srv/ssl/combined.key;
ssl_protocols SSLv3 TLSv1;

    server_name my.domain.com;


    location / {
            proxy_set_header Host $host;
            proxy_set_header  X-Real-IP  $remote_addr;
            proxy_next_upstream error timeout invalid_header;
            proxy_pass http://jboss_dev_servers;
    }

}

Any ideas what could be causing this? Is it bouncing the requests back
and forth between the two upstream servers somehow?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,223535,223535#msg-223535

On 03/06/2012 01:08 PM, mevans336 wrote:

server {
listen 10.0.3.28:80;
server_name my.domain.com;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_next_upstream error timeout invalid_header;
rewrite ^ https://$server_name$request_uri? permanent;
proxy_pass http://jboss_dev_servers;
}
}

Why does the application do a redirect if you are already doing one in
your Nginx config? It’s not clear why you even bother with the
proxy_pass in this location. Just redirect to the HTTPS location and be
done.

Also, be sure the application is properly detecting the scheme. In
your config, Nginx is talking to JBoss via HTTP, not HTTPS, so you may
have some issue there, something like proxy_set_header X-Scheme $scheme;
would do the trick.

I have no idea why this would work with a single backend and not two,
unless you are somehow inadvertently running two different versions of
the JBoss code (one with a redirect, one without).

Regards,
Cliff

Cliff,

The application performs a redirect because that was the way it was
originally designed. It uses the ACEGI security framework. I have turned
the ACEGI HTTP -> HTTPS redirect off with the same behavior however.

After my post, I stumbled across the ip_hash upstream variable and that
has resolved the issue. Or at least, masked it and made things
functional.

I am aware that Nginx is talking to JBoss via HTTP, not HTTPS. Nginx is
acting as our front-end SSL termination point, we don’t have JBoss
configured to answer SSL requests, although for security purposes, that
is on my ‘to-do’ list. Right now, clients don’t communicate directly
with the JBoss servers, only Nginx, so communication over the internet
is encrypted.

I am confused by your statement, “It’s not clear why you even bother
with the proxy_pass in this location. Just redirect to the HTTPS
location and be done.” I am new to Nginx, so I followed the Nginx load
balancing example (http://wiki.nginx.org/LoadBalanceExample). Is there
an easier/better way to configure load balancing among multiple upstream
servers?

Thank you for your input,

Matt

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,223535,223539#msg-223539

On 03/06/2012 02:07 PM, mevans336 wrote:

I am aware that Nginx is talking to JBoss via HTTP, not HTTPS. Nginx is
acting as our front-end SSL termination point, we don’t have JBoss
configured to answer SSL requests, although for security purposes, that
is on my ‘to-do’ list. Right now, clients don’t communicate directly
with the JBoss servers, only Nginx, so communication over the internet
is encrypted.
I was just making sure that was considered. In any case I suspect
Francis is right in his assessment, and that’s why using the ip_hash
directive solved your issue. Good work on that. I haven’t personally
encountered frameworks with this issue so it didn’t occur to me.

I am confused by your statement, “It’s not clear why you even bother
with the proxy_pass in this location. Just redirect to the HTTPS
location and be done.”

What I meant is that you are doing a “rewrite permanent”, followed by
the proxy_pass. This means that either the proxy_pass will never be
executed. Sorry I wasn’t clear.

Regards,
Cliff

On 03/06/2012 02:06 PM, Francis D. wrote:

My guess is that “first contact” information is not shared between the
two jboss servers, so if serial requests from a single client don’t
always go to the same server, bad things happen. You should be able to
see whether or not that is the case. With only one jboss server
configured, all requests must go to it, and things should work, as you
reported. All the best, f

Since the ip_hash worked, your assessment seems spot on. However, one
thing that’s still not making sense to me: in his reply, Matt states
that he tested with the HTTPS redirect disabled in JBoss and still had
the same issue. I’m a bit unsure of how that could be (unless it was
still in effect, or the browser remembered the redirect).

Regards,
Cliff

On Tue, Mar 06, 2012 at 04:08:44PM -0500, mevans336 wrote:

Hi there,

I have an Nginx reverse-proxy sitting in front of two JBoss servers. If
I attempt to add the second JBoss server to the upstream directive, I
receive an Error 310 (net::ERR_TOO_MANY_REDIRECTS) from Chrome, IE just
displays an “Internet Explorer cannot display this webpage” error.

If you test using something like “curl”, you’ll see exactly what the
response is, and you can compare it to what you expect it to be. That
tends to make things easier to follow.

If I
comment the second upstream server out a kill -HUP the master process,
everything works properly.

I am attempting to use Nginx to rewrite all HTTP requests to HTTPS and
upon first contact with our domain, the JBoss application redirects you
to https://my.domain.com/home.

What does the JBoss application understand by the phrase “first
contact”? I suspect that that’s going to be related to the fix.

            proxy_set_header  X-Real-IP  $remote_addr;
            proxy_next_upstream error timeout invalid_header;
            rewrite ^ https://$server_name$request_uri? permanent;
            proxy_pass http://jboss_dev_servers;

The “rewrite” there means that the “proxy_pass” will never be used.

curl -i http://my.domain.com/

should show a http 301 redirect to https://my.domain.com/, so that’s
what the client will ask for next.

server {
listen 10.0.3.28:443 default ssl;

    location / {
            proxy_set_header Host $host;
            proxy_set_header  X-Real-IP  $remote_addr;
            proxy_next_upstream error timeout invalid_header;
            proxy_pass http://jboss_dev_servers;

curl -i https://my.domain.com/

should get to here, and be proxied to a jboss server. If that thinks
that this is “first contact”, it will issue a http redirect to
https://my.domain.com/home.

If you modify each jboss server to add a “I’m #1” header, you’ll see
which jboss server you just contacted.

The next thing the client will do is

curl -i https://my.domain.com/home

which will be proxied to a jboss server. If that server thinks
that this is “first contact”, it will issue a http redirect to
https://my.domain.com/home, at which point the client can reasonably
think
“redirect loop; this is broken”.

If that server thinks that this isn’t “first contact”, it will
presumably
return http 200 with some useful content.

Which do you see when you test it like this?

And in this second response, do you see an “I’m #1” or an “I’m #2
header?

(You could look in logs to know which upstream server you were
eventually
talking to, but a header is more immediate.)

Any ideas what could be causing this? Is it bouncing the requests back
and forth between the two upstream servers somehow?

My guess is that “first contact” information is not shared between the
two jboss servers, so if serial requests from a single client don’t
always go to the same server, bad things happen.

You should be able to see whether or not that is the case.

With only one jboss server configured, all requests must go to it,
and things should work, as you reported.

All the best,

f

Francis D. [email protected]

Francis,

Thank you for the wonderfully detailed response. I have printed your
reply to PDF for safekeeping as you covered many things I didn’t know
curl was even capable of. You also provided me a much deeper
understanding of the HTTP redirect process.

I am extremely thankful for both.

Cheers,

Matt

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,223535,223577#msg-223577

On Wed, 2012-03-07 at 07:20 -0500, mevans336 wrote:

Even though the HTTPS redirect was disabled, the application itself
still sends a redirect to /home. For example, if you type in
http://my.domain.com you were successfully redirected to
https://my.domain.com/home. That initial redirect was happening and that
is where the error 310 would occur, at the https://my.domain.com/home
level.

Okay, I must have missed the redirect to /home.

I see that Francis is in agreement with you that the proxy_pass
directive is being negated by the permanent rewrite. Just for my
edification, would you (or anyone else who would like to take a stab at
it) mind confirming that if I remove the proxy_pass directive, I will
still be able to achieve load balancing between the two upstream
servers? If you could explain how load balancing works without that
directive so I have a clearer understanding, I’d be even happier! I have
searched the wiki and my Nginx book, but I’m either missing it, or this
specific scenario is not addressed.

Because you are also doing proxy_pass in the https section. So the flow
goes like this:

  1. request visits http section and is redirected to https section
  2. https section sends request via proxy_pass to load-balanced JBoss
    backends

So you should remove the proxy_pass from the first (HTTP) server block,
but leave the proxy_pass that exists in the second (HTTPS) server block.

Regards,
Cliff

That makes perfect sense, thank you Cliff.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,223535,223632#msg-223632

On Wed, Mar 07, 2012 at 07:23:22AM -0500, mevans336 wrote:

Hi Matt,

I am extremely thankful for both.

You’re welcome.

f

Francis D. [email protected]

Cliff W. Wrote:

things should work, as you
still in effect, or the browser remembered the
redirect).

Regards,
Cliff


nginx mailing list
[email protected]
http://mailman.nginx.org/mailman/listinfo/nginx

Cliff,

Thank you and Francis for your quick and detailed responses.

Even though the HTTPS redirect was disabled, the application itself
still sends a redirect to /home. For example, if you type in
http://my.domain.com you were successfully redirected to
https://my.domain.com/home. That initial redirect was happening and that
is where the error 310 would occur, at the https://my.domain.com/home
level.

I see that Francis is in agreement with you that the proxy_pass
directive is being negated by the permanent rewrite. Just for my
edification, would you (or anyone else who would like to take a stab at
it) mind confirming that if I remove the proxy_pass directive, I will
still be able to achieve load balancing between the two upstream
servers? If you could explain how load balancing works without that
directive so I have a clearer understanding, I’d be even happier! I have
searched the wiki and my Nginx book, but I’m either missing it, or this
specific scenario is not addressed.

Cheers,

Matt

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,223535,223576#msg-223576

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs