Hey folks,
We’ve got the following scenario occurring in production. The solution
is
non-obvious… to me, at least.
[client] ==> [nginx] ==> [service S]
- nginx fronting 2 application servers which provide a service S.
(Primary
and backup, both running Trinidad. Trinidad is a JRuby/Rails wrapper for
Tomcat.)
- hot-deploying a rails (jruby) app into Trinidad causes Trinidad to
return
a 404 to nginx
- nginx returns the 404 to the application. In this particular case, the
client is another service which expects service S to remain live during
deployments
So, nginx does provide an “http_404” case for the “proxy_next_upstream”
directive. However, this would require the “max_fails” setting to
pertain to
404s, which it doesn’t… otherwise legitimate 404s produce an infinite
loop.
Is there something like “max_fails” for 404s?
Is there another solution to this problem?
Is it Trinidad’s fault for returning 404s and not 503s? (I would say it
is
but I can’t find a solution to that problem just yet.)
Thanks!
-steven
Hello,
From my understanding you don’t even want to set max_fails. From the
wiki (http://wiki.nginx.org/HttpUpstreamModule#server):
“max_fails = NUMBER - number of unsuccessful attempts at communicating
with the server within the time period (assigned by parameter
fail_timeout) after which it is considered inoperative”
While you hot-deploy server1 it will return 404. Because of
“proxy_next_upstream http_404”, nginx will try server2 (which is
assumed to work). So, client sees response from server2.
When you’re done with hot-deploying on server1, it should start
returning non-404 and you can safely hot-deploy server2.
Of course, as you are asking, this might be the configuration you
already have and it might not have this (expected) behavior. If you
have proxy_next_upstream http_404 in your configuration and it’s not
working this way, you’ll have to provide logs corresponding to the
hot-deployment if you want people here to give you a hand.
A.
–
Take a break. Visit nginx-land: http://www.nginx-discovery.com
Hello!
On Wed, May 18, 2011 at 02:35:49PM -0500, Steven Deobald wrote:
Hey folks,
We’ve got the following scenario occurring in production. The solution is
non-obvious… to me, at least.
[client] ==> [nginx] ==> [service S]
- nginx fronting 2 application servers which provide a service S. (Primary
and backup, both running Trinidad. Trinidad is a JRuby/Rails wrapper for
Tomcat.)
Do you mean “backup” as in “server … backup;” in upstream
definition?
Is there something like “max_fails” for 404s?
Is there another solution to this problem?
Is it Trinidad’s fault for returning 404s and not 503s? (I would say it is
but I can’t find a solution to that problem just yet.)
Parameter max_fails is to mark backends down, and you probably
don’t want to mark backends down just because of some
real/legitimate 404’s.
On the other hand, using “server … backup” indeed will cause
infinite loop with “proxy_next_upstream http_404”. It’s a bug, it
should only ask each backend once.
Workaround is to don’t use “server … backup” but use either
“weight=” instead (big one for real server, small one for backup
one) or use error_page-based fallback instead.
Maxim D.
Do you mean “backup” as in “server … backup;” in upstream definition?
I do. And thank you for the detailed reply – this was exactly the
information I was looking for!
Cheers,
-steven