499 errors, fcgi sockets require restart python and custom proxy

Hi folks,

I posted in the paid help about this too… here is the issue:

We currently have a setup where we have multiple app servers sitting
behind NGINX. We want to provide HMAC based authentication comparing an
hash of the content or headers of the request against an authenticator
using a known key (which varies depending on which URL you are trying to
access).

We receive on port 80.
There are 50 or so sockets setup as a backend which feed into a python
script (gatekeeper.py).
gatekeeper.py has the logic to authenticate the request given params in
the headers, etc
It then makes a request to port :81
:81 is setup in nginx to route to our application servers
The app servers return to nginx, nginx returns to gatekeeper.py and then
gatekeeper.py returns back to nginx.

It has been mostly successful, and fast enough, but we’re having some
issues with fcgi threads timing out, and having to recreate them every
so often. Having an integrated solution would be ideal.

Problem is, we keep getting these in the log:

2009/04/04 20:59:28 29828#0: *754537 upstream timed out (110:
Connection timed out) while reading response header from upstream,
client: 174.36.146.67, server: balancer1.search.xxxxxxxx.net, request:
“GET
/solr/xxxxxxxxxxx/select?qt=mlt&fl=nid%2Ctitle%2Curl&mlt.fl=title%2Ctaxonomy_names&mlt.mintf=1&mlt.mindf=1&mlt.minwl=3&mlt.maxwl=15&mlt.maxqt=30&version=1.2&wt=json&json.nl=map&q=id%3Acdd16c5b4c3f3d823192cc429892d2ed%2Fnode%2F1733&start=0&rows=4&request_id=49d8021634646
HTTP/1.0”, upstream: “fastcgi://unix:/tmp/fcgi-gate-46.socket:”, host:
search.xxxxxx.com

And we get lots of them, and while the requests typically make it
through after trying 4 or 5 threads, it gets worse and worse until I
delete the sockets and recreate them, then it runs absolutely fine,
given the same traffic for hours.

Any ideas? This one is really killing me. Totally open to paying
someone to help with this too.

Best,
Jacob

Posted at Nginx Forum: 499 errors, fcgi sockets require restart python and custom proxy