Nginx forum returns 502 (2014-03-25T08:13Z)

http://forum.nginx.org/ fails to return proper content.

B. R.

On Mar 25, 2014, at 1:14 AM, B.R. [email protected] wrote:

http://forum.nginx.org/ fails to return proper content.

It looks like it was transient or Jim fixed it up. Thanks for the
report!

sarah

Hello,

On 3/25/14, 10:27 AM, Sarah Novotny wrote:

On Mar 25, 2014, at 1:14 AM, B.R. <[email protected]
mailto:[email protected]> wrote:

http://forum.nginx.org/ fails to return proper content.

It looks like it was transient or Jim fixed it up. Thanks for the report!

There was a segmentation fault in the upstream about 14 minutes before
this was sent but that was restarted automatically, and long before I
woke up.

Other than that, without more specific information (exact request, IP,
approximate time), it’s hard to say much. All seems to be functioning as
expected.


Jim O.

“Never argue with a fool, onlookers may not be able to tell the
difference.” - Mark Twain

Hello,

Still down from my side. I do not have cache anywhere, since my browser
purges it on every shutdown.
If you look at your frontend logs around the time I specified in the
subject, you should notice some 502 being thrown. Do you really need my
IP
address? Any 502 you would find under whatever circumstances would not
be a
good sign anyway… ;o)

I join a transcript of the HTTP requests/answer of the latest attempt.
You’ll notice that the favicon is perfectly served and that content type
for main content is correctly set.

If I can be of any further help…

B. R.

Attachment, attachment…

B. R.

Hello,

On 3/25/14, 4:57 PM, B.R. wrote:

Hello,

Still down from my side. I do not have cache anywhere, since my browser
purges it on every shutdown.
If you look at your frontend logs around the time I specified in the
subject, you should notice some 502 being thrown. Do you really need my
IP address? Any 502 you would find under whatever circumstances would
not be a good sign anyway… ;o)

If you would like me to track this down, then I need your IP address, or
at least enough of it to localize your requests and I need the time
(UTC +/- as I do not know that T08:13Z means) so that I can track down
the error. I’m busy, I do this for free and during my free time, and I
am trying to help you.

Since you seem to be the only one complaining as of this moment, I’m not
going to spend a lot of my time on this if you don’t want to cooperate.
Feel free to send it to me off-list if your privacy is so important to
you and you believe that disclosing your local IP is of any interest
to anyone reading this.

I join a transcript of the HTTP requests/answer of the latest attempt.
You’ll notice that the favicon is perfectly served and that content type
for main content is correctly set.

If I can be of any further help…


Jim O.

“Never argue with a fool, onlookers may not be able to tell the
difference.” - Mark Twain

Hello,

On 3/25/14, 6:50 PM, Jonathan M. wrote:

On 25 March 2014 21:17, Jim O. [email protected] wrote:

[what does] that “T08:13Z” mean

The Z suffix indicates UTC, as per section 2 of
http://www.ietf.org/rfc/rfc3339.txt.

HTH,
Jonathan

Thank you for educating me.

It’s a busy server and I turn over logs frequently so unfortunately
those requests are gone.

However, it seems that there are about half a dozen 502’s this morning
and I’m at a bit of a loss to explain them other than they all are
coming from IPv6 addresses.

A little background perhaps, and maybe someone can offer assistance.

I’ve run a variety of PHP apps over the years. My preferences is to use
FPM, but for some reason lately on this server, it has had several
episodes where it just stops accepting new requests. It’s running, it
just doesn’t seem to “see” the requests. I hadn’t made any changes to
the configuration, and use more or less the same configuration on a
bunch of other machines with no issue. About three or four days ago I
switched the forum to use an apache22 backend for PHP requests, with
RPAF enabled while I looked into the FPM issue. Of course that has gone
swimmingly, except that IPv6 requests seem to be generating 502’s.
Looking at the apache logs I don’t even see the requests. Of course, FPM
also hasn’t experienced the same issue either in that time.

I believe it has something to do RPAF handles IPv6 addresses. For now I
have put the forum back on a PHP-FPM backend while I try to sort this
out. If I continue to have this problem I’ll move the forum to another
server if need be.


Jim O.

“Never argue with a fool, onlookers may not be able to tell the
difference.” - Mark Twain

Thanks Jonathan for the pointer. I was going to throw some WIkipedia
page
about it but RFC is definitely the best!

I am glad you discovered I was not alone Jim, thought I am a little sad
that you were that harsh welcoming my input as, if I may, I also did the
reporting on my free time, and without it I am unsure whether you would
have investigated anything?
On a side note, I know what working for free in whatever structure is. I
just rarely used that as an excuse to take the right of sending people
of
good faith away.

I do not know if those 502 are directly tied to some IPv6 request, since
I
have both v4/v6 connectivity. I do not know how switch/fallback between
versions occur during normal browsing.
I recovered access to the URL, but following your explanation the 502
were
due to the temporary apache setup.

  • I know little about Apache and especially RPAF module, but Googling a
    little around I found that:
    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=726529
    I do not know if that would help, but who knows? If you are using
    Debian,
    it seems there was an update at the end of January.

  • For PHP stopping accepting requests, since it is not crashing and you
    confirm the traffic volume is high, I would bet on the exhaustion of
    threads being able to accept new requests. Looks like the usual
    symptoms.
    Ironically, I found some answers on the Nginx ML archive that would help
    improving threads pool and PHP jobs execution time limit:
    php-fpm stops responding

I hope you will find your way out of there,

B. R.

On 25 March 2014 21:17, Jim O. [email protected] wrote:

[what does] that “T08:13Z” mean

The Z suffix indicates UTC, as per section 2 of
http://www.ietf.org/rfc/rfc3339.txt.

HTH,
Jonathan

Jonathan M.
Oxford, London, UK
http://www.jpluscplusm.com/contact.html

Maybe you need to switch over to nginx :slight_smile:

Posted at Nginx Forum:

On Wed, 26 Mar 2014 14:56:09 +0100 B.R. wrote:

it seems there was an update at the end of January.
Yeah, vanilla mod_rpaf-0.6 does not handle IPv6 addresses well.

Be careful with the patch you choose, some fix the textual
representation of REMOTE_ADDR but still break on Apache-side access
control (e.g. on mis-match between proxy connection address family and
header-passed address family).

The patch I’m using successfully here is inlined below.

Bruno

  • For PHP stopping accepting requests, since it is not crashing and you
    confirm the traffic volume is high, I would bet on the exhaustion of
    threads being able to accept new requests. Looks like the usual symptoms.
    Ironically, I found some answers on the Nginx ML archive that would help
    improving threads pool and PHP jobs execution time limit:
    php-fpm stops responding

I hope you will find your way out of there,


diff -NurpP a/mod_rpaf.c b/mod_rpaf.c
— a/mod_rpaf.c 2014-02-17 09:21:08.278411786 +0100
+++ b/mod_rpaf.c 2014-02-17 10:20:18.083940819 +0100
@@ -173,6 +173,7 @@ static int change_remote_ip(request_rec
}

     if (fwdvalue) {
  •        apr_sockaddr_t *tmpsa;
           rpaf_cleanup_rec *rcr = (rpaf_cleanup_rec 
    

*)apr_pcalloc(r->pool, sizeof(rpaf_cleanup_rec));
apr_array_header_t arr = apr_array_make(r->pool, 0,
sizeof(char
));
while (*fwdvalue && (val = ap_get_token(r->pool, &fwdvalue,
1))) {
@@ -184,7 +185,8 @@ static int change_remote_ip(request_rec
rcr->r = r;
apr_pool_cleanup_register(r->pool, (void *)rcr,
rpaf_cleanup, apr_pool_cleanup_null);
r->connection->remote_ip = apr_pstrdup(r->connection->pool,
((char **)arr->elts)[((arr->nelts)-1)]);

  •        r->connection->remote_addr->sa.sin.sin_addr.s_addr = 
    

apr_inet_addr(r->connection->remote_ip);

  •        if (apr_sockaddr_info_get(&tmpsa, r->connection->remote_ip, 
    

APR_UNSPEC, r->connection->remote_addr->port, 0,
r->connection->remote_addr->pool) == APR_SUCCESS)

  •            memcpy(r->connection->remote_addr, tmpsa, 
    

sizeof(apr_sockaddr_t));
if (cfg->sethostname) {
const char *hostvalue;
if ((hostvalue = apr_table_get(r->headers_in,
“X-Forwarded-Host”))) {

Hello,

On 3/26/14, 9:56 AM, B.R. wrote:

I do not know if that would help, but who knows? If you are using
Debian, it seems there was an update at the end of January.

I saw that and others like it. Not using Debian but I suspect that RPAF
is the culprit. I could test it, but then I have to scour the logs to
find the IP’s of spammers and the like. They’d all show up as 137.0.0.1.

I could simply have Igor delete the AAAA record and that would solve the
problem for people with dual IPv4/IPv6 connectivity, but I’m not excited
to do that. I’ve been serving the forum on IPv6 for years. It is,
however an option, at least for now.

  • For PHP stopping accepting requests, since it is not crashing and you
    confirm the traffic volume is high, I would bet on the exhaustion of
    threads being able to accept new requests. Looks like the usual symptoms.
    Ironically, I found some answers on the Nginx ML archive that would help
    improving threads pool and PHP jobs execution time limit:
    php-fpm stops responding

That’s not the case. Not even close to out of resources. The FPM
children are there, they’re “listening”, they’re just not “hearing”.


Jim O.

“Never argue with a fool, onlookers may not be able to tell the
difference.” - Mark Twain

Hello,

On 3/26/14, 10:02 AM, itpp2012 wrote:

Maybe you need to switch over to nginx :slight_smile:

Haha. Did I miss the announcement that someone wrote a PHP handler for
nginx? :wink:

Seriously, perhaps you misunderstood, or perhaps, as I suspect, this is
a joke. I am using nginx (1.5.12).


Jim O.

“Never argue with a fool, onlookers may not be able to tell the
difference.” - Mark Twain

Hello,

On 3/26/14, 10:10 AM, Bruno Prmont wrote:

Bruno

  •        apr_sockaddr_t *tmpsa;
            if (cfg->sethostname) {
                const char *hostvalue;
                if ((hostvalue = apr_table_get(r->headers_in, 
    

“X-Forwarded-Host”))) {

Thank you Bruno! I will try this a bit later, when things have settled
down here.


Jim O.

“Never argue with a fool, onlookers may not be able to tell the
difference.” - Mark Twain