nginx/KQUEUE breaks proxy_ignore_client_abort

Server NGINX
1

Folks,

If nginx is built on FreeBSD, “proxy_ignore_client_abort on;” has
no/little effect, because TCP half-closes cause a connection drop
even if not speaking to a proxy backend.

Situation: PGP clients talk to PGP keyservers using the HKP protocol,
which is a very light layer over HTTP. In GnuPG, if the cURL library
was not available at build time, a mock-curl “curl-shim” implementation
is used instead. In GnuPG 2.0.x, this code uses a TCP half-close to
indicate when the sender has finished sending. This was a mistake and
has been fixed for the next release, but people running PGP keyservers
need to deal with the existing installed userbase.

For various (good) reasons, the common PGP keyserver software is run
with a reverse proxy in front of it, and nginx is a popular choice.
nginx will default to drop connections on the shutdown, for reasons
previously explained on this list. Enabling proxy_ignore_client_abort
is, as far as I understand matters, supposed to allow these shutdowns to
not be considered an abort.

Temporarily turning on an error log for the :11371 server block (that’s
the HKP default port) gives:

2013/02/28 09:11:54 [info] 34110#0: *51
kevent() reported that client closed connection while waiting for
request,
client: 2a02:898:0:20::57:1, server:
[2a02:898:31:0:48:4558:73:6b73]:11371

“proxy_ignore_client_abort on” avoids enabling logic in
src/http/ngx_http_upstream.c which would log:

kevent() reported that client prematurely closed
connection, so upstream connection is closed too

That’s not the error message here; instead, what we get comes from
src/http/ngx_http_request.c in ngx_http_keepalive_handler().

As far as I can tell, as long as NGX_HAVE_KQUEUE is defined, it is
impossible to avoid this handling.

nginx folks: is this something you’re likely to fix, or is this far
enough outside of supportable behaviour that you consider the current
situation a feature, not a bug?

I’m not sufficiently familiar with the nginx code base to find a fix for
this and don’t currently have time to get familiar, sorry. :frowning:

Thanks,
-Phil

PS: nginx mail-server configuration is broken; it’s checking SMTP
Envelope
Sender against the subscription list, not the RFC5322.From: header,
so
breaks on things such as PRVS. Posting via manual injection to your
mail-server. :frowning:

2

Hello!

On Fri, Mar 01, 2013 at 03:22:51AM -0500, Phil P. wrote:

indicate when the sender has finished sending. This was a mistake and
Temporarily turning on an error log for the :11371 server block (that’s
connection, so upstream connection is closed too

I’m not sufficiently familiar with the nginx code base to find a fix for
this and don’t currently have time to get familiar, sorry. :frowning:

It looks like you are running nginx with experimental SPDY patch,
and it broke things here. Try recompiling nginx without SPDY
patch to see if it helps.

PS: nginx mail-server configuration is broken; it’s checking SMTP Envelope
Sender against the subscription list, not the RFC5322.From: header, so
breaks on things such as PRVS. Posting via manual injection to your
mail-server. :frowning:

Unfortunately, there is no way to properly reject messages at SMTP
level (i.e. to avoid sending bounces) and doing checks based on
message headers at the same time.

If you use different envelope from and message from addresses and
have problems with posting - just subscribe your envelope from
address to the mailing list with mail delivery disabled.

–
Maxim D.
http://nginx.org/en/donation.html

3

[fixed Subject: to help others with issue track it]

On 2013-03-01 at 17:12 +0400, Maxim D. wrote:

It looks like you are running nginx with experimental SPDY patch,
and it broke things here. Try recompiling nginx without SPDY
patch to see if it helps.

That fixed things, thank you.

So, nginx+KQUEUE+SPDY breaks clients which shutdown on the write side,
without the ability to disable treating this as a client abort.

I’ll sacrifice SPDY for now, to have correctness for existing clients.

Do you think that the SPDY patch will change to include something like
proxy_ignore_client_abort or will write-shutdowns just be treated as
unacceptable?

Given that SPDY requires SSL which inherently requires bi-directional
connections at all times, the current behaviour with the SPDY patch is
absolutely correct, if SPDY is enabled for that server. In this case,
it’s a cleartext server so SPDY wasn’t enabled at all.

[I’ll reply to the list administrivia issue in a second email.]

Thanks,
-Phil

4

On 2013-03-01 at 17:12 +0400, Maxim D. wrote:

If you use different envelope from and message from addresses and
have problems with posting - just subscribe your envelope from
address to the mailing list with mail delivery disabled.

I understand the problem you’re fighting here, and why you’re doing this
at SMTP RCPT time, since Mailman doesn’t have content scanning hooks to
check if the message should be allowed based on the message headers.
You can do these checks safely enough, but it requires more caution.

When violating normal SMTP expectations by making a RCPT appear to only
exist for certain MAIL FROM senders, it’s important to understand
variations in senders at SMTP time: the checks you’re doing are not the
same as the membership tests done by Mailman itself, which looks at the
headers.

I did the same thing as you, for expediency and to avoid forking yet
more extra processes for scanning, but I made sure that the form of the
address being checked for membership has had VERP and BATV variations
stripped out first, to check a normalized address against the Mailman
membership roster.

BATV changes the SMTP Envelope Sender, with a crypto-hash embedded in
the address, and a secret and a daily timestamp going into the hash
inputs, so that if all messages from a domain are sent with BATV, then
bounces inherently must be to BATV targets if they’re legitimate.

This is the only tool that prevents joe-job backscatter from flooding
mailboxes.

So that’s a non-standard address-existence test breaking when exposed to
an address variation that does have an Internic draft, albeit expired.

I’ve sucked it up and configured up an exception mechanism, adding this
mailing-list to that, accepting that any time I enable the backscatter
filter, I’ll lose bounce messages from this list to me, with rejections
dropping into a blackhole. That’s got a lower risk of being triggered
than a joe-job (unfortunately) (and this varies depending on your
involvement with email infrastructure and how much spammers dislike
you).

Next time you’re touching your mailserver setup, could you please take a
look at adding a canonicalisation step to the addresses being checked
against list membership?

Thanks,
-Phil

5

On Saturday 02 March 2013 00:56:06 Phil P. wrote:

without the ability to disable treating this as a client abort.
it’s a cleartext server so SPDY wasn’t enabled at all.

SPDY patch also includes many changes for http core of nginx. The one
that
you see, is the unintended result of these changes. I’m going to fix it
in
upcoming revision, since it can break some setups as you have mentioned.

Thank you for the report.

wbr, Valentin V. Bartenev

–
http://nginx.org/en/donation.html

6

Hello!

On Fri, Mar 01, 2013 at 03:59:59PM -0500, Phil P. wrote:

If you use different envelope from and message from addresses and
have problems with posting - just subscribe your envelope from
address to the mailing list with mail delivery disabled.

I understand the problem you’re fighting here, and why you’re doing this
at SMTP RCPT time, since Mailman doesn’t have content scanning hooks to
check if the message should be allowed based on the message headers.
You can do these checks safely enough, but it requires more caution.

You probably didn’t understand the problem deep enough: content
scanning hooks, even if implemented, won’t help. To properly
reject message at SMTP level you have to check envelope sender,
and if you’ve accepted RCPT TO - it’s too late to reject message
at DATA stage, as the message might have other valid recipients.

So the only way to properly check list membership is to check
envelope addresses. Anything else means sending bounces, which is
not acceptable nowadays.

membership roster.
an address variation that does have an Internic draft, albeit expired.
look at adding a canonicalisation step to the addresses being checked
against list membership?

I personally think that BATV is awful, but normalization shouldn’t
make things worse and probably worth implementing.

–
Maxim D.
http://nginx.org/en/donation.html

7

On 2013-03-03 at 02:11 +0400, Maxim D. wrote:

You probably didn’t understand the problem deep enough: content

I’m one of the maintainers of the MTA which runs a plurality of the MTA
installs out there. Of course, I have crazy days and moments of pure
stupidity, but in general I have a decent understanding of email.

scanning hooks, even if implemented, won’t help. To properly
reject message at SMTP level you have to check envelope sender,
and if you’ve accepted RCPT TO - it’s too late to reject message
at DATA stage, as the message might have other valid recipients.

So the only way to properly check list membership is to check
envelope addresses. Anything else means sending bounces, which is
not acceptable nowadays.

Er, the only visible email addresses in nginx.org are for mailing-lists,
I wasn’t aware it had user-accounts. Even so, not a major issue.

So if the poster is a member of one mailing-list but not another, then
you’re not back-scattering to unknown addresses; if content-scanning
then deems the message okay, then it’s far more acceptable to bounce.

And if folks are cross-posting to many lists but only subscribed to one,
rejecting all of them with a “fix your sender address or don’t spam
lists you’re not subscribed to” message works well too. :slight_smile: This even
applies for user-accounts: protects from the crazies mailing every
address they can think of.

But yes, I understand the issue and do use RCPT-time checks myself,
after normalisation of the sender address, to work around the fact that
I’m doing something a little dodgy that might break legitimate mail.

Regards,
-Phil

8

Hello!

On Mon, Mar 04, 2013 at 05:33:42PM +0400, Sergey B. wrote:

it actually breaks what is batv used for, because bounce if any will be
sent to normalized address. It looks like catch-22 situation.

Addresses should be normalized only for lookup in the database,
and only in_addition to the normal lookup (to make sure we don’t
reject correctly subscribed address which looks like BATV one and
normalizes to something wrong).

–
Maxim D.
http://nginx.org/en/donation.html

9

On 3 Mar2013, at 08:28 , Phil P. [email protected] wrote:

But yes, I understand the issue and do use RCPT-time checks myself,
after normalisation of the sender address, to work around the fact that
I’m doing something a little dodgy that might break legitimate mail.

Although I have added sender address normalization, and new config
allows to check batv mangled addresses against address database,
it actually breaks what is batv used for, because bounce if any will be
sent to normalized address. It looks like catch-22 situation.

10

On 2013-03-05 at 19:33 +0400, Valentin V. Bartenev wrote:

Done.

http://nginx.org/patches/spdy/patch.spdy-66_1.3.14.txt

Thanks; I deployed this a day and a half ago; I could no longer trigger
the conneciton drops from another box on the same network, where I could
do so fairly reliably before. Nobody has replied to my request for
testing from more places, so I’m going to call this fixed.

Much appreciated!

-Phil

11

On Saturday 02 March 2013 02:09:46 Valentin V. Bartenev wrote:

So, nginx+KQUEUE+SPDY breaks clients which shutdown on the write side,
absolutely correct, if SPDY is enabled for that server. In this case,
it’s a cleartext server so SPDY wasn’t enabled at all.

SPDY patch also includes many changes for http core of nginx. The one that
you see, is the unintended result of these changes. I’m going to fix it in
upcoming revision, since it can break some setups as you have mentioned.

Done.

http://nginx.org/patches/spdy/patch.spdy-66_1.3.14.txt

wbr, Valentin V. Bartenev

–
http://nginx.org/en/donation.html