Issue in ssl negociation

Server NGINX
1

Hello,

This is my first POST, so please forgive me if I used the wrong
mailing-list…

I am using nginx v0.7.65 on Debian squeeze. I have a sporadic issue
occuring on requests coming from Android mobile phones, concerning ssl
negociation.

After receiving “Client Hello” from the phone, nginx sends 2 packets:
“Server Hello” and a 1st fragment on the “Certificate, Server Hello
Done” (line 7 in the trace below). Then, it waits for the Ack of the
“Server hello” and never sends the 2nd fragment of the “Certificate,
Server Hello Done”. It retransmitts the “Server Hello”, but the client
is stuck waiting for the second fragment of the certificate.

And the Android device, I have a 1minute timeout which expires and
throws the error “no peer certificate” (see the FIN,Ack, line 12).

Immediately after, nginx sends the second fragment of the “Certificate,
Server Hello Done” message (line 13).

My questions are: is nginx fragmenting packets on purpose ? Is there a
way to disable this and let IP layer do the fragmentation ?

Could upgrading to 0.7.69 help ?

Here is a summary of the network trace. I can provide the full trace if
required.
No. Time Source Destination
Protocol Length Info
1 2011-12-05 23:06:51 92.90.19.29 94.143.115.73 TCP
76 37352 > https [SYN] Seq=0 Win=64240 Len=0
2 2011-12-05 23:06:51 94.143.115.73 92.90.19.29 TCP
76 https > 37352 [SYN, ACK] Seq=0 Ack=1 Win=5792
3 2011-12-05 23:06:52 92.90.19.29 94.143.115.73 TCP
68 37352 > https [ACK] Seq=1 Ack=1 Win=64240
4 2011-12-05 23:06:52 92.90.19.29 94.143.115.73
TLSv1 148 Client Hello
5 2011-12-05 23:06:52 94.143.115.73 92.90.19.29 TCP
68 https > 37352 [ACK] Seq=1 Ack=81 Win=6144
6 2011-12-05 23:06:52 94.143.115.73 92.90.19.29
TLSv1 1516 Server Hello
7 2011-12-05 23:06:52 94.143.115.73 92.90.19.29 TCP
1516 [TCP segment of a reassembled PDU]
8 2011-12-05 23:06:56 94.143.115.73 92.90.19.29
TLSv1 1516 [TCP Retransmission] Server Hello
9 2011-12-05 23:07:02 94.143.115.73 92.90.19.29
TLSv1 1516 [TCP Retransmission] Server Hello
10 2011-12-05 23:07:14 94.143.115.73 92.90.19.29
TLSv1 1516 [TCP Retransmission] Server Hello
11 2011-12-05 23:07:39 94.143.115.73 92.90.19.29
TLSv1 1516 [TCP Retransmission] Server Hello
12 2011-12-05 23:07:52 92.90.19.29 94.143.115.73 TCP
68 37352 > https [FIN, ACK] Seq=81 Ack=1 =64240
13 2011-12-05 23:07:52 94.143.115.73 92.90.19.29
TLSv1 1007 Ignored Unknown Record
14 2011-12-05 23:07:52 92.90.19.29 94.143.115.73 TCP
56 37352 > https [RST] Seq=82 Win=0 Len=0

The result in ssl-access.log is a line like this:
92.90.19.29 - - [05/Dec/2011:23:07:51 +0100] “-” 400 0 “-” “-”

My nginx config:
server {

listen       94.143.115.73:443;

server_name www.myinwebo.com myinwebo.com;
ssl on;
ssl_certificate <path to my cert>;
ssl_certificate_key <path to my key>;

ssl_ciphers ALL:!ADH:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
}

Posted at Nginx Forum:

2

Hello!

On Wed, Dec 07, 2011 at 05:33:14AM -0500, brunoa wrote:

“Server Hello” and a 1st fragment on the "Certificate, Server Hello

My questions are: is nginx fragmenting packets on purpose ? Is there a
way to disable this and let IP layer do the fragmentation ?

nginx not fragmenting anything, it’s tcp/ip which do this.

3 2011-12-05 23:06:52 92.90.19.29 94.143.115.73 TCP
68 37352 > https [ACK] Seq=1 Ack=1 Win=64240
4 2011-12-05 23:06:52 92.90.19.29 94.143.115.73
TLSv1 148 Client Hello
5 2011-12-05 23:06:52 94.143.115.73 92.90.19.29 TCP
68 https > 37352 [ACK] Seq=1 Ack=81 Win=6144
6 2011-12-05 23:06:52 94.143.115.73 92.90.19.29
TLSv1 1516 Server Hello
7 2011-12-05 23:06:52 94.143.115.73 92.90.19.29 TCP
1516 [TCP segment of a reassembled PDU]

Two full-sized packets has been sent. None of them has been
ack’ed. TCP stack will wait for an ack here, this is expected
behaviour.

8 2011-12-05 23:06:56 94.143.115.73 92.90.19.29
TLSv1 1516 [TCP Retransmission] Server Hello
9 2011-12-05 23:07:02 94.143.115.73 92.90.19.29
TLSv1 1516 [TCP Retransmission] Server Hello
10 2011-12-05 23:07:14 94.143.115.73 92.90.19.29
TLSv1 1516 [TCP Retransmission] Server Hello
11 2011-12-05 23:07:39 94.143.115.73 92.90.19.29
TLSv1 1516 [TCP Retransmission] Server Hello

Four retransmission. Still no ack from the client.

12 2011-12-05 23:07:52 92.90.19.29 94.143.115.73 TCP
68 37352 > https [FIN, ACK] Seq=81 Ack=1 =64240

FIN from the client. Still no ack of the data sent.

This looks like client’s problem, most likely it just can’t
receive full-sized packets due to network misconfiguration (likely
low MTU on the network path and ICMP filtered).

Maxim D.

3

Hello!

On Wed, Dec 07, 2011 at 09:07:59AM -0500, brunoa wrote:

Hi Maxim,

Thanks for your help. I will investigate this and let you know. If the
issue is on the mobile operator network, I will be pretty much stuck
:frowning:

Most likely it is (make sure to check your network though).

If for some reason you need to be as compatible as possible with
such broken networks, you may want to disable path mtu discovery
in your OS.

Or maybe I will have to decrease the MTU on my server.

I just checked: all the packets from the trace are sent with the “Don’t
fragment” bit set, and an IP size of 1500 (1516 ethernet).

What surprises me though, is that the issue arises sporadically. A
mobile phone will have 1 request out of 4 failing (without changing of
IP address)…

This may indicate different network paths, with some of them
filtering ICMP frag-needed packets, and others don’t (or at least
doing MSS clamp). Alternatively, blackhole detection might come
and magically fix things.

Maxim D.

4

Hi Maxim,

Thanks for your help. I will investigate this and let you know. If the
issue is on the mobile operator network, I will be pretty much stuck
:frowning:

Or maybe I will have to decrease the MTU on my server.

I just checked: all the packets from the trace are sent with the “Don’t
fragment” bit set, and an IP size of 1500 (1516 ethernet).

What surprises me though, is that the issue arises sporadically. A
mobile phone will have 1 request out of 4 failing (without changing of
IP address)…

Posted at Nginx Forum:

5

Hi Maxim,

This may indicate different network paths, with
some of them
filtering ICMP frag-needed packets, and others
don’t (or at least
doing MSS clamp). Alternatively, blackhole
detection might come
and magically fix things.

Indeed, blackhole detection solved my issue. Thanks for your expertise
!

echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing

and also in sysctl.conf.

Now, TCP packets have a size of 564 octets (probably due to tcp_base_mss
default value), which may not be optimal but is ok for now.

Bruno

Posted at Nginx Forum: