Sorry, typo. “takes about 1GB for 10L connections” should have read
“takes about 1GB for 10K connections”, i.e. almost an order of magnitude
more than without SSL.
On Mon, Nov 09, 2009 at 05:34:12PM +0000, Edward Hibbert wrote:
Sorry, typo. “takes about 1GB for 10L connections” should have read “takes about 1GB for 10K connections”, i.e. almost an order of magnitude more than without SSL.
According to ktrace, OpenSSL takes about 80K per connection just on
the handshake phase.
- having the connection open with an outstanding
request takes about 1GB for 10L connections.
Using the OpenSSL 1.0 (thanks for the suggestion, Cliff) there’s a
significant drop in occupancy.
- 16K open connections now takes ~277MB (noise level difference from
previous measurement) - 16K connections with the outstanding GET request now takes ~1GB (down
from estimated 1.6GB previously)
So there’s still a massive occupancy cost of having an outstanding GET
request in the HTTPS case, which is surprising. Looks like I’d better
roll my sleeves up and dig into the code.
Edward.
Posted at Nginx Forum:
Hello!
On Thu, Nov 12, 2009 at 09:15:14AM -0500, edwh2 wrote:
about 16K connections
- having the connection open with an outstanding
request takes about 1GB for 10L connections.Using the OpenSSL 1.0 (thanks for the suggestion, Cliff) there’s a significant drop in occupancy.
- 16K open connections now takes ~277MB (noise level difference from previous measurement)
Are you sure you measure allocated memory? For 16k connections
openssl (at least 0.9.8, but you claim numbers are the same…)
should allocate about 1.2G (~80k per connection) on ssl handshake.
Numbers you see in resident memory (RES in top) may be quite
different, but it just means that relevant memory wasn’t yet
touched. And grow on request processing is expected.
- 16K connections with the outstanding GET request now takes ~1GB (down from estimated 1.6GB previously)
So there’s still a massive occupancy cost of having an outstanding GET request in the HTTPS case, which is surprising. Looks like I’d better roll my sleeves up and dig into the code.
For outstanding GET requests there is 16k ssl buffer allocated by
nginx on first output (see src/event/ngx_event_openssl.c) and
freed when connection goes to keepalive state. This gives 256M
for 16k connections.
Also there is a bunch of memory in nginx (output buffers, large
client header buffers, gzip buffers, various request-related data
and so on) which are freed when connection goes to keepalive
state. Depending on your settings, particular request and already
sent response this may contribute various numbers to keepalive vs.
outstanding request cases. This should be almost the same in
non-https case though, at least with sendfile not used.
Maxim D.
Hello!
On Thu, Nov 12, 2009 at 01:11:08PM -0500, edwh2 wrote:
16k connections
int loop;//close(sock); }The difference in occupancy comes from whether or not I comment out the http_request code. Would you expect the openssl calls above to have incurred the server-side occupancy hit?
See nothing obviously wrong here, but can’t say anything
for sure without full code.
Looking into openssl source, there are at least the following
allocation right after it got ClientHello message (ssl3/tls1
case):
~21k in ssl3_accept() for s->init_buf
~34k in ssl3_setup_buffers() for s->s3->rbuf (as nginx uses
SSL_OP_MICROSOFT_BIG_SSLV3_BUFFER workaround option)
~18k in ssl3_setup_buffers() for s->s3->wbuf
About ~70k in total.
Also there are about 8k for input/ouput bio buffers as well, and
various small allocations for state information (including copy of
server cert and so on). This looks much closer to 270M per 16k
connections you see, so probably your client doesn’t send
ClientHello for some reason.
You may try running nginx with debug logging enabled and take a
look if ssl handshake actually happens. There should be '[debug] …
SSL: …, cipher: …" lines right after ssl handshake completed.
Also you may try tcpdump to see what happens on the wire.
[…]
- free it in between getting a request in and sending the response?
It’s already allocated only when needed (i.e. when nginx starts
sending the response, and freed when connection goes to keepalive
state - i.e. after sending the response). Reducing it will likely
render it useless - it’s here to reduce ssl overhead by feeding
openssl with big chunks of data instead multiple small ones.
Anyway - it’s almost nothing compared to openssl’s own buffers.
Maxim D.
Thanks for your response.
Maxim D. Wrote:
Are you sure you measure allocated memory?
Yes, I’m looking at VIRT in top rather than RES.
For
16k connections
openssl (at least 0.9.8, but you claim numbers are
the same…)
should allocate about 1.2G (~80k per connection)
on ssl handshake.
That’s odd, because I don’t see that. I’m using some slightly hacky
code derived from some openssl examples, and here’s what it looks like:
/* Build our SSL context*/
ctx=initialize_ctx(KEYFILE,PASSWORD);
int loop;
for (loop = 0; loop < 20000; loop++)
{
printf("Loop %d\n", loop);
/* Connect the TCP socket*/
sock=tcp_connect(host,port);
/* Connect the SSL socket */
ssl=SSL_new(ctx);
sbio=BIO_new_socket(sock,BIO_NOCLOSE);
SSL_set_bio(ssl,sbio,sbio);
if(SSL_connect(ssl)<=0)
berr_exit("SSL connect error");
if(require_server_auth)
check_cert(ssl,host);
/* Now make our HTTP request */
http_request(ssl, loop);
/* Shutdown the socket */
//destroy_ctx(ctx);
//close(sock);
}
The difference in occupancy comes from whether or not I comment out the
http_request code. Would you expect the openssl calls above to have
incurred the server-side occupancy hit?
Numbers you see in resident memory (RES in top)
may be quite
different, but it just means that relevant memory
wasn’t yet
touched. And grow on request processing is
expected.
Yes, these are easily confused but I’m familiar with the problem.
buffer allocated by
nginx on first output (see
src/event/ngx_event_openssl.c) and
freed when connection goes to keepalive state.
This gives 256M
for 16k connections.
Ok, that’s useful. I’ll look at the code, but do you think it’s
feasible either to
- shrink this (how small could it be? what would happen if it was too
small?), or - free it in between getting a request in and sending the response?
That wouldn’t account for all the difference I see, but 256M would go
some way to reducing the occupancy hit.
to keepalive vs.
outstanding request cases. This should be almost
the same in
non-https case though, at least with sendfile not
used.
Yes, that’s what I would expect, and the occupancy in the non-https case
is pretty good. So I’m hoping not to have to touch that, and that this
is something https-specific.
Edward.
Posted at Nginx Forum:
Igor S. Wrote:
According to ktrace, OpenSSL takes about 80K per
connection just on
the handshake phase.
My point is that while there is increased occupancy just from having the
SSL connection, as you’d expect, there is massively increased occupancy
while a GET request is outstanding on the SSL connection. That’s what I
find surprising, because I’d expect that on an individual request you’d
need a bit of extra memory during the decryption of the request and the
encryption of the response, but not in between the two. I’m wondering
if anyone knows why that is and whether it’s possible to reduce it.
Edward
(Replying from a different address as my work spam filter seems to
dislike this list)
Posted at Nginx Forum: