Continuing issues with limit_conn

Stuart_McKim · April 29, 2008, 3:18am

I’m still unable to get limit_conn to work as expected, despite reading
as much documentation as I can find, of which there seems to be very
little.

In the http section of /etc/nginx/nginx.conf I have the two lines
limit_zone test-limit $binary_remote_addr 10m;
limit_conn test-limit 15;

The debug output from the error log, with line numbers added, is:
01 2008/04/28 16:06:51 [debug] 19050#0: *15 write new buf t:1 f:0
080F3778, pos 080F3778, size: 240 file: 0, size: 0
02 2008/04/28 16:06:51 [debug] 19050#0: *15 http write filter: l:0 f:0
s:240
03 2008/04/28 16:06:51 [debug] 19050#0: *15 http output filter
“/webtest.html?”
04 2008/04/28 16:06:51 [debug] 19050#0: *15 copy filter:
“/webtest.html?”
05 2008/04/28 16:06:51 [debug] 19050#0: *15 http postpone filter
“/webtest.html?” BFEA3E58
06 2008/04/28 16:06:51 [debug] 19050#0: *15 http postpone filter out
“/webtest.html?”
07 2008/04/28 16:06:51 [debug] 19050#0: *15 write old buf t:1 f:0
080F3778, pos 080F3778, size: 240 file: 0, size: 0
08 2008/04/28 16:06:51 [debug] 19050#0: *15 write new buf t:0 f:1
00000000, pos 00000000, size: 0 file: 0, size: 212
09 2008/04/28 16:06:51 [debug] 19050#0: *15 http write filter: l:1 f:0
s:452
10 2008/04/28 16:06:51 [debug] 19050#0: *15 http write filter limit 0
11 2008/04/28 16:06:51 [debug] 19050#0: *15 tcp_nopush
12 2008/04/28 16:06:51 [debug] 19050#0: *15 writev: 240
13 2008/04/28 16:06:51 [debug] 19050#0: *15 sendfile: @0 212
14 2008/04/28 16:06:51 [debug] 19050#0: *15 sendfile: 212, @0 212:212
15 2008/04/28 16:06:51 [debug] 19050#0: *15 http write filter 00000000
16 2008/04/28 16:06:51 [debug] 19050#0: *15 copy filter: 0
“/webtest.html?”
17 2008/04/28 16:06:51 [debug] 19050#0: *15 http finalize request: 0,
“/webtest.html?”
18 2008/04/28 16:06:51 [debug] 19050#0: *15 set http keepalive handler
19 2008/04/28 16:06:51 [debug] 19050#0: *15 http close request
20 2008/04/28 16:06:51 [debug] 19050#0: *15 http log handler
21 2008/04/28 16:06:51 [debug] 19050#0: *15 run cleanup: 080F3674
22 2008/04/28 16:06:51 [debug] 19050#0: *15 file cleanup: fd:58
23 2008/04/28 16:06:51 [debug] 19050#0: *15 run cleanup: 080F363C
24 2008/04/28 16:06:51 [debug] 19050#0: *15 free: 080F3178, unused:
2091
25 2008/04/28 16:06:51 [debug] 19050#0: *15 event timer add: 23:
75000:2538275930
26 2008/04/28 16:06:51 [debug] 19050#0: *15 free: 080B7770
27 2008/04/28 16:06:51 [debug] 19050#0: *15 free: 080F2D70
28 2008/04/28 16:06:51 [debug] 19050#0: *15 hc free: 00000000 0
29 2008/04/28 16:06:51 [debug] 19050#0: *15 hc busy: 00000000 0
30 2008/04/28 16:06:51 [debug] 19050#0: *15 http keepalive handler
31 2008/04/28 16:06:51 [debug] 19050#0: *15 malloc: 080F2D70:1024
32 2008/04/28 16:06:51 [debug] 19050#0: *15 recv: fd:23 -1 of 1024
33 2008/04/28 16:06:51 [debug] 19050#0: *15 recv() not ready (11:
Resource temporarily unavailable)
34 2008/04/28 16:06:51 [debug] 19050#0: *16 malloc: 080B7770:632
35 2008/04/28 16:06:51 [debug] 19050#0: *16 malloc: 080F3178:1024
36 2008/04/28 16:06:51 [debug] 19050#0: *16 malloc: 080F3580:4096
37 2008/04/28 16:06:51 [debug] 19050#0: *16 http process request line
38 2008/04/28 16:06:51 [debug] 19050#0: *16 recv: fd:24 122 of 1024
39 2008/04/28 16:06:51 [debug] 19050#0: *16 http request line: “GET
/webtest.html HTTP/1.0”
40 2008/04/28 16:06:51 [debug] 19050#0: *16 http uri: “/webtest.html”
41 2008/04/28 16:06:51 [debug] 19050#0: *16 http args: “”
42 2008/04/28 16:06:51 [debug] 19050#0: *16 http exten: “html”
43 2008/04/28 16:06:51 [debug] 19050#0: *16 http process request header
line
44 2008/04/28 16:06:51 [debug] 19050#0: *16 http header: “User-Agent:
ApacheBench/1.3d”
45 2008/04/28 16:06:51 [debug] 19050#0: *16 http header: “Connection:
Keep-Alive”
46 2008/04/28 16:06:51 [debug] 19050#0: *16 http header: “Host:
mckim1.osuosl.org”
47 2008/04/28 16:06:51 [debug] 19050#0: *16 http header: “Accept: /”
48 2008/04/28 16:06:51 [debug] 19050#0: *16 http header done
49 2008/04/28 16:06:51 [debug] 19050#0: *16 event timer del: 24:
2538800922
50 2008/04/28 16:06:51 [debug] 19050#0: *16 find location for
“/webtest.html”
51 2008/04/28 16:06:51 [debug] 19050#0: *16 using configuration “”
52 2008/04/28 16:06:51 [debug] 19050#0: *16 http cl:-1 max:1048576
53 2008/04/28 16:06:51 [debug] 19050#0: *16 generic phase: 1
54 2008/04/28 16:06:51 [debug] 19050#0: *16 add cleanup: 080F3A44
55 2008/04/28 16:06:51 [debug] 19050#0: *16 limit zone: E1F43BF3 1
56 2008/04/28 16:06:51 [debug] 19050#0: *16 access phase: 2
57 2008/04/28 16:06:51 [debug] 19050#0: *16 access phase: 3
58 2008/04/28 16:06:51 [debug] 19050#0: *16 post access phase: 4
59 2008/04/28 16:06:51 [debug] 19050#0: *16 content phase: 5
60 2008/04/28 16:06:51 [debug] 19050#0: *16 content phase: 6
61 2008/04/28 16:06:51 [debug] 19050#0: *16 content phase: 7
62 2008/04/28 16:06:51 [debug] 19050#0: *16 http set discard body
63 2008/04/28 16:06:51 [debug] 19050#0: *16 http filename:
“/var/www/nginx/htdocs1/webtest.html”
64 2008/04/28 16:06:51 [debug] 19050#0: *16 add cleanup: 080F3A7C
65 2008/04/28 16:06:51 [debug] 19050#0: *16 http static fd: 58
66 2008/04/28 16:06:51 [debug] 19050#0: *16 HTTP/1.1 200 OK
67 Server: nginx/0.6.29
68 Date: Mon, 28 Apr 2008 23:06:51 GMT
69 Content-Type: text/html
70 Content-Length: 212
71 Last-Modified: Fri, 11 Apr 2008 21:43:45 GMT
72 Connection: keep-alive
73 Keep-Alive: timeout=20
74 Accept-Ranges: bytes

This is repeated many times, with the *XX number being the only
significant change. The “limit zone: E1F43BF3 1” line is always
identical.

If I do not use the limit_zone and limit_conn lines in my config, or
exclude just the limit_conn line, the debug output appears to be the
same except I don’t see the lines
21 2008/04/28 16:06:51 [debug] 19050#0: *15 run cleanup: 080F3674
54 2008/04/28 16:06:51 [debug] 19050#0: *16 add cleanup: 080F3A44
55 2008/04/28 16:06:51 [debug] 19050#0: *16 limit zone: E1F43BF3 1

I’m not really sure what I should expect to see in the debug output, but
it would appear that the module is trying to at least do something.

The catch is that with 1000 connections, in sessions with 50 concurrent
connections, I’m expecting to see some 503 returns from the server, but
the server responds with 200 every time. This is not proper behaviour,
correct? As far as I can tell, this is a bug.

I am using version 0.6.29.

Has anybody had any success with using this module? If so, how did you
test it?

If more information would be useful, I can provide it.

Thanks,
Stuart

Stuart_McKim · April 29, 2008, 5:38am

Stuart,

Building Nginx 0.6.29, we are also unable to get limit_zone/limit_conn
to
work as expected.

As a test we setup the relative lines in the http and server sections.
Nginx
should only accept ONE concurrent connection for any single ip address.

http{
limit_zone gulag $binary_remote_addr 1m;
server{
limit_conn gulag 1;
}
}

When I run “ab -c 50 -n 10000 http://testbox/” the server answers all
requests with response code 200. As you mentioned, this is not the
expected behavior.

Perhaps we are missing something? The code could be at fault or perhaps
something has been omitted from the Wiki and the documentation. If there
is
a proper solution I will make sure to document it.

My working test configuration and full explanations can be found here:

Nginx “how to” - Fast, Secure and Efficient Web Server
Nginx Secure SSL Web Server @ Calomel.org

–
Calomel @ http://calomel.org
Open Source Research and Reference

Stuart_McKim · April 29, 2008, 5:51pm

Maxim,

Thank you for the response. I just want to make sure I understand this
directive correctly as the Wiki seems a bit unclear.

The ngx_http_limit_zone_module only limits the amount of connections
from a
single ip address which are currently being processed by the Nginx
daemon.

For example, if we declared “limit_conn gulag 1” and one
worker_processes
is enabled then Nginx should only process one connection at a time
(serial). If requests are coming in faster than Nginx can process them
with
the single worker, then the client will get a error 503. But, with more
workers enabled a remote ip would need to bog down all of them in order
to
get a error 503.

The ngx_http_limit_zone_module does NOT limit the total amount of
connections any single remote ip can open to the server.

Does this sound correct?

Nginx “how to”
Nginx Secure SSL Web Server @ Calomel.org

–
Calomel @ http://calomel.org
Open Source Research and Reference

Stuart_McKim · April 30, 2008, 1:25pm

Hello!

On Tue, Apr 29, 2008 at 11:40:40AM -0400, Calomel wrote:

Maxim,

Thank you for the response. I just want to make sure I understand this
directive correctly as the Wiki seems a bit unclear.

The ngx_http_limit_zone_module only limits the amount of connections from a
single ip address which are currently being processed by the Nginx daemon.

If you use limit_zone on $binary_remote_addr - yes.

For example, if we declared “limit_conn gulag 1” and one worker_processes
is enabled then Nginx should only process one connection at a time
(serial). If requests are coming in faster than Nginx can process them with
the single worker, then the client will get a error 503.

Not really. Error 503 will be returned if request processing
blocks at socket level and new request from same ip starts.
This won’t help if you workload is CPU or disk bound.

But, with more workers enabled a remote ip would need to bog down
all of them in order to get a error 503.

Not. With several workers 503 will be returned if two workers process
requests from the same ip at the same time. But this is unlikely
to happen with small requests.

The ngx_http_limit_zone_module does NOT limit the total amount of
connections any single remote ip can open to the server.

Yes.

Does this sound correct?

Not really.

Maxim D.

Stuart_McKim · April 29, 2008, 10:05am

Hello!

On Mon, Apr 28, 2008 at 11:29:08PM -0400, Calomel wrote:

}
}

When I run “ab -c 50 -n 10000 http://testbox/” the server answers all
requests with response code 200. As you mentioned, this is not the
expected behavior.

Perhaps we are missing something? The code could be at fault or perhaps
something has been omitted from the Wiki and the documentation. If there is
a proper solution I will make sure to document it.

It looks like there is some misunderstanding regarding to what
limit_conn actually limits. It limits concurrent connections
processed by nginx (not keep-alive ones), and only after header
has been received (and thus configuration for request has been
determined).

Since nginx is event-based, with one worker process you shouldn’t
expect requests to hit limit_conn unless they block at some stage
(i.e. responses bigger than socket buffers if sendfile off,
replies bigger than sendfile_max_chunk if sendfile on, proxy_pass
…). With many workers limit_conn may be hit without blocking,
but this generally requires very high concurrency for small
requests.

Maxim D.

Stuart_McKim · April 29, 2008, 9:07pm

On Tue, Apr 29, 2008 at 11:57:44AM +0400, Maxim D. wrote:

limit_conn may be hit without blocking, but this generally requires very
high concurrency for small requests.

Maxim D.

Maxim,

Thanks for the clarification. I ran another test, using a large test
file and multiple workers and saw that nearly all the requests received
a 503 response.

This makes sense; I wasn’t taking into account the way I was running the
test.

Thanks,
Stuart