Nginx limit_rate if in location - strange behaviour - possible bug?

Hi,

nginx -V
nginx version: nginx/1.4.2
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt=‘-I
/usr/local/include’ --with-ld-opt=‘-L /usr/local/lib’
–conf-path=/usr/local/etc/nginx/nginx.conf
–sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid
–error-log-path=/var/log/nginx-error.log --user=www --group=www
–http-client-body-temp-path=/var/tmp/nginx/client_body_temp
–http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp
–http-proxy-temp-path=/var/tmp/nginx/proxy_temp
–http-scgi-temp-path=/var/tmp/nginx/scgi_temp
–http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp
–http-log-path=/var/log/nginx-access.log
–with-http_image_filter_module --with-http_stub_status_module
–add-module=/usr/ports/www/nginx/work/naxsi-core-0.50/naxsi_src
–with-pcre

FreeBSD 9.1-STABLE #0: Sat May 18 00:32:18 EEST 2013 amd64

I’m using limit_rate case if in location. Regarding documentation “if in
location” context is avaiable.

My configuration is as follows:

location some_extension {

case 1

if match_something {
root …
break;
}

case 2

if match_another {
root …
break;
}

else (case3)

root …
something other …
here it is placed also limit_rate / limit_after directives
}

There is a root inside location with a (strong) reason :slight_smile: (nginx
pitfails case “root inside location block - BAD”).

When I open in my browser http://my.website.com/myfile.ext it matches
case 3 from the cofiguration. Limit_rate/limit_after works correct as
expected.
I want case1 not to have limit_rate / limit_after.

Test one:
In case1 I place limit_rate 0, case3 is the same limit_rate_after XXm;
limit_rate some_rate. When I open in my browser URL matching case1 -
limit_rate 0 is ignored. After hitting XXm from the file I get
limit_rate from case 3.

Test 2:
In case 1 I place limit_rate_after 0; limit_rate 0, case3 is the same
limit_rate_after XXm; limit_rate some rate. When I open in my browser
URL matching case 1 - limit_rate_after 0 and limit_rate 0 are ignored.
Worst is that when I try to download the file, I even didn’t match case3

  • my download starts from the first MB with limit_rate bandwidth from
    case3.

Both tests are made in interval from 20 minutes, 1 connection from my
IP, etc.

I don’t post my whole configuration, because may be it is unnessesary.
If cases are with http_referer.

Case 1 - if I see referer some_referer, I do something. (here I don’t
want to place any limits)
Case 2 - If I see another referer , I do something else.
Case 3 - Else … something other (here I have some limits)

I’m sure I match case1 when I test (nginx-error.log with debug say it),
I mean my configuration with cases is working as expected, the new thing
is limit_rate and limit_rate_after which is not working as expected.

Any thoughts ? Meanwhile I will test on another version.

Thanks

I’m sorry, misread documentation.

Placed set $limit_rate 0 in my case1 instead limit_rate 0 and now works
as expected.

Hello!

On Wed, Oct 09, 2013 at 05:21:51PM +0300, Bozhidara Marinchovska wrote:

[…]

break;
something other …
here it is placed also limit_rate / limit_after directives
}

There is a root inside location with a (strong) reason :slight_smile: (nginx
pitfails case “root inside location block - BAD”).

When I open in my browser http://my.website.com/myfile.ext it
matches case 3 from the cofiguration. Limit_rate/limit_after works
correct as expected.
I want case1 not to have limit_rate / limit_after.

The limit_rate directive is somewhat special (== wierd), and a
limit once applied on a location matched is preserved for future
use. Further matches with different limit_rate set doesn’t
influence the limit actually used.

If you want to override limit_rate on a specific condition, use

set $limit_rate <rate>;

instead, see Module ngx_http_core_module.


Maxim D.
http://nginx.org/en/donation.html

Hi, again

FreeBSD 9.1-STABLE #0: Sat May 18 00:32:18 EEST 2013 amd64

nginx version: nginx/1.4.4
TLS SNI support enabled
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt=’-I
/usr/local/include’ --with-ld-opt=’-L /usr/local/lib’
–conf-path=/usr/local/etc/nginx/nginx.conf
–sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid
–error-log-path=/var/log/nginx-error.log --user=www --group=www
–http-client-body-temp-path=/var/tmp/nginx/client_body_temp
–http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp
–http-proxy-temp-path=/var/tmp/nginx/proxy_temp
–http-scgi-temp-path=/var/tmp/nginx/scgi_temp
–http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp
–http-log-path=/var/log/nginx-access.log
–with-http_image_filter_module --with-http_stub_status_module
–add-module=/usr/ports/www/nginx/work/nginx_upload_module-2.2.0
–add-module=/usr/ports/www/nginx/work/masterzen-nginx-upload-progress-module-a788dea
–add-module=/usr/ports/www/nginx/work/naxsi-core-0.50/naxsi_src
–with-pcre --with-http_ssl_module

I’ve meet strange behaviour when connection is established via software
download manager.

My configuration is as follows:

location some_extension {

limit_rate_after 15m;
limit_rate 250k;

}

When I open or save as URL with matched extension via browser (desktop,
mobile, etc), limit_rate works perfectly, after I hit 15MB of the file,
my speed goes down up to 1Mb/s regarding my configuration.

When I place the same URL with matched extension in some download
manager like IDM (internet download manager), FDM (free download
manager) or others, limit_rate is not matched, download speed proceeded
at max possible. [the actual problem]

As differences from browser and software download manager I met only:

  • when download is processed via browser - if download is requested on
    chunks I get 206 partial content (multiple http sessions sent, multiple
    entries for this connection in my access log), download if processed as
    expected, after hitting 15MB of the file, download speed goes down to 1
    Mb/s

  • when download is processed via browser - if download is requested on
    one piece I get 200 ok (1 http session, 1 entry in my access log
    regarding this connection), download if processed as expected, after
    hitting 15MB of the file, download speed goes down to 1 Mb/s

  • when download is processed via software download manager FDM (free
    download manager) I receive 1 http connection with status 200 OK (1
    entry in my access log, multiple established connections from remote
    addr X.X.X.X and remote ports A,B,C,D,…) , but after inspecting
    headers, I see multiple requests regarding this single http session, in
    multiple requests is changed only start byte in Range header without
    sending end byte in range header; netstat confirms multiple connections
    via this single session and limit_rate is not matched in that case.

tcpdump - download performed via browser Firefox - 200 OK single
connection (1 entry in my access_log), no Range header:
GET [requested_file] HTTP/1.1
Host: [host]
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:27.0) Gecko/20100101
Firefox/27.0 AlexaToolbar/alxf-2.19
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: [referer]
Cookie: [cookie]
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

tcpdump - download performed via browser Firefox 206 partial (multiple
connections, multiple entries in my access log) - range header sent with
end byte specified:
GET [requested_file] HTTP/1.1
Host: [host]
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:27.0) Gecko/20100101
Firefox/27.0 AlexaToolbar/alxf-2.19
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: [referer]
Cookie: [cookie]
Connection: keep-alive
Range: bytes=2621440-2686975

tcpdump - download performed via FDM (single session, 1 entry in my
access log, multiple established connections) without end byte specified
in Range header:
GET [requested_file] HTTP/1.1
Referer: [referer]
Accept: /
Range: bytes=2541702-
User-Agent: FDM 3.x
Host: [host]
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: [cookie]

My question is what may be the reason when downloading the example file
with download manager not to match limit_rate directive and is there any
way to debug somehow limit_rate ?

I’ve used to inspect in the past the behaviour when end byte in Range
header is not specified (unfortunately I don’t remember if is was with
some browser, but I can reproduce it if needed), then Nginx returns 416
(and when I see 416 in my access log, I block certain client for some
time, let say 1h hour ).

Actually is not a problem for me, to match these download managers and
block them, or to set some inspection regarding Range header, or to
limit the connection with limit_conn instead limit_rate, but I don’t
want to limit speed by IP address or to block download managers, but I
would like to use limit_rate regarding file type extension.

Thank you

On 17 February 2014 10:02, Bozhidara Marinchovska
[email protected] wrote:

My question is what may be the reason when downloading the example file with
download manager not to match limit_rate directive

“Download managers” open multiple connections and grab different byte
ranges of the same file across those connections. Nginx’s limit_rate
function limits the data transfer rate of a single connection.