Ok this is weird I noticed tonight high CPU load on two out of three of
our stream servers, which are serving mp4 and flv via the pseudo
streaming modules.
After investigating the differences between the two high CPU boxes and
the one low CPU machine the only notable difference was that on the two
high CPU boxes the config was including our standard gzip options:-
gzip on;
gzip_static on;
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 6;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript
text/xml application/xml application/xml+rss text/javascript;
# make sure gzip does not lose large gzipped js or css files
gzip_buffers 16 8k;
# Disable gzip for certain browsers.
gzip_disable "MSIE [1-6].(?!.*SV1)";
Now as you can see the types is set to text based types so .mp4 and .flv
files really shouldn’t be touched, so I looked else where. After much
looking I really couldn’t find any significant differences so I added
the include of the above to the low CPU machine. Sure enough the CPU
instantly jumped up with each nginx process using 20-40% cpu instead of
0-2% it was before the reload.
Still not convinced and after double checking the gzip config to confirm
it really shouldn’t be going anywhere near the streaming content I added
gzip off and gzip_static off to the *.mp4 and *.flv locations in the
config on the high CPU machine and reloaded. An hour or two later ( time
for the existing users streams to finish ) and the CPU load has already
dropped to the very low values I expect from nginx for the connection
load on the machine e.g. 0 - 2% per nginx process instead of the 40-90%
I was seeing.
Just to confirm I checked the network load and connected users and sure
enough the actual connections and bandwidth is still the same so the
machine is still under the same amount of connection load as before just
with significantly lower CPU.
So what’s going on? Have I missed something totally obvious? Or is there
a nasty bug which means when gzip is enabled in this manner it causes
significant processing overhead for items its not even meant to be
processing?
In case its relavent our main http block includes:
include mime.types;
default_type application/octet-stream;
The current streaming bocks, which include the workaround (turning gzip
off) are:-
# Actual Streaming
location ~ /streaming/(.*\.mp4)$ {
# Fix strange CPU usage caused by gzip
gzip off;
gzip_static off;
limit_rate_after 10m;
limit_rate 400k;
alias /streaming/$1;
internal;
mp4;
}
location ~ /streaming/(.*\.flv)$ {
# Fix strange CPU usage caused by gzip
gzip off;
gzip_static off;
limit_rate_after 10m;
limit_rate 400k;
alias /streaming/$1;
internal;
flv;
}
On Fri, Jan 07, 2011 at 08:39:52PM -0500, steveh wrote:
gzip_http_version 1.1;
[/code]
gzip off and gzip_static off to the *.mp4 and *.flv locations in the
So what’s going on? Have I missed something totally obvious? Or is there
a nasty bug which means when gzip is enabled in this manner it causes
significant processing overhead for items its not even meant to be
processing?
Working with “gzip_static on;” implies extra open() syscall per
request (it work’s based on .gz file presense, not gzip_types) and
isn’t recommended unless actually used.
It shouldn’t make any difference with flv handler activated though
(flv disables general content handlers and gzip_static in
particular). Most likely the same applies to mp4.
Working with “gzip on;” implies extra check for content-type per
request (and some minimal checks during response sending). This
shouldn’t cause much load.
Actual Streaming
}
flv;
}
Could you please check if it’s gzip or gzip_static which causes
high cpu usage?
Also, could you please provide some more details about nginx
you’re using? I.e. nginx -V output, please.
Ok tested without gzip_static off, no change, commenting out the gzip
off however and CPU usage starts to climb within seconds of a reload so
its definitely gzip causing.
Ok found the problem the send of the flv or mp4 is being triggered by an
X-Accel-Redirect header who’s content type is text/html. It seems this
header is maintained by nginx when processing the flv send instead of
setting up one from the configured mime.types.
This is because the flv modules calls ngx_http_set_content_type to set
this up which returns NGX_OK without making any changes if there is
already content_type for the request. An existing content_type already
seems to be being inherited from the original response containing the
X-Accel-Redirect header, hence no change and the problem.
The docs clear state this is the case here:
This doesn’t effect mp4’s which I was testing with originally as the mp4
module blindly sets video/mp4 so works fine in this case.
So sorry for the noise all a case of gotcha I’m afraid.
For reference I’m pretty sure that its not connection setup /
negotiation that is causing the load when enabled as the machines aren’t
dealing with that may new connections per second, so its an overhead for
the entirety of the data send I believe.
Ok so found something interesting, while mp4’s are being sent with the
correct header flv’s aren’t and hence are getting compressed:
GET /streaming/test.flv?..
nginx/0.8.53
Date Sat, 08 Jan 2011 03:01:24 GMT
Content-Type text/html; charset=utf-8
Content-Length 693656659
Last-Modified Fri, 07 Jan 2011 16:19:25 GMT
Connection keep-alive
Cache-Control private, max-age=0, must-revalidate
Accept-Ranges bytes
GET /streaming/test.mp4?..
Server nginx/0.8.49
Date Sat, 08 Jan 2011 03:09:25 GMT
Content-Type video/mp4; charset=utf-8
Content-Length 53385372
Last-Modified Wed, 05 Jan 2011 14:55:46 GMT
Connection keep-alive
Cache-Control private, max-age=0, must-revalidate
X-Mod-H264-Streaming version=2.2.7
Accept-Ranges bytes
So the question is now why are flv’s getting text/html as their content
type?
Posted at Nginx Forum:
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.