Forum: NGINX Nginx Windows High Traffic issues

2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-26 15:41
(Received via mailing list)
So i spent a while on this one and turns out the problem is a little
function in nginx's core called "worker_rlimit_nofile".


But for me on windows (i don't know if it does it for linux users too.)
grinds my site down to a halt unless you increase its value.

Why does it do this ?
http://nginx.org/en/docs/ngx_core_module.html#work...

Even with the following nginx builds for high traffic production
enviorments
http://nginx-win.ecsds.eu/ It still seems like the worker_rlimit_nofile
is
to small by default on nginx start up so unless you know to add the
command
into your nginx config with some insanely high value to it your site
will
seem slow as hell.

I read on stackoverflow the following.
events {
    worker_connections  19000;  # It's the key to high performance -
have a
lot of connections available
}

worker_rlimit_nofile    20000;  # Each connection needs a filehandle (or
2
if you are proxying)

But in my config i set the following just to test
events {
    worker_connections  16384;
  multi_accept on;
}
worker_rlimit_nofile  9990000000;

And everything still works fine can anyone explain so i can understand
why
this value is so small in the first place ?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251186#msg-251186
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2014-06-26 16:03
(Received via mailing list)
The way things have been redesigned, worker_rlimit_nofile has no purpose
anymore, it's best not to set any value.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251187#msg-251187
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-26 16:18
(Received via mailing list)
Well without a value everything is very very slow. With a value its nice
and
fast.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251188#msg-251188
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-26 17:21
(Received via mailing list)
I recon its because i have media sites with lots of files and pictures
videos content etc so i need it to be a large limit.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251192#msg-251192
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2014-06-26 19:27
(Received via mailing list)
c0nw0nk Wrote:
-------------------------------------------------------
> Well without a value everything is very very slow. With a value its
> nice and fast.

Interesting to know, the Windows design and other portions scale
automatically between 4 API's to deal with high performance while
offloading
that to multiple workers at the same time. This design is limitless but
some
baseline values have been set fixed because you need to start somewhere
before tuning runs and after all workers have settled down.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251194#msg-251194
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-26 20:03
(Received via mailing list)
I don't know how you would try to replicate this issue because i have
thousands upon thousands of files being accessed simultaneously without
me
setting that value insanely high pages and access to thing take 10
seconds
and more even timeouts was occurring but as soon as i set that value it
all
stops and everything seems to be running fast as its suppose to.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251196#msg-251196
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2014-06-26 20:44
(Received via mailing list)
Hello!

On Thu, Jun 26, 2014 at 09:41:15AM -0400, c0nw0nk wrote:

> So i spent a while on this one and turns out the problem is a little
> function in nginx's core called "worker_rlimit_nofile".
>
>
> But for me on windows (i don't know if it does it for linux users too.)
> grinds my site down to a halt unless you increase its value.
>
> Why does it do this ?
> http://nginx.org/en/docs/ngx_core_module.html#work...

On unix, worker_rlimit_nofile does exactly what's documented: it
calls setrlimit(RLIMIT_NOFILE) within worker process.  This allows
to change OS-imposed limit without restarting nginx.  And there is
no default in nginx itself - the default is set by OS and it's
configuration.

Note well that the words "without restarting" is actually the
reason why this directive exists at all.  If a restart isn't a big
deal, then OS limit can be changed by native means ("ulimit -n"
and friends).

In official nginx on Windows, worker_rlimit_nofile does nothing.
Not sure if there is an equivalent limit on Windows at all.

--
Maxim Dounin
http://nginx.org/
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-26 21:27
(Received via mailing list)
Maxim Dounin Wrote:
-------------------------------------------------------
> > grinds my site down to a halt unless you increase its value.
> Note well that the words "without restarting" is actually the
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


I find it very hard to believe it does nothing without me setting it
everything is very slow ?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251200#msg-251200
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-26 21:34
(Received via mailing list)
Could it be possible my server slows down because all connections are in
use
?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251201#msg-251201
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2014-06-26 23:21
(Received via mailing list)
c0nw0nk Wrote:
-------------------------------------------------------
> Could it be possible my server slows down because all connections are
> in use ?

No, it's a recycling and auto-tuning issue as far as I can see, have you
determined at which value you noticed the difference or is this value
simply
a big number ?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251204#msg-251204
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-26 23:54
(Received via mailing list)
Now i am clueless because i dropped keepalive requests i also dropped
any
send_timeout values.

And this is what my bandwidth output looks like its very jumpy when it
should not be and my page loads are very slow even on static files like
html, mp4, flv etc and considering its nginx that delievers them i am
very
sure nginx is the problem.
http://i633.photobucket.com/albums/uu52/C0nw0nk/Un...

Something is very wrong with nginx i recon i am completely out of
connections avaliable and it is waiting for a connection to open to use.

Unless anyone knows what i could be needing to add to my config so that
while people are downloading/streaming videos that are upto 500mb in
size.

I already use limit_rate but with that my bandwidth output should not be
jumpy like in the picture it would be a straight smooth line like it
used to
be but with more traffic i recon my connection limit is reached ?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251206#msg-251206
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-27 00:29
(Received via mailing list)
When i said "my bandwidth output looks like its very jumpy". on a 1gig
per
second connection my output jumps up and down 10% (100mb) used then it
will
jump to like 40% (400mb) and it changes so much before when i had less
traffic it used to be a very stead and stable 400-500mb output and
hardly
ever changed so dramaticly.

In the following screenshot you will see me I/O usage from Nginx is
extremely high.
http://s633.photobucket.com/user/C0nw0nk/media/Unt...

And i would like to add the only reason in that picture the nginx
processes
are using so much memory is because i set the "worker_connections
1900000;"
to a high value. I don't know if it should use so much memory or if it
is
just wasting my system resources.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251207#msg-251207
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2014-06-27 07:57
(Received via mailing list)
This is a disk IO issue, not running out of connections, setting 1900000
is
pointless, 16k is more then enough, no more then 2 workers per cpu, I
see 12
workers so do you have enough cpu's to cover that?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251210#msg-251210
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-27 16:15
(Received via mailing list)
I also just to try and check if it was my connection limit enabled
nginx_status and this was my output.

Active connections: 1032
server accepts handled requests
 8335 8335 12564
Reading: 0 Writing: 197 Waiting: 835


How can i fix the I/O issue why is nginx consuming so much in the first
place if i close the nginx process nothing else is even using it.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251232#msg-251232
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2014-06-27 18:25
(Received via mailing list)
Looking at the disk activity access to disk is using all your resources
not
nginx.
Here http://s633.photobucket.com/user/C0nw0nk/media/Unt...
you
see nginx itself is waiting for disk IO to complete, all processes are
doing
just about nothing other then waiting for the harddisk, the main waiting
issue looks like it is writing to disk which isn't going fast enough.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251237#msg-251237
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-27 18:28
(Received via mailing list)
So the soloution could be a different hard drive possibly a solid state
drive ? This is my current hard drive
http://www.hgst.com/hard-drives/enterprise-hard-dr...

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251238#msg-251238
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2014-06-27 19:16
(Received via mailing list)
It all depends what you are writing, too small blocksize, many seeks,
onboard diskcache not working (writeback). Run some disk benchmarks to
see
what your storage is capable of and compare that to how much data your
attempting to write. At the moment your disks are not keeping up with
the
amount of write requests.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251246#msg-251246
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-27 19:23
(Received via mailing list)
Since i have never had to benchmark a hard drive before this will be a
new
experience for me any tools you recommend to use specifically.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251247#msg-251247
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2014-06-27 19:56
(Received via mailing list)
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-27 20:34
(Received via mailing list)
The results got even more fascinating as i increased the buffer size's
to
the following.

client_max_body_size 0;
client_body_buffer_size 1000m;
mp4_buffer_size 700m;
mp4_max_buffer_size 1000m;

http://s633.photobucket.com/user/C0nw0nk/media/Unt...

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251256#msg-251256
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2014-06-27 21:25
(Received via mailing list)
Which shows disk IO is much better which to me indicates there were/are
too
many small writes to disk, when some parts are slow tuning is a big time
issue with nginx no matter which OS your running.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251259#msg-251259
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-27 21:35
(Received via mailing list)
Hmm well i have figured out it is my mp4 buffers that need fixing but i
recon my largest video file size on the server is maybe 700mb as of
figuring
out what to set this to i am currently just playing around with it to
see
what works best.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251261#msg-251261
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-27 22:20
(Received via mailing list)
I think i found the soloution rather than buffer or envolve pseudo
streaming
mp4 already html5 compatible videos.

I just leave it to the browsers rather than my server.

So to solve my I/O usage issue i dropped "mp4;" from my server config
"#mp4;" and now my I/O usage is basically back at 0.

Perhaps nginx should look at the I/O usage to do with that function and
see
if they can make it better.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251264#msg-251264
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-27 22:30
(Received via mailing list)
My new soloution did not last very long everything shot up again so the
mp4
function is needed to drop I/O usage but as of what the optimal setting
for
the buffers are realy does baffle me

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251265#msg-251265
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2014-06-27 23:19
(Received via mailing list)
c0nw0nk Wrote:
-------------------------------------------------------
> Perhaps nginx should look at the I/O usage to do with that function
> and see if they can make it better.

Its a disk subsystem issue which is under control by the OS not nginx, a
good 15k sas does wonders.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251266#msg-251266
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-27 23:47
(Received via mailing list)
So a disk spinning at 15000 rpm compared to my current hard drive
spinning
at 7000 rpm does better than a SSD still ?

This is my current hard drive i posted earlyer i do believe
http://www.hgst.com/hard-drives/enterprise-hard-dr...

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251267#msg-251267
767bb1d9696ff15ae0114891842e5796?d=identicon&s=25 Paul Schlie (Guest)
on 2014-06-28 00:44
(Received via mailing list)
I don't know if what you're experiencing is related to a problem I'm
still tracking down, specifically that multiple redundant read-streams
and corresponding temp_files are being opened to read the same file from
a backend server for what appears to be a single initial get request by
a client for a large mp4 file which was not yet been locally reverse
proxy cashed by nginx as an substantially static file. This appears to
end up creating 6-10x more traffic and disk activity than is actually
required to cache the single file (depending on how many redundant
read-stream/temp_files are created.  If a server is attempting to
reverse proxy many such relatively large files, it could easily saturate
nginx with network/disk traffic until most such files requested were
eventually locally cached.
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-28 02:28
(Received via mailing list)
Paul Schlie Wrote:
-------------------------------------------------------
> easily saturate nginx with network/disk traffic until most such files
> > Posted at Nginx Forum:
> http://mailman.nginx.org/mailman/listinfo/nginx
Well i don't proxy anything everything is hosted locally and php is run
by
fastcgi but it is all on the same machine.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251270#msg-251270
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-28 06:38
(Received via mailing list)
Latest picture
http://s633.photobucket.com/user/C0nw0nk/media/Unt...

Everything utilizing the read's and writes is nginx and when i set the
following buffers i get massive spikes like that.

location ~ \.mp4$ {
mp4;
mp4_buffer_size 9000m;
mp4_max_buffer_size 9000m;
}

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251275#msg-251275
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-29 18:01
(Received via mailing list)
After benchmarking this was my output. I have no idea if it is good or
bad
or what, i am rather hoping someone with more understanding of I/O usage
and
if i have hit my max or not can tell me.

This is the version of crystal mark i benchmarked with
http://crystalmark.info/redirect.php?product=CrystalDiskMark
I used their 64bit not the 32bit.
-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   168.230 MB/s
          Sequential Write :   166.230 MB/s
         Random Read 512KB :    61.540 MB/s
        Random Write 512KB :    84.223 MB/s
    Random Read 4KB (QD=1) :     0.787 MB/s [   192.1 IOPS]
   Random Write 4KB (QD=1) :     1.701 MB/s [   415.3 IOPS]
   Random Read 4KB (QD=32) :     2.032 MB/s [   496.2 IOPS]
  Random Write 4KB (QD=32) :     1.389 MB/s [   339.0 IOPS]

  Test : 1000 MB [C: 21.9% (427.9/1953.1 GB)] (x9)
  Date : 2014/06/29 15:58:08
    OS : Windows Server 2008 R2 Web Server SP1 [6.1 Build 7601] (x64)

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251298#msg-251298
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-29 19:20
(Received via mailing list)
I ran a couple more tests with 2000mb then 4000mb
-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   167.799 MB/s
          Sequential Write :   166.560 MB/s
         Random Read 512KB :    59.075 MB/s
        Random Write 512KB :    83.099 MB/s
    Random Read 4KB (QD=1) :     0.760 MB/s [   185.6 IOPS]
   Random Write 4KB (QD=1) :     1.755 MB/s [   428.5 IOPS]
   Random Read 4KB (QD=32) :     2.003 MB/s [   489.0 IOPS]
  Random Write 4KB (QD=32) :     1.346 MB/s [   328.6 IOPS]

  Test : 2000 MB [C: 21.9% (427.9/1953.1 GB)] (x9)
  Date : 2014/06/29 16:57:08
    OS : Windows Server 2008 R2 Web Server SP1 [6.1 Build 7601] (x64)

-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :   168.459 MB/s
          Sequential Write :   166.944 MB/s
         Random Read 512KB :    58.245 MB/s
        Random Write 512KB :    81.734 MB/s
    Random Read 4KB (QD=1) :     0.741 MB/s [   180.8 IOPS]
   Random Write 4KB (QD=1) :     1.677 MB/s [   409.5 IOPS]
   Random Read 4KB (QD=32) :     1.943 MB/s [   474.4 IOPS]
  Random Write 4KB (QD=32) :     1.696 MB/s [   413.9 IOPS]

  Test : 4000 MB [C: 21.9% (427.9/1953.1 GB)] (x9)
  Date : 2014/06/29 17:20:31
    OS : Windows Server 2008 R2 Web Server SP1 [6.1 Build 7601] (x64)

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251300#msg-251300
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2014-06-30 19:54
(Received via mailing list)
Hello!

On Sat, Jun 28, 2014 at 12:38:12AM -0400, c0nw0nk wrote:

> }
Note that when using mp4 module, you should not set
mp4_buffer_size too high.  It's initial size of a buffer used to
read mp4 files, and if a value specified is bigger than size of
the moov atom in an mp4 file, reading extra bytes will be
meaningless.  It is generally recommended to set it to something
like 512k (the default) or slightly larger (1m, 2m).  Larger
values make sense only if most of your mp4 files have huge moov
atoms.

Note well that if you use mp4 pseudo streaming, you should make
sure that mp4 files are correctly prepared - i.e., with moov atom
placed before mdat.  While nginx will be able to work with files
with moov atom at the end, it won't be able to do this
efficiently.

--
Maxim Dounin
http://nginx.org/
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-06-30 20:21
(Received via mailing list)
Maxim Dounin Wrote:
-------------------------------------------------------
> >
> meaningless.  It is generally recommended to set it to something
> --
> Maxim Dounin
> http://nginx.org/
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


All videos are already set out like that i also use the following python
script to ensure that they have meta data at the start of the file
instead
of the end. https://gist.github.com/marpada/12e7d4a093aec402a989

I did a simple test to confirm it is I/O usage due to large files i run
nginx same traffic conditions but i block "deny all;" to mp4, flv files
and
my i/o usage by nginx is about 1mb max and a few kb so i can confirm it
is
large files that are doing it i am currently looking into what would be
better for me a SSD or a SAS 15K RPM.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251316#msg-251316
2974d09ac2541e892966b762aad84943?d=identicon&s=25 c0nw0nk (Guest)
on 2014-07-01 06:05
(Received via mailing list)
I actualy came accross a setting in my device manager called write cache
buffer flushing. When you disable Write Cache Buffer Flushing, this
allows
application software to blaze ahead after writing data to disk without
waiting for the physical write to complete.

http://noel.prodigitalsoftware.com/temp/WriteCache...

I have enabled it rebooted my machine and will post in a few hours how
things are going with it.

Now this i won't be a soloution but will deffinetly help write allot
more
data simultaneously. What should help until i find a SSD/SAS to dedicate
my
money to.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,251186,251347#msg-251347
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.