The nginx forum had a DDoS attack which took the site down this morning.
In approximately 23 seconds there were just under 900,000 lines in the
error log that looked like:
2009/10/09 10:21:38 [alert] 32576#0: accept() failed (24: Too many open
files)
First question is do each of these entries represent an attempted
connection?
Looking at the access log there were thousands of requests for the same
page from roughly 400 IP’s in that same 23 second span like this:
58.53.85.229 - - [09/Oct/2009:10:21:38 -0400] “GET /list.php?2,page=1
HTTP/1.1” 500 553 “http://forum.nginx.org” “Mozilla/4.0
(compatible; MSIE 6.0; Windows 5.1)”
60.177.29.231 - - [09/Oct/2009:10:21:38 -0400] “GET /list.php?2,page=1
HTTP/1.1” 500 553 “http://forum.nginx.org” “Mozilla/4.0
(compatible; MSIE 6.0; Windows 5.1)”
125.91.207.11 - - [09/Oct/2009:10:21:38 -0400] “GET /list.php?2,page=1
HTTP/1.1” 500 553 “http://forum.nginx.org” “Mozilla/4.0
(compatible; MSIE 6.0; Windows 5.1)”
125.119.65.194 - - [09/Oct/2009:10:21:38 -0400] “GET /list.php?2,page=1
HTTP/1.1” 500 553 “http://forum.nginx.org” “Mozilla/4.
0 (compatible; MSIE 6.0; Windows 5.1)”
All of the IP’s that I checked were legitimate IP’s that localize to
various regions in China.
I have set up limit_zone and limit_conn directives to hopefully mitigate
this in the future.
Second question is where to set limit_conn and what are the effects on
users if set low? The site generally responds quickly, at least here in
the US, and I don’t want it to be especially sluggish for people using
less fast connections in other parts of the world, but of course I want
to reduce the chances of this happening again. Bear in mind this is a
low traffic site (16K visits in the last month) on a small VPS.
Any advice would be appreciated.
On Fri, Oct 9, 2009 at 4:40 PM, Jim O. [email protected] wrote:
HTTP/1.1" 500 553 “http://forum.nginx.org” "Mozilla/4.0
this in the future.
Jim O.
Jim,
Your main issue is going ot be the system ‘open files’ setting, can
you show me the output of the following please, run the command as
room or the user that runs the nginx proc. ‘ulimit -a’
by default linux sets this value to which really means that you cant
have more than 1024 file descriptors open on the system at one time.
You can modify this setting to safely allow up to 150,000 on a more
resent dual cor box.
‘ulimit -n 150000’
try reading up on ulimit, its a huge for system performance, also make
sure your not running any iptables connection tracking else you will
need to fine tune those settings as well.
Payam C. wrote:
125.91.207.11 - - [09/Oct/2009:10:21:38 -0400] "GET /list.php?2,page=1
I have set up limit_zone and limit_conn directives to hopefully mitigate
by default linux sets this value to which really means that you cant
have more than 1024 file descriptors open on the system at one time.
You can modify this setting to safely allow up to 150,000 on a more
resent dual cor box.
‘ulimit -n 150000’
It’s hardly a dual core box. 
It’s a small VPS with 256 MB of RAM that
has a great deal of room to spare. Once again, we only receive ~500
visitors per day on average. Open files were set at 4096 which I
thought would have been more than enough for the (one) nginx worker
process max connections which is set at 1024. I have already increased
the number to 32768. I’m not totally convinced that will do much in this
situation given the intensity of the attack on such a small system. I
would think that eventually some system resource would have given even
if file descriptors had not run out, though I must admit that I have
been fortunate not to have dealt with this much in the past.
IMHO,
I have a bad experience using csf for a high traffic website.
Best Regards,
Glen L.
Need a larger vps? Lol
If it’s virtuozzo (perhaps Xen though I don’t have a lot of experience
that
that) your going to hit open file limits put on the instance as well.
Even
with sophisticated software based firewalls you can hit arbitrary
iptable
entry limits as well.
csf has a portflood feature that works fairly well that uses the
ipt_recent
module
or there are connection based monitors such as dos-deflate for
firewalling
ip’s that reach connection thresholds.
Though really you connection limit was probably the biggest issue. I
could
see nginx handling simple requests like that no sweat if you have
caching
enabled.
Chris Z. wrote:
Need a larger vps? Lol
I don’t really think so. Not for 500 visitors/day. It rarely uses even
half the allotted RAM and loads are generally low. If I recall correctly
two months ago we only had ~10,000 visitors so there is some growth but
I don’t anticipate it outgrowing its present confines any time soon. I
added access to the Russian mailing list last month and that has
seemingly attracted more visitors to the site from Russia and Ukraine,
and to a lesser extent from the Baltic nations, Belarus, and Kazakhstan.
If it’s virtuozzo (perhaps Xen though I don’t have a lot of experience
that that) your going to hit open file limits put on the instance as
well. Even with sophisticated software based firewalls you can hit
arbitrary iptable entry limits as well.
It’s a Virtuozzo VPS. I own the node so I can set the limits any way I
like. I could make the VPS huge (the node has A LOT of RAM) but there
seems little point. The volume does not justify it.
csf has a portflood feature that works fairly well that uses the
ipt_recent module
or there are connection based monitors such as dos-deflate for
firewalling ip’s that reach connection thresholds.
This was a few hundred IP’s at least in a 23 second period. The site was
evidently unresponsive from the first second of the attack (given the
500 responses) and dead at 23 seconds. I’m not sure that all of the
requests got logged before nginx was killed. How quickly do those tools
respond? I have a basic firewall installed and did not consider this
site likely to generate a DoS attack. It’s hardly controversial. 
Though really you connection limit was probably the biggest issue. I
could see nginx handling simple requests like that no sweat if you have
caching enabled.
I’ve added connection limits. Not going to publish the number but it’s
low.
I’m still curious if each of those lines in the error log represented
one request. If so, almost 900,000 logged requests in less than half a
minute was an atomic bomb for this little ant-sized server.
Maxim D. wrote:
First question is do each of these entries represent an attempted
connection?
No. This is configuration issue lead to infinite loop which can’t
be resolved until more files can be opened.
Thank you. I found it hard to imagine that there were that many requests
in such a small period directed at such a small site.
Hello!
On Fri, Oct 09, 2009 at 07:40:57PM -0400, Jim O. wrote:
The nginx forum had a DDoS attack which took the site down this
morning. In approximately 23 seconds there were just under 900,000
lines in the error log that looked like:
2009/10/09 10:21:38 [alert] 32576#0: accept() failed (24: Too many
open files)
First question is do each of these entries represent an attempted
connection?
No. This is configuration issue lead to infinite loop which can’t
be resolved until more files can be opened.
Since this looks like common configuration issue, probably we need
some accept pause in such situations…
Maxim D.