Hey,
Why not utilize net filters/iptables for this? Setup a redirect to say
localhost:81n then create a script to capture uri on each request, then
use the information as u like… “Ddos mitigation based on client
behavior and feedback loop which tells ur edge device how handle that
traffic”
If u need Help just email me off list and I can help you, no charge.
Payam
Hi Payam. Thanks for the offer, and you raise a valid point - it can be
done in a script as you say, but that means a lot more work for the
server, compared to handling it in a native nginx module, written in C.
Typically native C code is multiples faster than php, python, etc. This
is key if you are trying to stave off a DDoS attack coming in with tens
of thousands of attack requests every second.
Posted at Nginx Forum:
Hi,
Agreed, what Ive done in the past to get around that issue is to setup a
span port on our edge so it takes a packet and mirrors it to another
server, say nic1. You run a script on that server that does all the
number crunching, based on what it sees, you can have your script modify
routing on the edge router, inject iptables rules into your server or
any gw devices above the server. You can then not only provide a
layer3-4 protection (while taking away the immediate threat away) but
now can allow the attack to go on for say 1-5 min, monitor the uri and
log files and create a behavior for the traffic which then you can block
dirty and allow good traffic back in.
having a feedback loop system allows you to get rid of any false
positives. If say a good ip is somehow redirected to localhost:81 (where
there is a valid link with captcha saying “if you are in fact a GOOD
user, answer the question and click [go]”. Then have that action inject
a cookie into the session which then gets matched at the edge and passes
down to the proper segment.
there are dozens of ways to mitigate the issue, just depends on how you
want to go about it. Ive worked on designing advance ddos mitigation
networks/software and server based appliances.
Having said this, i do think that nginx requires a native ddos
mitigation module, it would save a lot of time and effort in the long
run! =)
Posted at Nginx Forum:
You can then not only provide a layer3-4
fact a GOOD user, answer the question and click ".
Then have that action inject a cookie into the
session which then gets matched at the edge and
passes down to the proper segment.there are dozens of ways to mitigate the issue,
just depends on how you want to go about it. Ive
worked on designing advance ddos mitigation
networks/software and server based appliances.
This would work to mitigate a small DDoS, but if they bring the bigger
guns out, a massive problem using the DDoS deflate mechanism you
describe here (blocking the attackers with IP tables, or other SW based
firewall) becomes impossible, because the Linux kernel (in the case of
IP tables) will spend HUGE amounts of cpu time scrutinizing all network
traffic, making sure its not going to or from a blocked IP. I
experienced this first hand after IP tables reached around 20k blocked
IPs. The attack was getting blocked effectively, but the machines load
was now higher than it was before the blocking began, due to all this
processing the kernel is forced to do on behalf of the firewall.
So I found that it was a lot more effective to just use the nginx limit
modules to deny the attackers ability to simply request any amount of
pages they wanted. It still meant serving some pages to the attackers,
but at least the kernel did not have to devote 90% of its time to IP
table processing.
Having said this, i do think that nginx requires a
native ddos mitigation module, it would save a lot
of time and effort in the long run! =)
I concur. I’ve seen a few DDoS attacks before which could easily be
mitigated with DDoS deflate techniques, but a module like I described
above sure would be helpful when the attack grows more serious.
Posted at Nginx Forum:
(max-src-conn 100, max-src-conn-rate 15/5,
overload flush
global)That takes care of all my DDoS protection needs.
Some of y’all mentioned big
guns though, I don’t know about that.
OpenBSDs PF is indeed the worlds finest software based firewall, I’ll be
the first to say. I think Linux should throw out IP tables and go for a
PF port, but I digress.
I haven’t tried mitigating a big DDoS with PF, and I don’t know if it
would fare any better once it has say 50k individual IPs to block. But
to me that is kind of beside the point. If I am not mistaken, a well
written nginx module would be the immensely helpful when faced with the
kind of DDoS I had on me last week.
If I can’t find anyone interested in writing it I might have a whack at
it myself next time I get some spare time.
Posted at Nginx Forum:
Just real quick:
What about one of the BSDs and pf? The latter is said to be the world’s
best
firewall. Real elegant syntax too:
block quick from
pass in on $ext_if inet proto tcp from any to any port 80 keep state
(max-src-conn 100, max-src-conn-rate 15/5, overload flush
global)
That takes care of all my DDoS protection needs. Some of y’all mentioned
big
guns though, I don’t know about that.
Thanks!
Weibin,
whats your email? ill contact you with a few things
cheers
Payam
malte at 2010-11-5 3:47 wrote:
If I can’t find anyone interested in writing it I might have a whack at
it myself next time I get some spare time.
We are facing the similar DDOS situation to you. I’m developing a module
which can deny the individual IPs. The module can get the IPs with a
POST request from a commander server in the intranet. If you have some
suggestions, you can contact to me.
The module will be here:
GitHub - yaoweibin/nginx_limit_access_module: support to deny specific variable with HTTP POST interface, but I need some
more days to finish it.
Posted at Nginx Forum:
Re: DDoS protection module suggestion
nginx mailing list
[email protected]
nginx Info Page
–
Weibin Y.
Weibin Y. Wrote:
dule, but I need some
more days to finish it.
Wonderful!
Being able to interrogate the server for a list of bad IPs is an
excellent idea, it would allow people to make their own firewall-block
scripts etc.
The main suggestion I have is that the module supports this kind of
rule:
If an IP has requested more than X pages in the last Y seconds, then
serve only 503 errors to that IP for the next Z seconds, and use at most
W megabytes of RAM for the bad-IP pool.
Posted at Nginx Forum:
Payam C. at 2010-11-5 12:03 wrote:
Weibin,
whats your email? ill contact you with a few things
yaoweibin AT gmail.comWhat about one of the BSDs and pf? The latter is
That takes care of all my DDoS protection needs.
would fare any better once it has say 50k individual IPs to block. But
have some suggestions, you can contact to me.
[email protected]
nginx Info Page
–
Weibin Y.
malte at 2010-11-5 13:58 wrote:
The module will be here:
The main suggestion I have is that the module supports this kind of
rule:
If an IP has requested more than X pages in the last Y seconds, then
serve only 503 errors to that IP for the next Z seconds,
Is it the similar feature with the limit_request module? Except the
limit_request module still is servicing the requests under the burst
value.
and use at most
W megabytes of RAM for the bad-IP pool.
Yes, I will use a fixed allocated shared memory to store the big bad-IP
hash table.
Posted at Nginx Forum:
Re: DDoS protection module suggestion
nginx mailing list
[email protected]
nginx Info Page
–
Weibin Y.
Hey,
Instead of a 503, i would redirect them localhost:81 and allow them to
validly themselves via captcha system in case its a false positive.
Like above, if a host logs the same src_ip more than $x times in $xy
min, u should be moving the acl up the chain, your sub-distribution,
distribution cor or even edge routers.
my 2 cents
-Payam
Hi,
On 05/11/2010 11:51, Weibin Y. wrote:
I think it’s good to divide the determination from the Nginx. It’s
hard to determine the IP by single Nginx whether is good or bad.
Actually we have 20+ reverse proxy Nginx servers in the front. Each
Nginx doesn’t known others status. In our DDOS attack, the bad-IP’s
request rate is a little higher than the normal request.
I agree it’s a good idea to split the determination, and I think it
might be good to put the lookup code inside the get handler for the
variable - so that the lookup is only made if it is required.
You might want to also think about having a setting to check for the
existence or value of a cookie before doing the IP lookup - to avoid
unnecessary overhead. It might also be better to handle the setting of
the cookie value inside your Nginx module, since it would make keeping
the generation and checking of cookie values consistent easier to
manage.
You could perhaps handle the setting / value of the cookie inside Nginx,
and have a system similar to Maxim’s auth_request module - whereby a
subrequest which would check the reCaptcha (or whatever) value, and
return 200 for success or anything else for failure.
You could have directives like :
limit_access_cookie [cookie_name];
limit_access_cookie_str [cookie_value];
and you might want to add optional hashing (e.g. MD5) of the cookie
string, to make it harder for determined hackers to get past cookie
authentication - e.g.
limit_access_cookie_hash md5;
I think the overhead of checking hashed values of a cookie wouldn’t be
too high, since in most cases under DDoS, the cookie wouldn’t exist, so
you’d generally only be hashing for genuine users.
Just a few ideas, anyway. Good luck with it!
Marcus.
Thanks, you give me some good ideas.
2010/11/5 Eugaia [email protected]
I agree it’s a good idea to split the determination, and I think it might
be good to put the lookup code inside the get handler for the variable - so
that the lookup is only made if it is required.You might want to also think about having a setting to check for the
existence or value of a cookie before doing the IP lookup - to avoid
unnecessary overhead. It might also be better to handle the setting of the
cookie value inside your Nginx module, since it would make keeping the
generation and checking of cookie values consistent easier to manage.
The IP lookup overhead is very low and quick, I allocated a big hash
table.
You could perhaps handle the setting / value of the cookie inside Nginx,
and have a system similar to Maxim’s auth_request module - whereby a
subrequest which would check the reCaptcha (or whatever) value, and return
200 for success or anything else for failure.
I think the overhead of checking hashed values of a cookie wouldn’t be too
high, since in most cases under DDoS, the cookie wouldn’t exist, so you’d
generally only be hashing for genuine users.
I think it’s a new and useful feature and should develop a different
module.
I will have a try after my limit_access module.
Just a few ideas, anyway. Good luck with it!
Thank you.
Payam C. at 2010-11-5 15:30 wrote:
Hey,
Instead of a 503, i would redirect them localhost:81 and allow them to
validly themselves via captcha system in case its a false positive.
Maybe I could add extra variable like this:
if ($limit_access_deny) {
add_header Location http://xxxx:81/;
return 302;
}
Like above, if a host logs the same src_ip more than $x times in $xy
min, u should be moving the acl up the chain, your sub-distribution,
distribution cor or even edge routers.
I think it’s good to divide the determination from the Nginx. It’s hard
to determine the IP by single Nginx whether is good or bad. Actually we
have 20+ reverse proxy Nginx servers in the front. Each Nginx doesn’t
known others status. In our DDOS attack, the bad-IP’s request rate is a
little higher than the normal request.
We decide to collect the log together and analyze it. I don’t know the
payload of log collection. Maybe it’s too high. We have not done the
performance test yet. Or we should do log analysis distributed in each
server and then collect the results together.
module can
Being able to interrogate the server for a list of bad IPs is an
Re: DDoS protection module suggestion
nginx mailing list
[email protected]
nginx Info Page
–
Weibin Y.
Maybe I could add extra variable like this:
if ($limit_access_deny) {
add_header Location http://xxxx:81/;
return 302;
}
Would work nicely.
We decide to collect the log together and analyze
it. I don’t know the
payload of log collection. Maybe it’s too high. We
have not done the
performance test yet. Or we should do log analysis
distributed in each
server and then collect the results together.
Hms. In my set up, I have 3 machines each running nginx. They all have
their own public IP, and I simply let DNS round robin in the requests to
them. When I was hit, all machines were hit simultaneously, but the
individual bots attacking each targeted one machine only.
I take it you have some sort of load balancer in front that distributes
your incoming traffic differently from me?
Posted at Nginx Forum:
distribution cor or even edge routers.
It would be nice to have it configurable either way, but when you are
hit with a 50k bot attack and you have IPs requesting 50 pages per
second, you want to put them down immediately, not spend server time
serving them a dynamic captcha page.
Posted at Nginx Forum:
also on the captcha you can implement cookie
the last time i setup a network to handle 400mbps
and 140k connection
(not packets) a second attack it was with the
suggestions and topology
ive described, its worked without issues for me
but perhaps you are
seeing something that i have not.
Yeah I’m not saying you are wrong at all. But I can vouch for that it
was a decidedly bad idea to block 50k IPs in IPtables like I did, that
made all network related activity slower than a dying turtle. And
personally, for an IP requesting 50 pages per second, I don’t feel bad
at all 503:ing them instead of giving them a captcha chance. For a lower
intensity captcha I can see how your captcha system would shine though.
I’d love to see a flexible nginx module that can support either
approach. And this one that Weibin is working on sounds pretty
promising!
Posted at Nginx Forum:
them instead of giving them a captcha chance. For
a lower intensity captcha I can see how your
captcha system would shine though.
I meant of course a lower intensity DDoS, not captcha.
Posted at Nginx Forum:
2010/11/6 malte [email protected]
to determine the IP by single Nginx whether is
have not done the
Yeswe use LVS in front of Nginx as the load banlancer.
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
Sponsor our Newsletter | Privacy Policy | Terms of Service | Remote Ruby Jobs