# Detect suspicious activity with nginx

**URL:** <https://www.ruby-forum.com/t/detect-suspicious-activity-with-nginx/210506>\
**Category:** NGINX\
**Created:** [August 9, 2011, 11:17pm UTC](https://www.ruby-forum.com/t/detect-suspicious-activity-with-nginx/210506 "2011-08-09T23:17:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\
**Post date:** [August 9, 2011, 11:17pm UTC](https://www.ruby-forum.com/t/detect-suspicious-activity-with-nginx/210506/1 "2011-08-09T23:17:43Z")

</div>

Hi guys

We are looking for a way to detect suspicious activity on high-traffic  
websites. Parsing log files is not good option here, our current nginx  
config generates around 90G of logs for around 412K http requests each  
days.

We are looking to use nginx to detect suspicious activity and generate  
precise log when it happens for post-processing.

Some tools we are looking for would be something like

- Detect IPs which accessed /uri1/ X times without accessing other URI  
in a period of time Y.

- Detect IPs that are indexing our site by accessing sequential uris  
like /uri123, /uri124, /uri125, …

We are using load balancing services (haproxy), we enabled realip module  
in nginx, we need something that can work with it.

If you have any pointers / ideas / module names that could help us,  
please let me know.

Have a good day

Max

---

<div class="post-metadata">

**Author:** ![Maxime\_Ducharme](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ad7895/32.png) [@Maxime\_Ducharme](https://www.ruby-forum.com/u/Maxime_Ducharme)\
**Post date:** [August 10, 2011, 2:49am UTC](https://www.ruby-forum.com/t/detect-suspicious-activity-with-nginx/210506/2 "2011-08-10T02:49:02Z")

</div>

Hi Max,

In my opinion you dont want to rely on nginx to do the analytics simply  
to  
log suspicious activity but rather need to look at a better  
log analyzing solution. Have you checked out splunk? It is a very  
powerful  
log analyzer that will allow you to more intelligently parse the logs  
and  
has a free licence.

Rami

---

<div class="post-metadata">

**Author:** ![Maxime\_Ducharme](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ad7895/32.png) [@Maxime\_Ducharme](https://www.ruby-forum.com/u/Maxime_Ducharme)\
**Post date:** [August 10, 2011, 2:55am UTC](https://www.ruby-forum.com/t/detect-suspicious-activity-with-nginx/210506/3 "2011-08-10T02:55:11Z")

</div>

I would also look into some kind of blacklist, like  
spam/drone/proxy/etc, to compare IP addresses that are suspicious and  
kindly redirect them to /dev/null

-C

---

<div class="post-metadata">

**Author:** ![Maxime\_Ducharme](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/ad7895/32.png) [@Maxime\_Ducharme](https://www.ruby-forum.com/u/Maxime_Ducharme)\
**Post date:** [August 13, 2011, 1:38am UTC](https://www.ruby-forum.com/t/detect-suspicious-activity-with-nginx/210506/4 "2011-08-13T01:38:37Z")

</div>

You can try ides from [GitHub - znarf/bouncer: A PHP library to log and analyse HTTP traffic, throttle and block suspicious agents.](https://github.com/znarf/Bouncer) and move then  
into  
Lua or Perl.