# Token bucket to limit bots and site grabbers

**URL:** <https://www.ruby-forum.com/t/token-bucket-to-limit-bots-and-site-grabbers/183482>\
**Category:** NGINX\
**Created:** [February 15, 2010, 11:34am UTC](https://www.ruby-forum.com/t/token-bucket-to-limit-bots-and-site-grabbers/183482 "2010-02-15T11:34:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tobia\_C](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a88e4f/32.png) [@Tobia\_C](https://www.ruby-forum.com/u/Tobia_C)\
**Post date:** [February 15, 2010, 11:34am UTC](https://www.ruby-forum.com/t/token-bucket-to-limit-bots-and-site-grabbers/183482/1 "2010-02-15T11:34:55Z")

</div>

Hello

Is there any module I can use to limit or deny access to bots and site  
grabbers, based on the long-term request rate?

I’m thinking of a token bucket with a timeframe of hours or days, where  
a legitimate user will only download, say, 50 pages (images and css  
excluded) per day, from a single ip address. Bots will obviously try and  
grab more content than that. Even if they set a long delay between  
requests, the overall number of requests per day will be much higher  
than that of a legitimate user.

limit\_req is not what I’m looking for, because it has a short timeframe  
of seconds or minutes, and because this kind of limit requires a token  
bucket, not a leaky bucket.

Is there anything available, or should I write my own module?

Tobia

---

<div class="post-metadata">

**Author:** ![Tobia\_C](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/a88e4f/32.png) [@Tobia\_C](https://www.ruby-forum.com/u/Tobia_C)\
**Post date:** [February 15, 2010, 12:34pm UTC](https://www.ruby-forum.com/t/token-bucket-to-limit-bots-and-site-grabbers/183482/2 "2010-02-15T12:34:36Z")

</div>

Hello!

On Mon, Feb 15, 2010 at 11:34:16AM +0100, Tobia C. wrote:

> limit\_req is not what I’m looking for, because it has a short  
> timeframe of seconds or minutes, and because this kind of limit  
> requires a token bucket, not a leaky bucket.

To turn limit\_req into token bucket it’s enough to specify  
“nodelay” flag.

It should be relatively easy to extend supported time frames, too.  
Not as easy as just adding another line of configuration parsing,  
but I believe it’s something that should be done.

Maxim D.