# Robots.txt protocol support for Ruby

**URL:** <https://www.ruby-forum.com/t/robots-txt-protocol-support-for-ruby/72031>\
**Category:** Ruby\
**Created:** [October 13, 2006, 6:58pm UTC](https://www.ruby-forum.com/t/robots-txt-protocol-support-for-ruby/72031 "2006-10-13T18:58:25Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nicolas\_DesprSSSSs](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/n/74df32/32.png) [@Nicolas\_DesprSSSSs](https://www.ruby-forum.com/u/Nicolas_DesprSSSSs)\
**Post date:** [October 13, 2006, 6:58pm UTC](https://www.ruby-forum.com/t/robots-txt-protocol-support-for-ruby/72031/1 "2006-10-13T18:58:25Z")

</div>

Hi there,

I’m looking for a ruby gem supporting the robots.txt protocol such as  
described at:

> **[robots.txt](https://en.wikipedia.org/wiki/Robots.txt)**
>
> robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.
> This relies on voluntary compliance. Not all robots comply with the standard; email harvesters, spambots, malware and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out.
> The "robots.txt" file c...

[http://www.robotstxt.org/](http://www.robotstxt.org/)

> **[An Extended Standard for Robot Exclusion](https://www.conman.org/people/spc/robots2.html)**
>
> A proposed update to the Robot Exclusion Standard.

I’ve found this one:

[http://groups.google.com/group/comp.lang.ruby/msg/16f16935bf83c8c6?dmode=source&output=gplain](http://groups.google.com/group/comp.lang.ruby/msg/16f16935bf83c8c6?dmode=source&output=gplain)

on comp.lang.ruby but it doesn’t support it totally.  
Any other pointer is welcomed.

Cheers,