Hi there,
I’m looking for a ruby gem supporting the robots.txt protocol such as
described at:
http://www.robotstxt.org/
I’ve found this one:
http://groups.google.com/group/comp.lang.ruby/msg/16f16935bf83c8c6?dmode=source&output=gplain
on comp.lang.ruby but it doesn’t support it totally.
Any other pointer is welcomed.
Cheers,