Uses of Package
crawlercommons.robots

Packages that use crawlercommons.robots

Package	Description
crawlercommons.robots	The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.

Classes in crawlercommons.robots used by crawlercommons.robots

Class and Description
BaseRobotRules Result from parsing a single robots.txt file - which means we get a set of rules, and a crawl-delay.
BaseRobotsParser
SimpleRobotRules.RobotRule Single rule that maps from a path prefix to an allow flag.
SimpleRobotRules.RobotRulesMode