Uses of Class
crawlercommons.robots.SimpleRobotRules

Packages that use SimpleRobotRules

Package	Description
crawlercommons.robots	The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.

Uses of SimpleRobotRules in crawlercommons.robots

Methods in crawlercommons.robots that return SimpleRobotRules

Modifier and Type	Method	Description
SimpleRobotRules	SimpleRobotRulesParser.failedFetch(int httpStatusCode)	The fetch of robots.txt failed, so return rules appropriate for the given HTTP status code.
SimpleRobotRules	SimpleRobotRulesParser.parseContent(String url, byte[] content, String contentType, String robotNames)	Deprecated.
SimpleRobotRules	SimpleRobotRulesParser.parseContent(String url, byte[] content, String contentType, Collection<String> robotNames)	Parse the robots.txt file in content, and return rules appropriate for processing paths by userAgent.