Uses of Class
crawlercommons.robots.BaseRobotRules

Packages that use BaseRobotRules

Package	Description
crawlercommons.robots	The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.

Uses of BaseRobotRules in crawlercommons.robots

Subclasses of BaseRobotRules in crawlercommons.robots

Modifier and Type	Class	Description
class	SimpleRobotRules	Result from parsing a single robots.txt file – a set of allow/disallow rules to check whether a given URL is allowed, and optionally a Crawl-delay and Sitemap URLs.

Methods in crawlercommons.robots that return BaseRobotRules

Modifier and Type	Method	Description
abstract BaseRobotRules	BaseRobotsParser.failedFetch(int httpStatusCode)	The fetch of robots.txt failed, so return rules appropriate for the given HTTP status code.
abstract BaseRobotRules	BaseRobotsParser.parseContent(String url, byte[] content, String contentType, String robotNames)	Deprecated. since 1.4 - replaced by BaseRobotsParser.parseContent(java.lang.String,byte[],java.lang.String,java.util.Collection<java.lang.String>).
abstract BaseRobotRules	BaseRobotsParser.parseContent(String url, byte[] content, String contentType, Collection<String> robotNames)	Parse the robots.txt file in content, and return rules appropriate for processing paths by userAgent.