crawlercommons.robots

Class SimpleRobotRules

java.lang.Object

crawlercommons.robots.BaseRobotRules

crawlercommons.robots.SimpleRobotRules

All Implemented Interfaces:

Serializable

public class SimpleRobotRules
extends BaseRobotRules

Result from parsing a single robots.txt file - which means we get a set of rules, and a crawl-delay.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SimpleRobotRules.RobotRule Single rule that maps from a path prefix to an allow flag.
static class	SimpleRobotRules.RobotRulesMode

Field Summary

Fields inherited from class crawlercommons.robots.BaseRobotRules

UNSET_CRAWL_DELAY

Constructor Summary

Constructors

Constructor and Description
SimpleRobotRules()
SimpleRobotRules(SimpleRobotRules.RobotRulesMode mode)

Method Summary

Modifier and Type	Method and Description
void	addRule(String prefix, boolean allow)
void	clearRules()
boolean	equals(Object obj)
int	hashCode()
boolean	isAllowAll() Is our ruleset set up to allow all access?
boolean	isAllowed(String url)
boolean	isAllowNone() Is our ruleset set up to disallow all access?
void	sortRules() In order to match up with Google's convention, we want to match rules from longest to shortest.

Methods inherited from class crawlercommons.robots.BaseRobotRules

addSitemap, getCrawlDelay, getSitemaps, isDeferVisits, setCrawlDelay, setDeferVisits

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleRobotRules

public SimpleRobotRules()

SimpleRobotRules

public SimpleRobotRules(SimpleRobotRules.RobotRulesMode mode)

Method Detail

clearRules

public void clearRules()

addRule

public void addRule(String prefix,
                    boolean allow)

isAllowed

public boolean isAllowed(String url)

Specified by:

isAllowed in class BaseRobotRules

sortRules

public void sortRules()

In order to match up with Google's convention, we want to match rules from longest to shortest. So sort the rules.

isAllowAll

public boolean isAllowAll()

Is our ruleset set up to allow all access?

Specified by:

isAllowAll in class BaseRobotRules

Returns:

true if all URLs are allowed.

isAllowNone

public boolean isAllowNone()

Is our ruleset set up to disallow all access?

Specified by:

isAllowNone in class BaseRobotRules

Returns:

true if no URLs are allowed.

hashCode

public int hashCode()

Overrides:

hashCode in class BaseRobotRules

equals

public boolean equals(Object obj)

Overrides:

equals in class BaseRobotRules