crawlercommons.robots

Class BaseRobotRules

java.lang.Object

crawlercommons.robots.BaseRobotRules

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

SimpleRobotRules

public abstract class BaseRobotRules
extends Object
implements Serializable

Result from parsing a single robots.txt file - which means we get a set of rules, and a crawl-delay.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static long	UNSET_CRAWL_DELAY

Constructor Summary

Constructors

Constructor and Description
BaseRobotRules()

Method Summary

Modifier and Type	Method and Description
void	addSitemap(String sitemap)
boolean	equals(Object obj)
long	getCrawlDelay()
List<String>	getSitemaps()
int	hashCode()
abstract boolean	isAllowAll()
abstract boolean	isAllowed(String url)
abstract boolean	isAllowNone()
boolean	isDeferVisits()
void	setCrawlDelay(long crawlDelay)
void	setDeferVisits(boolean deferVisits)

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

UNSET_CRAWL_DELAY

public static final long UNSET_CRAWL_DELAY

See Also:

Constant Field Values

Constructor Detail

BaseRobotRules

public BaseRobotRules()

Method Detail

isAllowed

public abstract boolean isAllowed(String url)

isAllowAll

public abstract boolean isAllowAll()

isAllowNone

public abstract boolean isAllowNone()

getCrawlDelay

public long getCrawlDelay()

setCrawlDelay

public void setCrawlDelay(long crawlDelay)

isDeferVisits

public boolean isDeferVisits()

setDeferVisits

public void setDeferVisits(boolean deferVisits)

addSitemap

public void addSitemap(String sitemap)

getSitemaps

public List<String> getSitemaps()

hashCode

public int hashCode()

Overrides:

hashCode in class Object

equals

public boolean equals(Object obj)

Overrides:

equals in class Object