crawlercommons.robots

Class BaseRobotRules

java.lang.Object

crawlercommons.robots.BaseRobotRules

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

SimpleRobotRules

public abstract class BaseRobotRules
extends Object
implements Serializable

Result from parsing a single robots.txt file - which means we get a set of rules, and a crawl-delay.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static long	UNSET_CRAWL_DELAY

Constructor Summary

Constructors

Constructor and Description
BaseRobotRules()

Method Summary

Modifier and Type	Method and Description
void	addSitemap(String sitemap) Add sitemap URL to rules if not a duplicate
boolean	equals(Object obj)
long	getCrawlDelay()
List<String>	getSitemaps() Get URLs of sitemap links found in robots.txt
int	hashCode()
abstract boolean	isAllowAll()
abstract boolean	isAllowed(String url)
abstract boolean	isAllowNone()
boolean	isDeferVisits()
void	setCrawlDelay(long crawlDelay)
void	setDeferVisits(boolean deferVisits)
String	toString() Returns a string with the crawl delay as well as a list of sitemaps if they exist (and aren't more than 10)

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

UNSET_CRAWL_DELAY

public static final long UNSET_CRAWL_DELAY

See Also:

Constant Field Values

Constructor Detail

BaseRobotRules

public BaseRobotRules()

Method Detail

isAllowed

public abstract boolean isAllowed(String url)

isAllowAll

public abstract boolean isAllowAll()

isAllowNone

public abstract boolean isAllowNone()

getCrawlDelay

public long getCrawlDelay()

setCrawlDelay

public void setCrawlDelay(long crawlDelay)

isDeferVisits

public boolean isDeferVisits()

setDeferVisits

public void setDeferVisits(boolean deferVisits)

addSitemap

public void addSitemap(String sitemap)

Add sitemap URL to rules if not a duplicate

getSitemaps

public List<String> getSitemaps()

Get URLs of sitemap links found in robots.txt

hashCode

public int hashCode()

Overrides:

hashCode in class Object

equals

public boolean equals(Object obj)

Overrides:

equals in class Object

toString

public String toString()

Returns a string with the crawl delay as well as a list of sitemaps if they exist (and aren't more than 10)

Overrides:

toString in class Object