crawlercommons.robots

Class RobotUtils

java.lang.Object

crawlercommons.robots.RobotUtils

public class RobotUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
RobotUtils()

Method Summary

Modifier and Type	Method and Description
static BaseHttpFetcher	createFetcher(BaseHttpFetcher fetcher)
static BaseHttpFetcher	createFetcher(UserAgent userAgent, int maxThreads)
static long	getMaxFetchTime()
static BaseRobotRules	getRobotRules(BaseHttpFetcher fetcher, BaseRobotsParser parser, URL robotsUrl) Externally visible, static method for use in tools and for testing.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RobotUtils

public RobotUtils()

Method Detail

createFetcher

public static BaseHttpFetcher createFetcher(BaseHttpFetcher fetcher)

createFetcher

public static BaseHttpFetcher createFetcher(UserAgent userAgent,
                                            int maxThreads)

getMaxFetchTime

public static long getMaxFetchTime()

getRobotRules

public static BaseRobotRules getRobotRules(BaseHttpFetcher fetcher,
                                           BaseRobotsParser parser,
                                           URL robotsUrl)

Externally visible, static method for use in tools and for testing. Fetch the indicated robots.txt file, parse it, and generate rules.

Parameters:

fetcher - Fetcher for downloading robots.txt file

parser - a BaseRobotsParser to use for obtaining appropriate rules

robotsUrl - URL to robots.txt file

Returns:

Robot rules