Package | Description |
---|---|
crawlercommons.fetcher.http |
This package concerns the fetching of files over the HTTP protocol:
Extending from
BaseHttpFetcher (which itself extends BaseFetcher ) the
SimpleHttpFetcher provides the Crawler Commons HTTP fetching implementation. |
crawlercommons.robots |
The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.
|
Modifier and Type | Field and Description |
---|---|
protected UserAgent |
BaseHttpFetcher._userAgent
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
UserAgent |
BaseHttpFetcher.getUserAgent()
Deprecated.
|
Constructor and Description |
---|
BaseHttpFetcher(int maxThreads,
UserAgent userAgent)
Deprecated.
|
SimpleHttpFetcher(int maxThreads,
UserAgent userAgent)
Deprecated.
|
SimpleHttpFetcher(UserAgent userAgent)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
static BaseHttpFetcher |
RobotUtils.createFetcher(UserAgent userAgent,
int maxThreads) |
Copyright © 2009–2016 Crawler-Commons. All rights reserved.