| Package | Description |
|---|---|
| crawlercommons.fetcher.http |
This package concerns the fetching of files over the HTTP protocol:
Extending from
BaseHttpFetcher (which itself extends BaseFetcher) the
SimpleHttpFetcher provides the Crawler Commons HTTP fetching implementation. |
| crawlercommons.robots |
The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.
|
| Modifier and Type | Field and Description |
|---|---|
protected UserAgent |
BaseHttpFetcher._userAgent
Deprecated.
|
| Modifier and Type | Method and Description |
|---|---|
UserAgent |
BaseHttpFetcher.getUserAgent()
Deprecated.
|
| Constructor and Description |
|---|
BaseHttpFetcher(int maxThreads,
UserAgent userAgent)
Deprecated.
|
SimpleHttpFetcher(int maxThreads,
UserAgent userAgent)
Deprecated.
|
SimpleHttpFetcher(UserAgent userAgent)
Deprecated.
|
| Modifier and Type | Method and Description |
|---|---|
static BaseHttpFetcher |
RobotUtils.createFetcher(UserAgent userAgent,
int maxThreads) |
Copyright © 2009–2016 Crawler-Commons. All rights reserved.