| Package | Description | 
|---|---|
| crawlercommons.fetcher.http | 
 This package concerns the fetching of files over the HTTP protocol: 
Extending from  
BaseHttpFetcher (which itself extends BaseFetcher) the 
SimpleHttpFetcher provides the Crawler Commons HTTP fetching implementation. | 
| crawlercommons.robots | 
 The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons. 
 | 
| Modifier and Type | Field and Description | 
|---|---|
protected UserAgent | 
BaseHttpFetcher._userAgent
Deprecated.  
  | 
| Modifier and Type | Method and Description | 
|---|---|
UserAgent | 
BaseHttpFetcher.getUserAgent()
Deprecated.  
  | 
| Constructor and Description | 
|---|
BaseHttpFetcher(int maxThreads,
               UserAgent userAgent)
Deprecated.  
  | 
SimpleHttpFetcher(int maxThreads,
                 UserAgent userAgent)
Deprecated.  
  | 
SimpleHttpFetcher(UserAgent userAgent)
Deprecated.  
  | 
| Modifier and Type | Method and Description | 
|---|---|
static BaseHttpFetcher | 
RobotUtils.createFetcher(UserAgent userAgent,
             int maxThreads)  | 
Copyright © 2009–2016 Crawler-Commons. All rights reserved.