| Package | Description | 
|---|---|
| crawlercommons.fetcher.http | 
 This package concerns the fetching of files over the HTTP protocol: 
Extending from  
BaseHttpFetcher (which itself extends BaseFetcher) the 
SimpleHttpFetcher provides the Crawler Commons HTTP fetching implementation. | 
| crawlercommons.robots | 
 The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons. 
 | 
| Class and Description | 
|---|
| BaseHttpFetcher
 Deprecated. 
 
As of release 0.6. We recommend directly using Apache HttpClient,
             async-http-client, or any other robust, industrial-strength HTTP
             clients. 
 | 
| BaseHttpFetcher.RedirectMode
 Deprecated.  
  | 
| UserAgent
 Deprecated. 
 
As of release 0.6. We recommend directly using Apache HttpClient,
             async-http-client, or any other robust, industrial-strength HTTP
             clients. 
 | 
| Class and Description | 
|---|
| BaseHttpFetcher
 Deprecated. 
 
As of release 0.6. We recommend directly using Apache HttpClient,
             async-http-client, or any other robust, industrial-strength HTTP
             clients. 
 | 
| UserAgent
 Deprecated. 
 
As of release 0.6. We recommend directly using Apache HttpClient,
             async-http-client, or any other robust, industrial-strength HTTP
             clients. 
 | 
Copyright © 2009–2016 Crawler-Commons. All rights reserved.