crawlercommons.fetcher.http

Class BaseHttpFetcher

java.lang.Object

crawlercommons.fetcher.BaseFetcher

crawlercommons.fetcher.http.BaseHttpFetcher

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

SimpleHttpFetcher

Deprecated.

As of release 0.6. We recommend directly using Apache HttpClient, async-http-client, or any other robust, industrial-strength HTTP clients.

@Deprecated
public abstract class BaseHttpFetcher
extends BaseFetcher

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BaseHttpFetcher.RedirectMode Deprecated.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	_acceptLanguage Deprecated.
protected int	_maxConnectionsPerHost Deprecated.
protected int	_maxRedirects Deprecated.
protected int	_maxThreads Deprecated.
protected int	_minResponseRate Deprecated.
protected BaseHttpFetcher.RedirectMode	_redirectMode Deprecated.
protected UserAgent	_userAgent Deprecated.
static String	DEFAULT_ACCEPT_LANGUAGE Deprecated.
static int	DEFAULT_MAX_CONNECTIONS_PER_HOST Deprecated.
static int	DEFAULT_MAX_REDIRECTS Deprecated.
static int	DEFAULT_MIN_RESPONSE_RATE Deprecated.
static BaseHttpFetcher.RedirectMode	DEFAULT_REDIRECT_MODE Deprecated.
static int	NO_MIN_RESPONSE_RATE Deprecated.
static int	NO_REDIRECTS Deprecated.

Fields inherited from class crawlercommons.fetcher.BaseFetcher

_defaultMaxContentSize, _maxContentSizes, _validMimeTypes, DEFAULT_MAX_CONTENT_SIZE

Constructor Summary

Constructors

Constructor and Description
BaseHttpFetcher(int maxThreads, UserAgent userAgent) Deprecated.

Method Summary

Modifier and Type	Method and Description
String	getAcceptLanguage() Deprecated.
int	getMaxConnectionsPerHost() Deprecated.
int	getMaxRedirects() Deprecated.
int	getMaxThreads() Deprecated.
int	getMinResponseRate() Deprecated. Return the minimum response rate.
BaseHttpFetcher.RedirectMode	getRedirectMode() Deprecated.
UserAgent	getUserAgent() Deprecated.
void	setAcceptLanguage(String acceptLanguage) Deprecated.
void	setMaxConnectionsPerHost(int maxConnectionsPerHost) Deprecated.
void	setMaxRedirects(int maxRedirects) Deprecated.
void	setMinResponseRate(int minResponseRate) Deprecated.
void	setRedirectMode(BaseHttpFetcher.RedirectMode mode) Deprecated.

Methods inherited from class crawlercommons.fetcher.BaseFetcher

abort, addValidMimeType, addValidMimeTypes, get, get, getDefaultMaxContentSize, getMaxContentSize, getMimeTypeFromContentType, getValidMimeTypes, setDefaultMaxContentSize, setMaxContentSize, setValidMimeTypes

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NO_MIN_RESPONSE_RATE

public static final int NO_MIN_RESPONSE_RATE

Deprecated.

See Also:

Constant Field Values

NO_REDIRECTS

public static final int NO_REDIRECTS

Deprecated.

See Also:

Constant Field Values

DEFAULT_MIN_RESPONSE_RATE

public static final int DEFAULT_MIN_RESPONSE_RATE

Deprecated.

See Also:

Constant Field Values

DEFAULT_MAX_CONNECTIONS_PER_HOST

public static final int DEFAULT_MAX_CONNECTIONS_PER_HOST

Deprecated.

See Also:

Constant Field Values

DEFAULT_MAX_REDIRECTS

public static final int DEFAULT_MAX_REDIRECTS

Deprecated.

See Also:

Constant Field Values

DEFAULT_ACCEPT_LANGUAGE

public static final String DEFAULT_ACCEPT_LANGUAGE

Deprecated.

See Also:

Constant Field Values

DEFAULT_REDIRECT_MODE

public static final BaseHttpFetcher.RedirectMode DEFAULT_REDIRECT_MODE

Deprecated.

_maxThreads

protected int _maxThreads

Deprecated.

_userAgent

protected UserAgent _userAgent

Deprecated.

_maxRedirects

protected int _maxRedirects

Deprecated.

_maxConnectionsPerHost

protected int _maxConnectionsPerHost

Deprecated.

_minResponseRate

protected int _minResponseRate

Deprecated.

_acceptLanguage

protected String _acceptLanguage

Deprecated.

_redirectMode

protected BaseHttpFetcher.RedirectMode _redirectMode

Deprecated.

Constructor Detail

BaseHttpFetcher

public BaseHttpFetcher(int maxThreads,
                       UserAgent userAgent)

Deprecated.

Method Detail

getMaxThreads

public int getMaxThreads()

Deprecated.

getUserAgent

public UserAgent getUserAgent()

Deprecated.

setMaxConnectionsPerHost

public void setMaxConnectionsPerHost(int maxConnectionsPerHost)

Deprecated.

getMaxConnectionsPerHost

public int getMaxConnectionsPerHost()

Deprecated.

setMinResponseRate

public void setMinResponseRate(int minResponseRate)

Deprecated.

getMinResponseRate

public int getMinResponseRate()

Deprecated.

Return the minimum response rate. If the speed at which bytes are being returned from the server drops below this, the fetch of that page will be aborted.

Returns:

bytes/second

setAcceptLanguage

public void setAcceptLanguage(String acceptLanguage)

Deprecated.

getAcceptLanguage

public String getAcceptLanguage()

Deprecated.

setMaxRedirects

public void setMaxRedirects(int maxRedirects)

Deprecated.

getMaxRedirects

public int getMaxRedirects()

Deprecated.

setRedirectMode

public void setRedirectMode(BaseHttpFetcher.RedirectMode mode)

Deprecated.

getRedirectMode

public BaseHttpFetcher.RedirectMode getRedirectMode()

Deprecated.