crawlercommons.fetcher

Class BaseFetcher

java.lang.Object

crawlercommons.fetcher.BaseFetcher

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

BaseHttpFetcher, SimpleFileFetcher

Deprecated.

As of release 0.6. We recommend directly using Apache HttpClient, async-http-client, or any other robust, industrial-strength HTTP clients.

@Deprecated
public abstract class BaseFetcher
extends Object
implements Serializable

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	_defaultMaxContentSize Deprecated.
protected Map<String,Integer>	_maxContentSizes Deprecated.
protected Set<String>	_validMimeTypes Deprecated.
static int	DEFAULT_MAX_CONTENT_SIZE Deprecated.

Constructor Summary

Constructors

Constructor and Description
BaseFetcher() Deprecated.

Method Summary

Modifier and Type	Method and Description
abstract void	abort() Deprecated. Terminate any async request being processed.
void	addValidMimeType(String validMimeType) Deprecated.
void	addValidMimeTypes(Set<String> validMimeTypes) Deprecated.
FetchedResult	get(String url) Deprecated.
abstract FetchedResult	get(String url, Payload payload) Deprecated. Get the content stored in the resource referenced by the 'url' parameter.
int	getDefaultMaxContentSize() Deprecated.
int	getMaxContentSize(String mimeType) Deprecated.
protected static String	getMimeTypeFromContentType(String contentType) Deprecated.
Set<String>	getValidMimeTypes() Deprecated.
void	setDefaultMaxContentSize(int defaultMaxContentSize) Deprecated.
void	setMaxContentSize(String mimeType, int maxContentSize) Deprecated.
void	setValidMimeTypes(Set<String> validMimeTypes) Deprecated.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_MAX_CONTENT_SIZE

public static final int DEFAULT_MAX_CONTENT_SIZE

Deprecated.

See Also:

Constant Field Values

_maxContentSizes

protected Map<String,Integer> _maxContentSizes

Deprecated.

_defaultMaxContentSize

protected int _defaultMaxContentSize

Deprecated.

_validMimeTypes

protected Set<String> _validMimeTypes

Deprecated.

Constructor Detail

BaseFetcher

public BaseFetcher()

Deprecated.

Method Detail

setDefaultMaxContentSize

public void setDefaultMaxContentSize(int defaultMaxContentSize)

Deprecated.

getDefaultMaxContentSize

public int getDefaultMaxContentSize()

Deprecated.

setMaxContentSize

public void setMaxContentSize(String mimeType,
                              int maxContentSize)

Deprecated.

getMaxContentSize

public int getMaxContentSize(String mimeType)

Deprecated.

getValidMimeTypes

public Set<String> getValidMimeTypes()

Deprecated.

setValidMimeTypes

public void setValidMimeTypes(Set<String> validMimeTypes)

Deprecated.

addValidMimeTypes

public void addValidMimeTypes(Set<String> validMimeTypes)

Deprecated.

addValidMimeType

public void addValidMimeType(String validMimeType)

Deprecated.

get

public FetchedResult get(String url)
                  throws BaseFetchException

Deprecated.

Throws:

BaseFetchException

getMimeTypeFromContentType

protected static String getMimeTypeFromContentType(String contentType)

Deprecated.

get

public abstract FetchedResult get(String url,
                                  Payload payload)
                           throws BaseFetchException

Deprecated.

Get the content stored in the resource referenced by the 'url' parameter.

Parameters:

url - a string url for which to get content

payload - a populated Payload

Returns:

the FetchedResult associated with the URL

Throws:

BaseFetchException - if an error results from fetching the url.

abort

public abstract void abort()

Deprecated.

Terminate any async request being processed.