crawlercommons.filters.basic

Class BasicURLNormalizer

java.lang.Object

crawlercommons.filters.URLFilter

crawlercommons.filters.basic.BasicURLNormalizer

public class BasicURLNormalizer
extends URLFilter

Code borrowed from Apache Nutch. Converts URLs to a normal form:

• remove dot segments in path: /./ or /../
• remove default ports, e.g. 80 for protocol http://
• normalize percent-encoding in URL paths

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
BasicURLNormalizer()

Method Summary

Modifier and Type	Method and Description
String	filter(String urlString) Returns a modified version of the input URL or null if the URL should be removed
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

BasicURLNormalizer

public BasicURLNormalizer()

Method Detail

filter

public String filter(String urlString)

Description copied from class: URLFilter

Returns a modified version of the input URL or null if the URL should be removed

Specified by:

filter in class URLFilter

Parameters:

urlString - a URL string to check against filter(s)

Returns:

a filtered URL

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException