SiteMapParser (Crawler-commons 0.7 API)

java.lang.Object
- crawlercommons.sitemaps.SiteMapParser

public class SiteMapParser
extends Object

Field Summary

Fields
Modifier and Type	Field and Description
`static org.slf4j.Logger`	`LOG`
`static int`	`MAX_BYTES_ALLOWED` Sitemap docs must be limited to 10MB (10,485,760 bytes)
`protected boolean`	`strict` True (by default) meaning that invalid URLs should be rejected, as the official docs allow the siteMapURLs to be only under the base url: http://www.sitemaps.org/protocol.html#location

Constructor Summary

Constructors
Constructor and Description

SiteMapParser()

SiteMapParser(boolean strict)

Constructors
Constructor and Description
`SiteMapParser()`
`SiteMapParser(boolean strict)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`addUrlIntoSitemap(String urlStr, SiteMap siteMap, String lastMod, String changeFreq, String priority, int urlIndex)` Adds the given URL to the given sitemap while showing the relevant logs
`protected String`	`getElementAttributeValue(Element elem, String elementName, String attributeName)` Get the element's attribute value.
`protected String`	`getElementValue(Element elem, String elementName)` Get the element's textual content.
`boolean`	`isStrict()`
`protected void`	`parseAtom(SiteMap sitemap, Element elem, Document doc)` Parse the XML document which is assumed to be in Atom format.
`protected void`	`parseRSS(SiteMap sitemap, Document doc)` Parse XML document which is assumed to be in RSS format.
`AbstractSiteMap`	`parseSiteMap(byte[] content, URL url)` Parse a sitemap, given the content bytes and the URL.
`AbstractSiteMap`	`parseSiteMap(String contentType, byte[] content, AbstractSiteMap sitemap)` Returns a processed copy of an unprocessed sitemap object, i.e.
`AbstractSiteMap`	`parseSiteMap(String contentType, byte[] content, URL url)` Parse a sitemap, given the MIME type, the content bytes, and the URL.
`AbstractSiteMap`	`parseSiteMap(URL onlineSitemapUrl)` Returns a SiteMap or SiteMapIndex given an online sitemap URL
`protected SiteMapIndex`	`parseSitemapIndex(URL url, NodeList nodeList)` Parse XML that contains a Sitemap Index.
`protected SiteMap`	`parseSyndicationFormat(URL sitemapUrl, Document doc)` Parse the XML document, looking for a feed element to determine if it's an Atom doc rss to determine if it's an RSS doc.
`protected SiteMap`	`parseXmlSitemap(URL sitemapUrl, Document doc)` Parse XML that contains a valid Sitemap.
`protected AbstractSiteMap`	`processGzip(URL url, byte[] response)` Decompress the gzipped content and process the resulting XML Sitemap.
`protected SiteMap`	`processText(String sitemapUrl, byte[] content)` Process a text-based Sitemap.
`protected AbstractSiteMap`	`processXml(URL sitemapUrl, byte[] xmlContent)` Parse the given XML content.
`protected AbstractSiteMap`	`processXml(URL sitemapUrl, InputSource is)` Parse the given XML content.
`protected boolean`	`urlIsValid(String sitemapBaseUrl, String testUrl)` See if testUrl is under sitemapBaseUrl.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail
- LOG
```
public static final org.slf4j.Logger LOG
```
- MAX_BYTES_ALLOWED
```
public static final int MAX_BYTES_ALLOWED
```
  Sitemap docs must be limited to 10MB (10,485,760 bytes)
  
  See Also:
  
  Constant Field Values
- strict
```
protected boolean strict
```
  True (by default) meaning that invalid URLs should be rejected, as the official docs allow the siteMapURLs to be only under the base url: http://www.sitemaps.org/protocol.html#location

Constructor Detail

SiteMapParser
```
public SiteMapParser()
```

SiteMapParser
```
public SiteMapParser(boolean strict)
```

Method Detail

isStrict
```
public boolean isStrict()
```
Returns:

whether invalid URLs will be rejected (where invalid means that the url is not under the base url)

parseSiteMap
```
public AbstractSiteMap parseSiteMap(URL onlineSitemapUrl)
                             throws UnknownFormatException,
                                    IOException
```
Returns a SiteMap or SiteMapIndex given an online sitemap URL

Please note that this method is a static method which goes online and fetches the sitemap then parses it
This method is a convenience method for a user who has a sitemap URL and wants a "Keep it simple" way to parse it.

Parameters:

onlineSitemapUrl - URL of the online sitemap

Returns:

Extracted SiteMap/SiteMapIndex or null if the onlineSitemapUrl is null

Throws:

UnknownFormatException - if there is an error parsing the sitemap

IOException - if there is an error reading in the site map URL

parseSiteMap
```
public AbstractSiteMap parseSiteMap(String contentType,
                                    byte[] content,
                                    AbstractSiteMap sitemap)
                             throws UnknownFormatException,
                                    IOException
```
Returns a processed copy of an unprocessed sitemap object, i.e. transfer the value of getLastModified(). Please note that the sitemap input stays unchanged. Note that contentType is assumed to be correct; in general it is more robust to use the method that doesn't take a contentType, but instead detects this using Tika.

Parameters:

contentType - MIME type of content

content - raw bytes of sitemap file

sitemap - an AbstractSiteMap implementation

Returns:

Extracted SiteMap/SiteMapIndex

Throws:

UnknownFormatException - if there is an error parsing the sitemap

IOException - if there is an error reading in the site map URL

parseSiteMap
```
public AbstractSiteMap parseSiteMap(byte[] content,
                                    URL url)
                             throws UnknownFormatException,
                                    IOException
```
Parse a sitemap, given the content bytes and the URL.

Parameters:

content - raw bytes of sitemap file

url - URL to sitemap file

Returns:

Extracted SiteMap/SiteMapIndex

Throws:

UnknownFormatException - if there is an error parsing the sitemap

IOException - if there is an error reading in the site map URL

parseSiteMap
```
public AbstractSiteMap parseSiteMap(String contentType,
                                    byte[] content,
                                    URL url)
                             throws UnknownFormatException,
                                    IOException
```
Parse a sitemap, given the MIME type, the content bytes, and the URL. Note that contentType is assumed to be correct; in general it is more robust to use the method that doesn't take a contentType, but instead detects this using Tika.

Parameters:

contentType - MIME type of content

content - raw bytes of sitemap file

url - URL to sitemap file

Returns:

Extracted SiteMap/SiteMapIndex

Throws:

UnknownFormatException - if there is an error parsing the sitemap

IOException - if there is an error reading in the site map URL

processXml
```
protected AbstractSiteMap processXml(URL sitemapUrl,
                                     byte[] xmlContent)
                              throws UnknownFormatException
```
Parse the given XML content.

Parameters:

sitemapUrl - URL to sitemap file

xmlContent - the byte[] backing the sitemapUrl

Returns:

The site map

Throws:

UnknownFormatException - if there is an error parsing the sitemap

processText
```
protected SiteMap processText(String sitemapUrl,
                              byte[] content)
                       throws IOException
```
Process a text-based Sitemap. Text sitemaps only list URLs but no priorities, last mods, etc.

Parameters:

sitemapUrl - a string sitemap URL

sitemapUrl - URL to sitemap file

content - the byte[] backing the sitemapUrl

Returns:

The site map

Throws:

IOException - if there is an error reading in the site map String

processGzip
```
protected AbstractSiteMap processGzip(URL url,
                                      byte[] response)
                               throws IOException,
                                      UnknownFormatException
```
Decompress the gzipped content and process the resulting XML Sitemap.

Parameters:

url - - URL of the gzipped content

response - - Gzipped content

Returns:

the site map

Throws:

UnknownFormatException - if there is an error parsing the gzip

IOException - if there is an error reading in the gzip URL

processXml

protected AbstractSiteMap processXml(URL sitemapUrl,
                                     InputSource is)
                              throws UnknownFormatException

Parse the given XML content.

Parameters:: sitemapUrl - a sitemap URL; is - an InputSource backing the sitemap
Returns:: the site map
Throws:: UnknownFormatException - if there is an error parsing the InputSource

parseXmlSitemap

protected SiteMap parseXmlSitemap(URL sitemapUrl,
                                  Document doc)

Parse XML that contains a valid Sitemap. Example of a Sitemap:

  
 <?xml version="1.0" encoding="UTF-8"?> 
   <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> 
     <url>
       <loc>http://www.example.com/</loc> 
       <lastmod>lastmod>2005-01-01</lastmod>
       <changefreq>monthly</changefreq>
       <priority>0.8</priority>
     </url> 
     <url>
       <loc>http://www.example.com/catalog?item=12&amp;desc=vacation_hawaii</loc>
       <changefreq>weekly</changefreq>
     </url>
   </urlset>

Parameters:: sitemapUrl - a sitemap URL; doc - a Document sitemap snippet
Returns:: The sitemap

parseSitemapIndex

protected SiteMapIndex parseSitemapIndex(URL url,
                                         NodeList nodeList)

Parse XML that contains a Sitemap Index. Example Sitemap Index:

 
 <?xml version="1.0" encoding="UTF-8"?> 
   <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
     <sitemap>
       <loc>http://www.example.com/sitemap1.xml.gz</loc>
       <lastmod>2004-10-01T18:23:17+00:00</lastmod>
     </sitemap>
     <sitemap>
       <loc>http://www.example.com/sitemap2.xml.gz</loc>
       <lastmod>2005-01-01</lastmod>
     </sitemap>
   </sitemapindex>

Parameters:: url - - URL of Sitemap Index; nodeList - a NodeList backing the sitemap
Returns:: The site map index

parseSyndicationFormat
```
protected SiteMap parseSyndicationFormat(URL sitemapUrl,
                                         Document doc)
                                  throws UnknownFormatException
```
Parse the XML document, looking for a feed element to determine if it's an Atom doc rss to determine if it's an RSS doc.

Parameters:

sitemapUrl - the URL location of the Sitemap

doc - - XML document to parse

Returns:

The sitemap

Throws:

UnknownFormatException - if XML does not appear to be Atom or RSS

parseAtom

protected void parseAtom(SiteMap sitemap,
                         Element elem,
                         Document doc)

Parse the XML document which is assumed to be in Atom format. Atom 1.0 example:

 
 <?xml version="1.0" encoding="utf-8"?>
   <feed xmlns="http://www.w3.org/2005/Atom">
     <title>Example Feed</title>
     <subtitle>A subtitle.</subtitle>
     <link href="http://example.org/feed/" rel="self"/>
     <link href="http://example.org/"/>
     <modified>2003-12-13T18:30:02Z</modified>
     <author>
       <name>John Doe</name>
       <email>johndoe@example.com</email>
     </author>
     <id>urn:uuid:60a76c80-d399-11d9-b91C-0003939e0af6</id>
     <entry>
       <title>Atom-Powered Robots Run Amok</title>
       <link href="http://example.org/2003/12/13/atom03"/>
       <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
       <updated>2003-12-13T18:30:02Z</updated>
       <summary>Some text.</summary>
     </entry>
     ...
   </feed>

Parameters:: sitemap - a SiteMap backing the Atom feed; elem - Element's to populate from the Sitemap; doc - Document to populate with the parse output

parseRSS

protected void parseRSS(SiteMap sitemap,
                        Document doc)

Parse XML document which is assumed to be in RSS format. RSS 2.0 example:

 
 <?xml version="1.0"?>
   <rss version="2.0">
     <channel> 
       <title>Lift Off News</title>
       <link>http://liftoff.msfc.nasa.gov/</link>
       <description>Liftoff to Space Exploration.</description>
       <language>en-us</language>
       <pubDate>Tue, 10 Jun 2003 04:00:00 GMT</pubDate>
       <lastBuildDate>Tue, 10 Jun 2003 09:41:01 GMT</lastBuildDate>
       <docs>http://blogs.law.harvard.edu/tech/rss</docs>
       <generator>Weblog Editor 2.0</generator>
       <managingEditor>editor@example.com</managingEditor>
       <webMaster>webmaster@example.com</webMaster>
       <ttl>5</ttl>
       <item>
         <title>Star City</title>
         <link>http://liftoff.msfc.nasa.gov/news/2003/news-starcity.asp</link>
         <description>How do Americans get ready to work with Russians aboard the
         International Space Station? They take a crash course in culture,
         language and protocol at Russia's Star City.
         </description>
         <pubDate>Tue, 03 Jun 2003 09:39:21 GMT</pubDate>
         <guid>http://liftoff.msfc.nasa.gov/2003/06/03.html#item573</guid>
       </item>
       <item>
         <title>Space Exploration</title>
         <link>http://liftoff.msfc.nasa.gov/</link>
         <description>Sky watchers in Europe, Asia, and parts of Alaska and Canada 
         will experience a partial eclipse of the Sun on Saturday, May 31.
         </description>
         <pubDate>Fri, 30 May 2003 11:06:42 GMT</pubDate>
         <guid>http://liftoff.msfc.nasa.gov/2003/05/30.html#item572</guid>
       </item>
     </channel>
   </rss>

Parameters:: sitemap - a SiteMap object to populate with the RCC content; doc - Document to populate with the parse output

getElementValue

protected String getElementValue(Element elem,
                                 String elementName)

Get the element's textual content.

Parameters:: elem -; elementName -
Returns:: The element value

getElementAttributeValue

protected String getElementAttributeValue(Element elem,
                                          String elementName,
                                          String attributeName)

Get the element's attribute value.

Parameters:: elem -; elementName -; attributeName -
Returns:: The element attribute value

addUrlIntoSitemap

protected void addUrlIntoSitemap(String urlStr,
                                 SiteMap siteMap,
                                 String lastMod,
                                 String changeFreq,
                                 String priority,
                                 int urlIndex)

Adds the given URL to the given sitemap while showing the relevant logs

Parameters:: urlStr - an URL string to add to the SiteMap; siteMap - the sitemap to add URL(s) to; lastMod - last time the SiteMapURL was modified; changeFreq - the SiteMapURL change frquency; priority - priority of this SiteMapURL; urlIndex - index position to which this entry has been added

urlIsValid
```
protected boolean urlIsValid(String sitemapBaseUrl,
                             String testUrl)
```
See if testUrl is under sitemapBaseUrl. Only URLs under sitemapBaseUrl are valid.

Parameters:

sitemapBaseUrl -

testUrl -

Returns:

true if testUrl is under sitemapBaseUrl, false otherwise

Class SiteMapParser

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG

MAX_BYTES_ALLOWED

strict

Constructor Detail

SiteMapParser

SiteMapParser

Method Detail

isStrict

parseSiteMap

parseSiteMap

parseSiteMap

parseSiteMap

processXml

processText

processGzip

processXml

parseXmlSitemap

parseSitemapIndex

parseSyndicationFormat

parseAtom

parseRSS

getElementValue

getElementAttributeValue

addUrlIntoSitemap

urlIsValid