Package | Description |
---|---|
crawlercommons.robots |
The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.
|
Modifier and Type | Field and Description |
---|---|
protected ArrayList<SimpleRobotRules.RobotRule> |
SimpleRobotRules._rules |
Modifier and Type | Method and Description |
---|---|
List<SimpleRobotRules.RobotRule> |
SimpleRobotRules.getRobotRules() |
Modifier and Type | Method and Description |
---|---|
int |
SimpleRobotRules.RobotRule.compareTo(SimpleRobotRules.RobotRule o) |
Copyright © 2009–2021 Crawler-Commons. All rights reserved.