| Package | Description |
|---|---|
| crawlercommons.robots |
The robots package contains all of the robots.txt rule inference, parsing and utilities contained within Crawler Commons.
|
| Modifier and Type | Field and Description |
|---|---|
protected ArrayList<SimpleRobotRules.RobotRule> |
SimpleRobotRules._rules |
| Modifier and Type | Method and Description |
|---|---|
List<SimpleRobotRules.RobotRule> |
SimpleRobotRules.getRobotRules() |
| Modifier and Type | Method and Description |
|---|---|
int |
SimpleRobotRules.RobotRule.compareTo(SimpleRobotRules.RobotRule o) |
Copyright © 2009–2021 Crawler-Commons. All rights reserved.