-
Methods Method Description crawlercommons.robots.BaseRobotsParser.parseContent(String, byte[], String, String) since 1.4 - replaced byBaseRobotsParser.parseContent(java.lang.String,byte[],java.lang.String,java.util.Collection<java.lang.String>)
. Passing a collection of robot names gives users more control how user-agent and robot names are matched. Passing a list of names is also more efficient as it does not require to split the robot name string again and again on every robots.txt file to be parsed.crawlercommons.robots.SimpleRobotRulesParser.parseContent(String, byte[], String, String)