Product
RobotFileParser
A class in Python's standard library urllib.robotparser used to parse and evaluate robots.txt rules. It implements the looser early semantics and does not fully conform to RFC 9309.
Timeline (last 90 days)
Oct 13
Python's standard library RobotFileParser does not implement all the details of RFC 9309, such as the longest-match-wins rule between Allow and Disallow.
Unverified50%