Paper
RFC 9309
The formal standard for the robots.txt protocol published by the IETF in 2022. It specifies details including user agent matching, longest-match precedence, and handling of Allow/Disallow conflicts.
Timeline (last 90 days)
Oct 13
RFC 9309 was published by the IETF in 2022 and is the first formal standard for the robots.txt protocol.
Unverified50%
Oct 13
Python's standard library RobotFileParser does not implement all the details of RFC 9309, such as the longest-match-wins rule between Allow and Disallow.
Unverified50%