Protego is a pure-Python robots.txt parser with support for modern conventions.

Homepage:
https://github.com/scrapy/protego
