HTML downloader: robots.txt and performance
Play this lessonThis lesson comes with an interactive animation you can break, a live-code challenge and a chapter quiz.The Robots Exclusion Protocol is how websites tell crawlers which pages they may download. Check robots.txt before crawling a site, and cache it (re-download it periodically). Here's what one might look like for an online shop:
The rest of this lesson is part of Pro
Pro unlocks every lesson with its animation, code challenge and quiz, plus every interview problem. ₹299 a month or ₹1,999 a year.
See Pro