Design a Web Crawler
A web crawler (robot, spider) discovers new or updated content on the web: pages, images, videos, PDFs. It starts from a few pages, follows their links, and keeps going.
- 9.1Uses, requirements and estimationCrawlers are used for: Search engine indexing: the most common use. Googlebot powers Google Search. Web archiving: national libraries (US…
- 9.2High-level design: the crawler workflowComponent Job Seed URLs Starting points. A good strategy: divide the URL space by locality (countries) or topic (shopping, sports,…
- 9.3DFS vs BFSThe web is a directed graph: pages are nodes, hyperlinks are edges. Crawling is graph traversal. DFS is usually a bad choice because the…
- 9.4The URL frontier: politeness, priority, freshnessPoliteness Download pages from the same host one at a time, with a delay between them. Map each host to its own download queue: Queue…
- 9.5HTML downloader: robots.txt and performanceRobots.txt The Robots Exclusion Protocol is how websites tell crawlers which pages they may download. Check robots.txt before crawling a…
- 9.6Robustness, extensibility and problematic contentRobustness Consistent hashing spreads load among downloaders, which can be added or removed easily. Save crawl states and data, so a failed…
- 9.7Wrap upA good crawler is scalable, polite, extensible and robust. Extra talking points: Server-side rendering: many sites generate links with…