System Design Web Crawler: Interview Talk Track
"Design a web crawler" is a classic system design prompt because it touches almost everything: queues, distributed workers, deduplication, storage, rate limiting, and failure handling. The trap is diving into components before you know what the crawler is for. This system design web crawler interview talk track walks through the conversation in the order interviewers usually expect.
1. Clarify the goal (first 5 minutes)
Ask questions that change the design:
- What's the purpose? Search indexing, price monitoring, archiving, or training data? It affects freshness and what you store.
- Scale? Thousands of sites or a large slice of the web? Ask for a target pages-per-day number, or propose one and confirm it.
- Content types? HTML only, or also images and PDFs?
- Freshness? Crawl once, or re-crawl popular pages frequently?
- Politeness requirements? Respect
robots.txtand per-site rate limits (always yes, but say it).
Then summarize: "We're building a polite, distributed crawler for search indexing, HTML only, targeting roughly N pages per day, with periodic re-crawls."
2. Back-of-the-envelope estimates
Walk through the math with round numbers you agree on with the interviewer:
- Pages per day divided by 86,400 seconds gives pages per second.
- Multiply by an assumed average page size to get bandwidth and daily storage.
- Multiply storage by retention to get the total.
The point isn't precision. It's showing that the numbers drive decisions, such as needing many workers in parallel and compressed blob storage instead of a single database.
3. High-level architecture
Describe the main loop first, then the components:
- Seed URLs go into the URL frontier.
- Fetcher workers pull URLs from the frontier.
- A DNS resolver (with caching) resolves hosts.
- The fetcher downloads the page, respecting robots.txt and rate limits.
- A content parser extracts text and links.
- A content dedup check skips pages already seen.
- A URL filter and dedup step normalizes links and drops seen or disallowed ones.
- New URLs go back into the frontier; content goes to storage for indexing.
Draw this as a loop. Interviewers like seeing that the crawler feeds itself.
4. The URL frontier: the heart of the design
The frontier isn't just a queue. It balances two goals:
- Priority: important or frequently changing pages should be crawled sooner.
- Politeness: never hammer a single host.
A common design has two layers:
- Front queues by priority. A prioritizer assigns each URL a score (for example, based on page importance or change frequency) and places it in one of several priority queues.
- Back queues by host. Each back queue holds URLs for one host, and a scheduler only releases the next URL from a host after its delay has passed.
Explaining this two-layer idea clearly is often the strongest moment in the interview.
5. Politeness and robots.txt
- Fetch and cache
robots.txtper host, and refresh it periodically. - Enforce a minimum delay between requests to the same host.
- Identify the crawler with a clear user agent.
- Back off on errors or slow responses.
Say it plainly: a crawler that ignores politeness gets blocked and is irresponsible.
6. Deduplication
There are two kinds:
- URL dedup: normalize URLs (lowercase host, strip fragments, sort query parameters where safe) and check against a "seen" set. At scale, a Bloom filter saves memory at the cost of rare false positives, which only means occasionally skipping a page.
- Content dedup: hash page content to catch exact duplicates. For near-duplicates, mention techniques such as SimHash at a high level.
7. Storage
- Raw pages: compressed blobs in object storage, keyed by URL hash and crawl time.
- Metadata: URL, last crawl time, status code, content hash, next scheduled crawl, in a scalable key-value or wide-column store.
- Extracted links and text: passed downstream to the indexing pipeline, often through a message queue.
8. Scaling and distribution
- Partition by host. Assign each host to one worker group (for example, by consistent hashing of the hostname). This keeps politeness state local and avoids two workers hitting the same site.
- Horizontal workers. Add fetchers to increase throughput; the frontier controls the pace.
- Distributed frontier. Persist queues so a crash doesn't lose progress.
- Geographic placement. Workers closer to target servers can cut latency.
9. Failure handling and traps
- Retries with exponential backoff for transient errors; give up after a limit.
- Timeouts on slow servers so workers don't hang.
- Spider traps: infinitely generated URLs (calendars, session IDs). Cap URL length and depth per site, and detect repetitive patterns.
- Checkpointing the frontier so restarts resume cleanly.
10. Re-crawl strategy
Pages change at different rates. Track how often a page's content hash changes, and schedule re-crawls more often for pages that change frequently and less often for static ones. Mention conditional requests (such as If-Modified-Since) to save bandwidth.
How to pace the conversation
| Minutes | Focus |
|---|---|
| 0–5 | Requirements and scope |
| 5–10 | Estimates |
| 10–20 | High-level loop and components |
| 20–35 | Deep dive: frontier, politeness, dedup |
| 35–45 | Scaling, failures, trade-offs |
Let the interviewer pick the deep dive if they show interest in one area.
Practice the talk track out loud
System design is a spoken skill. Knowing the components isn't the same as presenting them clearly in 45 minutes. If you want a structured way to rehearse that delivery before your interview, Luna Interview is built for practice and calm preparation—not for use during a live interview.
Checklist
- [ ] Clarified purpose, scale, and freshness
- [ ] Drew the crawl loop end to end
- [ ] Explained the two-layer frontier (priority plus politeness)
- [ ] Covered robots.txt and per-host rate limits
- [ ] Explained URL and content dedup, including Bloom filters
- [ ] Discussed partitioning by host and handling spider traps
Rehearse with an AI interview copilot, then browse commonly reported interview questions.