System Design Web Crawler: Interview Talk Track

System Design Web Crawler: Interview Talk Track

"Design a web crawler" is a classic system design prompt because it touches almost everything: queues, distributed workers, deduplication, storage, rate limiting, and failure handling. The trap is diving into components before you know what the crawler is for. This system design web crawler interview talk track walks through the conversation in the order interviewers usually expect.

1. Clarify the goal (first 5 minutes)

Ask questions that change the design:

Then summarize: "We're building a polite, distributed crawler for search indexing, HTML only, targeting roughly N pages per day, with periodic re-crawls."

2. Back-of-the-envelope estimates

Walk through the math with round numbers you agree on with the interviewer:

The point isn't precision. It's showing that the numbers drive decisions, such as needing many workers in parallel and compressed blob storage instead of a single database.

3. High-level architecture

Describe the main loop first, then the components:

  1. Seed URLs go into the URL frontier.
  2. Fetcher workers pull URLs from the frontier.
  3. A DNS resolver (with caching) resolves hosts.
  4. The fetcher downloads the page, respecting robots.txt and rate limits.
  5. A content parser extracts text and links.
  6. A content dedup check skips pages already seen.
  7. A URL filter and dedup step normalizes links and drops seen or disallowed ones.
  8. New URLs go back into the frontier; content goes to storage for indexing.

Draw this as a loop. Interviewers like seeing that the crawler feeds itself.

4. The URL frontier: the heart of the design

The frontier isn't just a queue. It balances two goals:

A common design has two layers:

Explaining this two-layer idea clearly is often the strongest moment in the interview.

5. Politeness and robots.txt

Say it plainly: a crawler that ignores politeness gets blocked and is irresponsible.

6. Deduplication

There are two kinds:

7. Storage

8. Scaling and distribution

9. Failure handling and traps

10. Re-crawl strategy

Pages change at different rates. Track how often a page's content hash changes, and schedule re-crawls more often for pages that change frequently and less often for static ones. Mention conditional requests (such as If-Modified-Since) to save bandwidth.

How to pace the conversation

MinutesFocus
0–5Requirements and scope
5–10Estimates
10–20High-level loop and components
20–35Deep dive: frontier, politeness, dedup
35–45Scaling, failures, trade-offs

Let the interviewer pick the deep dive if they show interest in one area.

Practice the talk track out loud

System design is a spoken skill. Knowing the components isn't the same as presenting them clearly in 45 minutes. If you want a structured way to rehearse that delivery before your interview, Luna Interview is built for practice and calm preparation—not for use during a live interview.

Checklist