System Design Metrics Monitoring and Alerting: Interview Talk Track
"Design a metrics monitoring and alerting system" is a common system design prompt, especially for infrastructure and platform roles. It tests whether you understand high write volume, time-series data, aggregation, retention, and how to turn raw numbers into alerts that wake the right person at the right time.
This system design metrics monitoring interview talk track gives you a clear structure for a 45-minute round. Adapt the depth to your level and the interviewer's questions.
1. Clarify requirements (about 5 minutes)
Functional requirements:
- Collect metrics such as CPU, memory, request rate, error rate, and latency from many services and hosts.
- Store metrics as time series with labels (service, host, region).
- Query and graph metrics on dashboards.
- Define alert rules ("error rate above 5% for 5 minutes") and send notifications by email, chat, or paging.
Non-functional requirements:
- Scale: assume 100,000 hosts, each emitting a few hundred metrics every 10 seconds. That's millions of data points per second.
- Freshness: dashboards and alerts within about 30 seconds of real time.
- Retention: high-resolution data for a couple of weeks, downsampled data for a year.
- Availability: the monitoring system must stay up when the systems it watches are failing.
Do a quick estimate out loud: "100,000 hosts × 300 metrics ÷ 10 seconds is about 3 million points per second. At roughly 16 bytes per point before compression, that's around 50 MB per second of raw data, so compression and downsampling matter."
2. High-level design (about 10 minutes)
Components:
- Agents on each host collect metrics and send them in batches.
- Ingestion service receives batches, validates them, and writes to a queue.
- Message queue (a log-based queue) buffers spikes and decouples ingestion from storage.
- Storage writers consume from the queue and write to a time-series database.
- Time-series database optimized for append-heavy writes and range queries.
- Query service serves dashboards and alert evaluation.
- Alert manager evaluates rules, deduplicates alerts, and routes notifications.
- Downsampling jobs produce lower-resolution rollups for long-term storage.
Mention the push vs pull choice: "Agents can push to the collectors, or a scraper can pull from service endpoints. Pull makes it easy to detect dead targets; push works better for short-lived jobs and across network boundaries. I'll go with push through agents and note that pull works too."
3. Data model (about 5 minutes)
A time series is identified by a metric name plus labels:
http_requests_total{service="checkout", region="eu", status="500"}
Each series holds (timestamp, value) points.
Key ideas:
- Series ID: hash of name and sorted labels.
- Inverted index from label values to series IDs, so queries like "all series where service=checkout" are fast.
- Cardinality warning: labels with unbounded values (user IDs, request IDs) explode the number of series. Say you'd enforce limits and reject high-cardinality labels.
4. Storage (about 8 minutes)
Time-series data has a special shape: writes are almost always appends at the latest timestamp, and reads are usually recent ranges.
- In-memory head block: recent points are buffered in memory and in a write-ahead log for durability.
- Immutable blocks: every couple of hours, the head is compressed into a block on disk, partitioned by time.
- Compression: delta-of-delta encoding for timestamps and XOR encoding for values reduce each point to around one or two bytes on average for regular data.
- Sharding: partition by series ID hash across storage nodes, with replication (for example, factor 3) for durability.
- Retention: drop whole time blocks when they expire. That's far cheaper than deleting individual points.
Downsampling: a background job rolls 10-second data into 1-minute and 1-hour aggregates (min, max, sum, count). Long-range queries read rollups instead of raw points.
5. Query path (about 5 minutes)
- The query service parses an expression like "rate of 500 errors per service over 5 minutes."
- It uses the inverted index to find matching series, fetches the needed time range from the right shards, and aggregates.
- Recent queries hit the in-memory head; older ones read compressed blocks or rollups.
- Cache popular dashboard queries for a short time, since many people often view the same dashboard.
6. Alerting (about 7 minutes)
- Rule evaluation: the alert manager evaluates each rule on a schedule, for example every 30 seconds, using the query service.
- Pending state: a condition must hold for a duration ("for 5 minutes") before firing, which avoids alerts on brief spikes.
- Deduplication and grouping: if 200 hosts fail at once, group them into one notification instead of 200 pages.
- Routing: send to the right team based on labels, with escalation if nobody acknowledges.
- Silences: allow muting during planned maintenance.
- Resolve notifications when the condition clears.
Mention reliability: "The alerting path should be independent enough that a storage slowdown doesn't silently stop alerts. I'd also have a heartbeat alert that fires if the alerting system itself stops evaluating rules."
7. Scaling and failure handling (about 5 minutes)
- The queue absorbs ingestion spikes; writers scale horizontally by partition.
- If a storage node fails, replicas serve reads and the shard is rebuilt.
- Agents buffer locally for a short time if ingestion is unavailable.
- Consider deploying monitoring per region, with a global view that federates queries, so a regional outage doesn't take down monitoring everywhere.
- Watch the monitoring system with a separate, smaller monitoring setup.
8. Wrap up
Summarize: "Agents push batches through a queue into a sharded time-series store with compressed time blocks and downsampled rollups. A query service powers dashboards, and an alert manager evaluates rules with pending periods, grouping, routing, and silences." Then offer deeper topics: cardinality control, logs and traces, or anomaly detection.
Common mistakes
- Using a general relational table with one row per point and ignoring write volume.
- Forgetting cardinality limits.
- Treating alerting as an afterthought instead of a core reliability path.
- Not planning for the monitoring system to stay up when everything else is down.
Practice the talk track
Infrastructure design questions reward clear structure and confident estimates. Luna Interview helps you rehearse system design walkthroughs out loud with a timer, so the numbers and trade-offs come out calmly in the real round.
Run through this structure a few times, including the quick estimate, and you'll be ready for the follow-up questions.
Rehearse with an AI interview copilot, then browse commonly reported interview questions.