System Design Metrics Monitoring and Alerting: Interview Talk Track

System Design Metrics Monitoring and Alerting: Interview Talk Track

"Design a metrics monitoring and alerting system" is a common system design prompt, especially for infrastructure and platform roles. It tests whether you understand high write volume, time-series data, aggregation, retention, and how to turn raw numbers into alerts that wake the right person at the right time.

This system design metrics monitoring interview talk track gives you a clear structure for a 45-minute round. Adapt the depth to your level and the interviewer's questions.

1. Clarify requirements (about 5 minutes)

Functional requirements:

Non-functional requirements:

Do a quick estimate out loud: "100,000 hosts × 300 metrics ÷ 10 seconds is about 3 million points per second. At roughly 16 bytes per point before compression, that's around 50 MB per second of raw data, so compression and downsampling matter."

2. High-level design (about 10 minutes)

Components:

Mention the push vs pull choice: "Agents can push to the collectors, or a scraper can pull from service endpoints. Pull makes it easy to detect dead targets; push works better for short-lived jobs and across network boundaries. I'll go with push through agents and note that pull works too."

3. Data model (about 5 minutes)

A time series is identified by a metric name plus labels:

http_requests_total{service="checkout", region="eu", status="500"}

Each series holds (timestamp, value) points.

Key ideas:

4. Storage (about 8 minutes)

Time-series data has a special shape: writes are almost always appends at the latest timestamp, and reads are usually recent ranges.

Downsampling: a background job rolls 10-second data into 1-minute and 1-hour aggregates (min, max, sum, count). Long-range queries read rollups instead of raw points.

5. Query path (about 5 minutes)

6. Alerting (about 7 minutes)

Mention reliability: "The alerting path should be independent enough that a storage slowdown doesn't silently stop alerts. I'd also have a heartbeat alert that fires if the alerting system itself stops evaluating rules."

7. Scaling and failure handling (about 5 minutes)

8. Wrap up

Summarize: "Agents push batches through a queue into a sharded time-series store with compressed time blocks and downsampled rollups. A query service powers dashboards, and an alert manager evaluates rules with pending periods, grouping, routing, and silences." Then offer deeper topics: cardinality control, logs and traces, or anomaly detection.

Common mistakes

Practice the talk track

Infrastructure design questions reward clear structure and confident estimates. Luna Interview helps you rehearse system design walkthroughs out loud with a timer, so the numbers and trade-offs come out calmly in the real round.

Run through this structure a few times, including the quick estimate, and you'll be ready for the follow-up questions.