SYSTEM DESIGN · QUESTION 119
How would you design a centralized logging and metrics platform?
On this page
I would clarify ingestion volume, retention, query patterns, alert latency, and tenant boundaries. Agents or SDKs batch and buffer telemetry, send it through an authenticated ingestion tier, and partition it into durable queues and storage optimized for logs or time series. Add sampling, cardinality controls, retention tiers, and backpressure. Dashboards and alerts need clear freshness and data-loss indicators.
LET'S MAKE IT SIMPLE
First, the idea in plain English
Many services produce logs and measurements, but engineers need one place to search and detect problems. The platform must accept bursts without blocking applications, store data efficiently, and make recent signals queryable. High-cardinality labels and excessive logs can become expensive quickly.
A USEFUL COMPARISON
Picture it this way
A city collects traffic reports from many intersections, stores them by time and location, and alerts operators when patterns look unusual. If sensors flood the center with duplicate reports, the system needs buffering and filtering instead of slowing every road.
FOLLOW THE JOURNEY
Buffer telemetry ingestion and store by query and retention needs
Trace collection, buffering, indexing, querying, and alerting with backpressure.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Operators need to know whether a missing signal means healthy or lost data.
Service agents batch telemetry into authenticated ingestion, durable queues buffer bursts, specialized stores index logs and metrics, and query/alert services monitor freshness and loss.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Operators need to know whether a missing signal means healthy or lost data.
UNDER THE HOOD
How it works, step by step
Collect with bounded buffering
Batch events locally and apply sampling or backpressure.
Why it matters: Telemetry should not take down the monitored service.
Ingest durably
Authenticate producers and queue data for downstream processing.
Why it matters: Buffers absorb spikes and isolate storage failures.
Index and retain
Partition by time and tenant, apply retention and cardinality controls.
Why it matters: Storage and query costs grow with data volume.
Query and alert
Serve searches and time-series alerts with freshness indicators.
Why it matters: Operators need to know whether a missing signal means healthy or lost data.
OUTSIDE THE TEXTBOOK
A practical example
A fleet of services emits request errors and latency histograms. Agents batch records and drop or sample low-priority data under pressure; a durable ingest tier stores high-priority events. Metrics use bounded labels, while searchable logs have tiered retention. Dashboards show ingestion lag so operators can distinguish no errors from telemetry loss.
GO A LITTLE DEEPER
Technical details that matter
- Metrics with unbounded labels create high-cardinality time series and storage pressure.
- Sampling policies should preserve rare errors and trace relationships where possible.
- Multi-tenant systems need isolation, quotas, and access controls over logs that may contain sensitive data.
MAKE A GOOD CHOICE
When is this useful?
- Production debugging and incident response.
- Service-level monitoring and alerting.
- Centralizing telemetry from distributed workloads.
Managed observability services can reduce operational burden, though retention, query cost, and data residency still need review.
WATCH OUT FOR
Common misunderstandings
- Allowing unbounded metric labels such as raw user IDs.
- Blocking application requests on telemetry backends.
- Interpreting missing telemetry as proof that a service is healthy.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- High-volume telemetry can be expensive.
- Sampling loses some detail.
- Logs may contain sensitive information and require redaction and retention controls.
WHAT TO REMEMBER
Key takeaways
- 1Buffer and isolate telemetry ingestion from application traffic.
- 2Control cardinality, sampling, retention, and tenant access.
- 3Expose freshness and loss so operators can trust the signals.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What is metric cardinality?
It is the number of distinct time series produced by label combinations. Unbounded labels can grow storage and query cost rapidly.
How do you prevent telemetry from hurting the application?
Use asynchronous bounded buffers, sampling, timeouts, and drop policies that prioritize critical signals under pressure.
SAY IT OUT LOUD
How to explain this in an interview
“I would buffer telemetry asynchronously, ingest it through an authenticated durable tier, and store logs and metrics according to their query patterns and retention. I would control cardinality and sampling, isolate tenants, and ensure the telemetry path cannot block the application. Dashboards should expose ingestion lag and data loss so missing data is not mistaken for health.”
Connect this question to the roadmap topic it supports.