SYSTEM DESIGN · QUESTION 115
How would you design a real-time event analytics platform?
On this page
I would clarify event volume, freshness, query patterns, retention, and correctness needs. Producers send validated events to a durable log; stream processors enrich and aggregate them into a serving store, while raw events are archived for replay. Use schema contracts, partitioning, consumer checkpoints, idempotent writes, and event-time handling. Dashboards should expose freshness and late-data behavior.
LET'S MAKE IT SIMPLE
First, the idea in plain English
Product teams want to see activity soon after it happens, such as signups or purchases. Events need to be accepted reliably, processed into useful summaries, and stored for queries. Some events arrive late or more than once, so the platform needs rules for correction and replay.
A USEFUL COMPARISON
Picture it this way
A stadium counts attendance as people enter, updates a live display, and keeps ticket records for later audits. The display is fast, while the archive helps correct totals after a scanner reconnects and uploads delayed entries.
FOLLOW THE JOURNEY
Ingest durable events, aggregate streams, and retain replayable history
Trace event producers, log, stream processing, serving store, and raw archive.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Fast aggregates need an auditable source of truth.
Validated producers publish events to a durable log; stream processors aggregate and write idempotently to a query store while raw events are archived for replay.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Fast aggregates need an auditable source of truth.
UNDER THE HOOD
How it works, step by step
Validate and publish
Apply event contracts and send records with stable IDs and timestamps.
Why it matters: Reliable identity and schema support later recovery.
Buffer durably
Store events in a partitioned log with retention and offsets.
Why it matters: A durable buffer absorbs bursts and enables replay.
Process and aggregate
Consume events, handle event time, and write idempotent results.
Why it matters: Processing must account for retries and late arrivals.
Serve and reconcile
Expose low-latency queries and retain raw data for correction.
Why it matters: Fast aggregates need an auditable source of truth.
OUTSIDE THE TEXTBOOK
A practical example
A product dashboard shows active sessions by minute. Clients emit versioned events to a log; stream processors deduplicate by event ID and aggregate by event-time window. Raw events remain available for backfills, and the dashboard displays data freshness so users know a recent interval may still change.
GO A LITTLE DEEPER
Technical details that matter
- Partition keys influence ordering and throughput; hot keys can limit parallelism.
- Watermarks and allowed lateness balance result finalization against delayed events.
- Serving stores should match dashboard query patterns rather than forcing scans of the raw log.
MAKE A GOOD CHOICE
When is this useful?
- Live operational dashboards and product metrics.
- Event monitoring and near-real-time anomaly signals.
- Systems needing both immediate aggregates and replayable history.
Scheduled batch analytics may be simpler when hourly or daily freshness is acceptable.
WATCH OUT FOR
Common misunderstandings
- Assuming events arrive once and in order.
- Discarding raw events so bugs cannot be replayed.
- Showing a live metric without its freshness or correction semantics.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Streaming adds continuous operational cost and complexity.
- Late data can revise previously displayed results.
- High-cardinality dimensions can overwhelm serving stores.
WHAT TO REMEMBER
Key takeaways
- 1Use a durable log and preserve raw events for replay.
- 2Handle duplicates, ordering, and event time explicitly.
- 3Design the serving store and freshness contract around queries.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
Why keep raw events after aggregation?
They support audit, reprocessing, and corrected logic when a processor bug or schema issue is discovered.
How do you make aggregates retry-safe?
Use stable event IDs, checkpoint coordination, and transactional or idempotent updates to the serving state.
SAY IT OUT LOUD
How to explain this in an interview
“I would send validated, versioned events to a durable partitioned log, process them with event-time and deduplication rules, and write aggregates idempotently to a query store. I would archive raw events for replay and expose freshness and late-data behavior to dashboard users. If the business accepts slower updates, batch processing may be simpler.”
Connect this question to the roadmap topic it supports.