Big data, in near-real time 2

When events arrive faster than a person could ever read them, batch jobs that run "once a night" stop being enough. Streaming pipelines process data as it happens — but volume is the easy part. The hard part is staying correct when events arrive late, twice, or out of order.

Scale is not the hard part

Partitioned logs like Kafka and elastic compute make raw throughput a solved problem. What earns its keep is the processing model: exactly-once semantics, windowing on event time rather than arrival time, and watermarks that decide when a window is really closed.

Correct under pressure

We design every stage to be idempotent and keyed, so a replay never double-counts. State lives in the stream processor with checkpointing, so a crashed worker resumes exactly where it left off instead of guessing.

-- Aggregate on event time, in 5-minute tumbling windows.
SELECT
  window_start,
  product_id,
  COUNT(*) AS views
FROM TABLE(
  TUMBLE(TABLE events, DESCRIPTOR(event_time), INTERVAL '5' MINUTES)
)
GROUP BY window_start, product_id;

A fast pipeline that double-counts is just a quicker way to be wrong.

What we watch

• Consumer lag — are we keeping up with the firehose?
• Late and dropped events, surfaced rather than silently lost.
• A dead-letter queue for anything that can't be parsed.

Done well, a streaming pipeline feels invisible: dashboards that are simply current, alerts that fire while it still matters, and decisions made on what is happening — not on what happened yesterday.