Every streaming pipeline we build ends up having the same argument, and it is almost never about throughput. It is about one word in a ticket: late. Someone reports that a record arrived late, and the question that actually decides the design usually goes unasked: late relative to what?
An event does not carry lateness. It carries an event time, which is when the thing happened in the business, and it arrives whenever the network, the mobile client, the batch gateway and the retry policy allow. Arrival disorder is the normal condition of a distributed producer fleet, not a fault to be engineered away. What makes a record late is the watermark held by one specific operator at the moment that record reaches it. A watermark is the engine's estimate of how far event time has progressed, which is to say it is a decision to stop waiting so the computation can produce an answer. It is not proof that nothing earlier will ever arrive.
Two things follow from that, and both are easy to miss.
The first is that the same record can be on time and late in the same pipeline. It clears the watermark at an early operator, gets buffered or repartitioned, and then misses the watermark at a downstream window that advanced in the meantime. There is no global verdict to look up. So we treat lateness as a property of a pair, a record and an operator, and we preserve event time and useful source metadata through every stage, so a later stage can reason about the record instead of inheriting a verdict formed upstream.
The second is that the threshold cannot be copied. There is no universally correct delay such as five or ten minutes. Delay distributions differ by source, by region, by event class and by hour, and the frameworks do not agree on what the knob even means. Flink watermarks flow through operators and can account for idleness and alignment. Beam couples watermarks with triggers and allowed lateness. Spark uses a watermark to bound state and to determine how late records are treated under the operations that support it. The same number is a different instrument in each, so we tune inside the chosen engine's contract rather than importing a value from somewhere else.
What this changes in practice is where the design effort goes. Most arguments about thresholds are really arguments about what happens to the records that fall outside them, and that question has a default answer when nobody chooses one: those records are dropped, quietly, and nothing on the dashboard says so.
So we make the late path explicit before tuning anything. Every windowed output gets a named destination for records that arrive beyond the watermark: update the result, emit a retraction or upsert, write to a side output or a correction table, trigger a bounded backfill, or reject with evidence. The sink has to support the semantics we picked, because appending a second aggregate to a table that consumers treat as final does not correct the answer, it double counts. Corrections carry a result version, a window identity and a reason, so a consumer can apply them deterministically instead of guessing which row wins.
The late path is also owned. It gets count, source, delay distribution and repair age, like any other production surface. Discarding late data is a legitimate choice, but it is a business decision with a documented impact and a way to quantify what was excluded, not a side effect of a threshold nobody revisited.
Timestamp quality comes before all of it. Client clocks are wrong, timezone conversion shifts values, producers reuse creation time after a retry, and a future-dated event pushes progress in a way no lateness policy can undo. We quarantine invalid time separately from honest lateness, because a broken clock contract is a different failure and no watermark setting repairs it.
Building the data flows underneath this is what our data and analytics practice does at Edilec. The full write-up, with the state and cost tradeoffs behind the threshold choice, is at Event-Time Watermark Tuning and Late Data.
Top comments (0)