Original AI-generated illustration: a zero-lag dashboard can hide a missing committed offset.
A dashboard declares that your Kafka consumer lag is zero. Support closes the incident. The business report is still missing records.
The chart is happy. The business would like its data back.
The apparent contradiction is not necessarily a Kafka defect. It may be a defect in the meaning assigned to the metric. Kafka offsets tell us how a particular reader relates to a particular partition’s log. They do not, on their own, certify that every business event was processed, that a downstream database commit succeeded, or that a latency target was met.
The distinction matters because the common formula, log_end_offset - committed_offset, is only meaningful after checking that both offsets exist, refer to the same topic-partition and are compatible with the log's retention boundary. Before summing anything across partitions, I want to know which measurements are trustworthy.
Three offsets, three different questions
For each topic-partition, collect these values:
Beginning offset (B). The earliest offset still available to read from the log. Retention and log compaction can affect the relationship between offsets and actual accessible records.
End offset (E). The boundary representing the next offset at the end of the readable log, according to the API and isolation level used by your measurement.
Committed consumer-group offset, C. The offset the group has recorded as its next position. A committed offset is not a receipt from your destination system.
When B <= C <= E, the familiar backlog estimate is:
partition_offset_lag = E - C
Example: if a group has committed offset 140 and the observed end is 150, the offset lag is 10. If C equals E, lag is zero. That is a specific statement about the observed positions, not proof of end-to-end correctness.
The assumption that breaks many homegrown monitors is that C is always present and always falls in that range. It isn’t.
Four statuses instead of one deceptive number
Consider an orders topic observed at 08:00. Four partitions show different states:
Partition: 0; B: 100; E: 150; C: 140; Correct interpretation: Offset lag = 10; position is within the retained range.
Partition: 1; B: 200; E: 260; C: null; Correct interpretation: No committed offset is available. Lag is unknown , not zero.
Partition: 2; B: 500; E: 620; C: 480; Correct interpretation: The committed position precedes retention by 20 offsets. Investigate potential unrecoverable gaps.
Partition: 3; B: 700; E: 740; C: 750; Correct interpretation: The committed position is beyond the observed end. Validate snapshot consistency and log history.
These become four operational states:
OK: B <= C <= E. Compute E - C.
NO_COMMITTED_OFFSET: C is absent. Do not substitute zero. This may be a new group, a deleted commit, a partition with no recorded group offset, or a mismatch between group and topic-partition inventory.
BEHIND_RETENTION: C < B. The span B - C is an unavailable offset range. It is not automatically the number of lost business events, especially on compacted topics, but it is a concrete reason to stop presenting the group as healthy.
AHEAD_OF_LOG_END: C > E. Never clamp a negative result to zero. A racing measurement, broker/log changes or an incorrect partition association could have produced inconsistent boundaries; investigate before claiming anything about throughput.
When even one partition is unknown or inconsistent, my group-level total_lag is null. A partial sum of healthy partitions can still be shown, but it must not be labeled the total.
A reproducible audit in plain Python
You can test these rules with synthetic snapshots without standing up Kafka. Save this as lag_demo.py and run python lag_demo.py with Python 3.10 or later:
from dataclasses import dataclass
@dataclass(frozen=True)
class Snapshot:
partition: int
beginning: int
end: int
committed: int | None
def classify(row: Snapshot) -> tuple[str, int | None]:
b, e, c = row.beginning, row.end, row.committed
if b < 0 or e < b:
raise ValueError("invalid retained log boundary")
if c is None:
return ("NO_COMMITTED_OFFSET", None)
if c < 0:
raise ValueError("negative committed offset")
if c < b:
return ("BEHIND_RETENTION", None)
if c > e:
return ("AHEAD_OF_LOG_END", None)
return ("OK", e - c)
snapshots = [
Snapshot(0, 100, 150, 140),
Snapshot(1, 200, 260, None),
Snapshot(2, 500, 620, 480),
Snapshot(3, 700, 740, 750),
]
results = [classify(row) for row in snapshots]
for row, (status, lag) in zip(snapshots, results):
print(f"partition={row.partition}, status={status}, lag={lag}")
assert results == [
("OK", 10),
("NO_COMMITTED_OFFSET", None),
("BEHIND_RETENTION", None),
("AHEAD_OF_LOG_END", None),
]
Expected output:
partition=0, status=OK, lag=10
partition=1, status=NO_COMMITTED_OFFSET, lag=None
partition=2, status=BEHIND_RETENTION, lag=None
partition=3, status=AHEAD_OF_LOG_END, lag=None
This example classifies provided snapshots; it does not connect to a broker. A production integration would fetch committed group offsets through Kafka’s Admin API (listConsumerGroupOffsets), and the beginning/end boundaries through listOffsets, grouping by exact topic and partition. It would also attach group_id, collection timestamp, cluster identity and client/isolation configuration to each snapshot. Because these calls are not an atomic multi-partition snapshot, transient inconsistencies should trigger a refresh and diagnosis rather than automatic data-loss claims.
The collector and the calculation should be separate components. The code above is the complete local calculation demo for this article; the public lab does not yet contain a Kafka collector or broker integration. For the same unknown-versus-zero principle in a runnable Spark example, see optional operation metrics in my Community lab. That example reads synthetic metric maps, not Kafka offsets.
Why low lag can coexist with a broken pipeline
Commits can run ahead of downstream durability. A consumer might commit offsets and then fail while writing to its warehouse. After restart, the consumer group position can look healthy while the target lacks events. Whether this is possible in a particular architecture depends on commit ordering, destination guarantees and transaction design. Kafka’s at-least-once and exactly-once discussions are about specific processing boundaries, not a magic property of the number in a dashboard.
A group can be fast but stale. A source may not have received events for an hour. Its end offset does not move, so its consumer can show zero lag. Your business freshness SLO is still breached. Measure source freshness, event-time age and sink visibility separately.
One aggregate hides a hot partition. A group with 200 partitions can have modest total lag but one partition carrying the orders that matter. Keep per-partition maxima and age percentiles, not just sum and average.
Offsets are not a wall clock. E - C = 10,000 can mean seconds or hours depending on the arrival rate. Converting count to time by dividing by a short-term rate is an estimate that fails when traffic is bursty or paused.
Retention changes recovery choices. If C falls below B, the consumer may need a policy decision about offset reset or reprocessing from a separate durable source. An automatic reset-to-latest can make the lag chart beautiful while discarding the very history the incident team needs. Define this behavior explicitly and log the decision.
Compacted topics complicate counts. Offsets remain positions in the log, but not every offset within a numeric range necessarily represents a currently readable user record. Do not market offset differences as exact record counts for compacted workloads.
An on-call contract I would actually sign
A useful consumer dashboard has at least three panels. Position health shows per-partition B/C/E, group assignment, committed-offset availability and retention gaps. Pipeline health tracks successful writes, retries, rejected records, deduplication/transaction results and error budgets. Business freshness measures when the most recent valid business event became queryable in the serving layer, relative to the time it was expected.
Alerts should distinguish “backlog above threshold” from “cannot calculate lag,” “committed position outside retention,” and “data not visible downstream.” The first is often a capacity or rate problem; the others can be correctness incidents. Treating all four as the same red/green threshold destroys useful diagnosis.
When a pipeline is repaired, I would require a recovery test: record a known input key, trace its topic-partition-offset, verify the target effect, force a controlled retry, and confirm the intended duplicate-handling behavior. A lag graph cannot run this test for you.
Limitations and where to be careful
The Python example is a local calculation test with fabricated values. It does not validate authorization, Kafka brokers, consumer rebalances, committed-offset races, aborted transactions, compaction behavior or sink idempotency. Kafka’s Admin and consumer APIs differ in how they expose group state and offset boundaries; production collectors need a tested client/version policy. Do not infer that every missing committed offset means data loss, or that every retention gap equals lost events.
If you use Structured Streaming with Kafka, its checkpointed progress and sink semantics must be analyzed as a separate implementation, not conflated with an arbitrary Kafka consumer group’s commits. That is another reason I keep source recovery, pipeline state and business outcomes as separate evidence.
The takeaway
Offset lag is useful once you have proved that the inputs make sense. It tells you how far a recorded position is from an observed log boundary. It doesn’t answer whether the pipeline fulfilled its contract. Keep unknowns unknown, surface retention boundaries, and make the business SLO visible on its own terms.
Official sources
Apache Kafka, Admin API and consumer API; consumer groups and delivery semantics.
Continue in the lab
Data Engineering Lab contains small reproducible examples and explicit validation limits. CDC preparation covers replay, ordering and conflicting versions with synthetic data; it does not validate a live Kafka sink.
For recovery design, read Your S3 Pipeline Needs a Replay Plan, Not More Optimism. For deterministic target updates, read Your Delta MERGE Is Not a Mind Reader.
Top comments (0)