How file checkpoints, event identity, and source versions work together in a Databricks ingestion design on AWS.
A file lands in S3. The job turns green. Everyone enjoys a brief, suspiciously peaceful moment.
Then a recovery process delivers the same business events in a different file. The ingestion reads both objects correctly, and the downstream table now has duplicate entities or an older customer address. The green indicator has done its job: it reported that the process finished. Unfortunately, nobody had asked it to settle questions of identity and history.
Reliable replay starts by giving each layer a smaller, clearer responsibility. A file checkpoint, an event identifier, and a source version solve different problems.
This reference design includes a sandbox configuration and a recovery rehearsal plan.
This article assumes familiarity with Auto Loader, Delta Lake, and structured streaming on Databricks. If you are new to those concepts, the Auto Loader overview and Delta Lake documentation are good starting points before continuing.
There are three different things to recognize
First, identify the physical input. Which object did the reader discover and process? File-level identity and ingestion metadata help answer that question and support recovery of the reading process.
Second, identify the logical event. Two different objects can carry the same source event. Reading each object once does not establish that each business event appears once.
Third, identify precedence for an entity. A customer can have many legitimate events. A current-state table must know which version takes precedence, while a history table may need to retain every meaningful transition.
Write down the contract before choosing a deduplication expression. For each layer, name the identity, the ordering rule, the treatment of conflict, and the recovery behavior. “Exactly once” is more informative when followed by an explanation of exactly what is counted once.
Give the layers concrete responsibilities
A small reference architecture might look like this:
Producer -> S3 -> Auto Loader -> Bronze Delta
|
validation / quarantine
|
Silver state or history
|
contract-based consumption
Bronze preserves received data and provenance. Silver applies the business interpretation: keys, versions, operations, and the destination model. Consumers receive a documented view rather than inheriting every ingestion assumption.
These layer names are convenient, not mandatory. The useful design decisions are who owns each output, how long it is retained, how it is rebuilt, and which failures stop a batch from being promoted.
S3 notifications can be delivered more than once, and their arrival does not establish a global business order. Treat a notification as a discovery signal, not as a reliable “this customer version is newest” certificate. Amazon S3 documentation.
A checkpoint tracks the reader’s progress
Auto Loader uses checkpoint-associated state to track discovered files. Its supported file-ingestion guarantees do not turn repeated events inside different files into unique business events. Auto Loader overview.
Here is an illustrative sandbox configuration:
from pyspark.sql import functions as F
# Replace these with authorized sandbox locations and identifiers.
source_path = "s3://YOUR_BUCKET/customer-events/"
schema_location = "s3://YOUR_STATE_BUCKET/customer-schema/"
checkpoint_location = "s3://YOUR_STATE_BUCKET/customer-checkpoint/"
bronze_table = "sandbox.ingestion.customer_events"
incoming = (
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", schema_location)
.option("cloudFiles.schemaEvolutionMode", "none")
.option("rescuedDataColumn", "_rescued_data")
.schema("customer_id STRING, source_version BIGINT, value STRING")
.load(source_path)
.select(
"*",
F.col("_metadata.file_path").alias("source_file"),
F.col("_metadata.file_modification_time").alias("source_file_mtime"),
F.current_timestamp().alias("ingested_at"),
)
)
query = (
incoming.writeStream
.option("checkpointLocation", checkpoint_location)
.trigger(availableNow=True)
.toTable(bronze_table)
)
query.awaitTermination()
The selected metadata fields record the source file and its modification time alongside the ingestion timestamp. This makes it possible to trace the same business event arriving in two different objects. These are provenance fields, not a source-version ordering rule. Reserve these output column names in the ingestion contract. File metadata reference.
With a supplied schema, Auto Loader defaults to schema evolution mode none. Without a rescued data column, unexpected fields are ignored. The explicit rescuedDataColumn option above retains them for review instead of silently losing them. It does not add them as typed business columns or automatically quarantine malformed JSON. Schema and rescued data reference.
Use a dedicated checkpoint for each ingestion workload and preserve it during ordinary recovery. Without its checkpoint, Auto Loader starts without its previous file-processing history. With the default cloudFiles.includeExistingFiles=true, existing eligible files under the source path are included again, which can append their rows to Bronze a second time. Source filters and file availability still apply; this cannot recover files already removed. Auto Loader options.
Watch checkpoint size as the workload grows; state accumulates. The production docs cover the operational details that go beyond this configuration.Auto Loader production guidance.
Deleting a bookmark does not make a book shorter. It mainly makes remembering your place more exciting.
Business replay belongs in the destination contract
Imagine that customer A is already stored at source version 10. A replay brings version 8. The ingestion may have done everything correctly. Silver must still prevent the state from moving backward.
A possible current-state contract is:
- Higher source version: eligible to replace the older state.
- Same version, same payload: exact retry.
- Same version, different payload: conflict requiring a defined resolution path.
- Lower source version: older state; do not overwrite the newer state.
The complete implementation must enforce this contract across both the incoming batch and the existing target. It also needs explicit delete and concurrency rules; naming a column customer_id does not enforce uniqueness.
The PySpark lab demonstrates one part of the problem: validating a finite batch, collapsing exact retries, rejecting internal conflicts, and selecting the highest version per normalized key. I covered that part in Your Delta MERGE Is Not a Mind Reader.
For a history requirement, keep the distinction sharp. Reducing a batch to its latest event can be correct for a current-state table and wrong for a consumer that needs the intermediate changes.
Original AI-generated editorial illustration. The returning delivery is a metaphor for replay, not an AWS service diagram.
When your ingestion design already handles file checkpoints and event identity, the next question is whether managed abstractions add anything. The short answer: they handle sequencing and SCD mechanics, but not the business specification that determines what a meaningful change is.
Managed CDC still needs a business specification
Lakeflow’s AUTO CDC APIs offer source sequencing and SCD processing. Their usefulness does not remove the need to define keys, meaningful change columns, delete behavior, and the desired history model. CVerify that these APIs are available in your runtime before building on them.AUTO CDC reference.
For example, a technical ingestion timestamp should not create a new business version merely because the same event was delivered again. Conversely, a real change must not disappear because the comparison excluded an attribute the consumer cares about.
Review a small set of explicit examples with the producer and consumer. Specifications become much easier to evaluate when someone can point at two contradictory records rather than another paragraph containing the word “robust.”
Quarantine needs an exit strategy
Moving a bad row into a quarantine table is the beginning of resolution. Record the reason, a safe reference to the source, the rule version, and enough identity to connect the issue to its batch. Keep unnecessary personal data out of logs.
Then define the return path. Does a correction arrive with a new source version? Is the original event amended? Who approves a replay, and how is the destination protected from duplication?
Without that path, quarantine becomes a very well-organized place where data goes to wait indefinitely.
Do not resolve a conflicting version by picking the alphabetically largest payload merely to keep the job green. That choice is a business rule in disguise. If no valid precedence rule exists, stopping with an actionable explanation can be more useful than quietly storing an arbitrary answer.
Rehearse recovery with a small, unfriendly dataset
Start with synthetic data: two customers, multiple versions of one customer, and an exact retry. Check the expected bronze records and silver state. Record input rows, distinct identities, applied versions, and rejected conflicts.
Restart with the same checkpoint and inspect what actually happens. Deliver the same event in another object. Include an older version and a conflicting equal version. The target should follow the documented rules, and the conflict should reach the defined resolution path.
Also test an interruption at a relevant point in your chosen write mechanism. If the pipeline writes several tables, do not presume that those writes form one atomic transaction. Recovery needs to explain how an incomplete run is detected and reconciled.
Keep evidence from each rehearsal: input files, run identifiers, checkpoint used, expected and actual rows, and the reason each assertion passed or failed. A local transformation test, a service integration test, and an end-to-end recovery exercise answer different questions.
A green job should still have something useful to say
A process can finish successfully after receiving no fresh input. Monitor source freshness, processing delay, rejected records, and the consumer’s actual availability requirement alongside job completion.
Every alert needs an owner and a documented response path. Decide how to distinguish an upstream delivery failure from a transformation defect. The right freshness threshold comes from the use case, not from a universal number borrowed from somebody else’s dashboard.
The design holds up when you can explain, without looking at the code, what happens to a repeated event, an older version, and a conflict. Give those cases names and expected outcomes before they appear during an incident.
Your future on-call self may never send a thank-you note. A quiet pager is a perfectly respectable substitute.
Scope: a reference design with synthetic examples. The companion batch-preparation lab was validated on local Spark 3.5.3. The Auto Loader configuration and recovery rehearsal were not executed against AWS or a Databricks workspace. Running the configuration requires authorized S3 and state-storage access, a compatible runtime, and catalog/schema and target-table permissions. Malformed input, deletes, concurrent writers and full history need their own implementation and tests.
References checked on October 18, 2026.
Top comments (0)