On September 11, I left a comment under Tilde Thurium's post about graph engineering. Tae Kim had described the moment the term clicked: the microservices analogy turned the graph from relationships in a knowledge graph into an agent's control flow, something you can inspect.
Same click here, on several subjects lately. Fifteen years of systems architecture, and I still catch myself arriving, after a day of thinking, at something that was a reflex in microservices four years ago. But what strikes me most is how similar the subjects turn out to be.
The part I meant was the day of thinking. Recognizing an old problem eventually is not the achievement. Recognizing it before spending the day on it would be.
That week I had been discussing idempotent webhooks, then a reconciliation cron with a dead-letter queue, then graph engineering. Different conversations, different starting points, and each one ended on a mechanism I already knew, after I had spent time treating the problem as new.
Click one: the double charge
In a discussion on r/nocode about verifying integrations, the conversation came to duplicate effects when a webhook is retried. My first reaction: a second charge exposes a missing idempotency guarantee. Testing can reveal that gap. Sending another request does not fill it.
The follow-up was about concurrency. Fair point. Send the same event twice, one after the other, and the second request finds the record the first one left. That shows the check works after a completed write. The bug lives in the other case, two requests arriving together.
Both ask whether the event has been processed. Both see that it has not. Both proceed. Every line of code behaves exactly as written, and the system does the wrong thing twice.
For duplicate deliveries of one event, put a non-null database uniqueness constraint on its identity, scoped to the provider and the account where needed. Attempt that insert before the local business writes, and let only the successful insertion proceed. Commit the deduplication record and those writes in the same transaction, so a rollback leaves the event retryable. The database has to arbitrate the competing inserts; an application-level lookup followed by a write leaves a race. PostgreSQL's notes on uniqueness checks explain why the conflict check belongs inside the insertion.
The transaction boundary matters. Commit "processed" first, crash before doing the work, and the next delivery gets discarded forever. Move the record to the end without protecting the writes, and the duplicates come back.
An external charge or email crosses another boundary. Write the outgoing intent in the same database transaction with a transactional outbox; its worker can still deliver twice. Where the recipient supports it, retries need a stable operation key under that provider's idempotency contract, retention window included. A local unique row cannot enforce uniqueness inside somebody else's service.
For a projection of current provider state, I also prefer rereading the provider to blindly applying an old notification. Stripe does not guarantee event delivery order. Concurrent refreshes still need serialization, or a version check before writing locally.
The test I would ask for: deliver the same event concurrently, then inspect the actual effects. If the operation should create one record and send one email, count both. Then interrupt processing around the commit boundary and retry. Those are two different failure modes.
My bias in that conversation was out in the open: I said I come from the development world, and that I am probably biased toward solving this in the database.
I had arrived at a unique constraint again.
Click two: green all the way down
Under my piece on tracking state instead of an agent's history, Mudassir Khan described a webhook that confirmed delivery while the downstream consumer silently dropped messages on a schema mismatch. Clean delivery log, no data. The loss only showed up once a reconciliation step compared inbound and stored counts. The question that came with it was practical: can you wire reconciliation into an existing pipeline without a rewrite?
My answer was an independent second reader. A read-only job on its own schedule, reading the system of record and comparing what exists with what should exist. The pipeline keeps running; the reader starts exposing discrepancies.
A read-back inside the run shares the run's blind spots: same cache, same credentials, same definition of success. An independent reconciler derives the expected records from a durable source and reads the persisted results through a path chosen on purpose. A separate schedule is not enough on its own: the reader has to stay clear of the assumptions that produced the false success. What it produces is a discrepancy report, not one more claim that the run completed.
It is the reconciliation cron that microservices impose on you the day you give up distributed transactions. You need a way to discover that one side advanced and the other did not. The cron exists because nobody gets to infer agreement from their own successful request.
Counting is a good first check, and it has two holes. Matching totals can hide missing records offset by duplicates. Matching identities can hide objects with nothing inside. I keep one in a test app: a French draft with a title, a slug and zero paragraphs, left by a June translation run I remember as green. A count would have passed it; the draft exists. The reader needs a content invariant per object type on top of the count. In my case, at least one paragraph.
Upstream, a schema rejection should land in a dead-letter queue as a recoverable record, not in nothing. Once a fixed batch of unique messages has finished processing, one outcome per message, the accounting is simple:
inbound = stored + dead-lettered
While work is pending, that category belongs in the equation too. Count delivery attempts on one side and unique records on the other, and the equation means nothing. Dead-lettered messages also need an owner and a recovery path: storing the rejection does not repair the operation. The reconciler is the backstop, not the fix.
What struck me was how little of this answer depended on agents. I had been asked about a modern automation pipeline, and I had described a reconciliation job I would recognize in an older integration system.
Click three: the diagram with a new name
Then came the graph engineering discussion.
The useful part of Tae's comment was the shift in what the graph represents. Once I stopped thinking about relationships between pieces of knowledge, I could see the execution structure: which work can start, what it depends on, and what happens when a branch fails.
In the sense used in that discussion, graph engineering makes an agent workflow's control flow explicit and inspectable. Nodes are steps; edges and conditions describe the allowed transitions, including how parallel work rejoins. That structure gives the surrounding software places to enforce state contracts and failure handling, even when a node calls a model whose answer varies.
Tae's comment ended on the case I would test first: a fan-out where one node fails silently. Say a review workflow starts two checks and combines their results. One check returns; the other times out. What lets the join proceed? Does it expose the missing result, retry that branch, or mark the review incomplete? A diagram that leaves those decisions out has drawn the happy path, not the workflow.
I know those questions from service orchestration. Putting a model inside one of the boxes makes the box more interesting. It also gives me more reasons to care about what the arrows permit.
The model can help choose a next step, but that choice still needs a place in an execution contract. Otherwise the diagram describes what usually happens, and the actual control flow lives somewhere in a conversation.
The delay before recognition
Across those threads, the recurring problem was that each participant held only part of the evidence. A worker knew it had received an event. A sender knew its request had been acknowledged. A coordinator had a result from one branch. None of them held the outcome of the whole operation.
Agents add a particularly persuasive participant to that arrangement. It can explain why the task succeeded in a paragraph that reads better than the error message of the component that failed. I find it easy to give that explanation more weight than it has earned.
The analogy has a limit. A transaction can prevent a duplicate write; it cannot tell you the content deserved to be written. A graph can make a decision inspectable, not correct. The old mechanisms contain some of the failures. Judging the model is still extra work.
In Kaamelott, a French TV comedy set at King Arthur's court, Perceval falls back on "C'est pas faux", roughly "can't say that's wrong", whenever a word escapes him. I recognize something of myself there. Sometimes I understand the mechanism before I understand the new name. Sometimes the new name delays the recognition.
That is the part of my own comment I keep coming back to. Fifteen years did not spare me the day of thinking. They gave me somewhere useful to land afterward.
I would just like to hear the click a little earlier.
Which old mechanism did you last rediscover under a new name? Genuinely curious.
Top comments (3)
The quickest bridge I have found between the old and new vocabulary is to classify every agent failure by where durable truth lives. If it is local, use transactions and constraints. If it crosses an effect boundary, use an outbox plus the receiver idempotency contract. If the outcome is unknown, reconcile from the effect owner before retrying. Calling the caller an agent changes the retry pressure and observability, but it does not repeal those boundaries.
Thanks, Jo. "Where does durable truth live" is the earlier click I was asking for. My three conversations reached your three answers in the wrong order, mechanism first and question last, with the day of thinking in between. Your thirty-nine hellos are the third case seen from the effect owner's side. The only failure the classification cannot place is the write that landed exactly once, as requested, and should not have been requested. Nobody to reconcile from, so I guess that one is judgment.
Two recent rediscoveries of my own: sagas under the name 'self-healing agent workflows' (the compensation action was the whole point, and nobody writes it), and event sourcing under 'agent memory' — same pattern with the event log replaced by a summarised conversation, i.e. the part that made it auditable removed first.
One agent-era wrinkle on the dedup section: those mechanisms assume 'the same event' has a stable identity. When the producer is stochastic, content-derived keys break — the same intent retried produces different text, so a hash dedups genuinely different work and lets semantically identical work through. The fix is the same as enqueue-time idempotency keys in job queues: identity has to be chosen at intent time and carried through the run, not derived from the output.
The 'day of thinking' framing is the honest part most retrospectives skip.