The failure nobody demos
The agent demo always works. The customer asks for a refund, the agent calls the right tool, and the money moves. Then it ships, and a week later it refunds an order that was never delivered. Or it issues a refund larger than the purchase. Or it approves an expense for a "manager" who has no reports.
None of these are reasoning failures. The model reasoned fine. It just acted on something that isn't true in your business, because nothing in the system told it what is true.
Prompts ask. Ontologies define.
The usual fix is a longer system prompt: never refund more than the order total; only refund delivered orders. That helps, until the prompt grows to 4,000 tokens and the rule you care about competes with forty others for the model's attention. A prompt is advice. The model follows it most of the time.
An ontology is a different kind of object. It is a formal, machine-readable statement of what can exist in your domain and how things relate: an Order has exactly one Customer, a Refund points to exactly one Order, a Manager has at least one direct report. It doesn't ask the model to behave. It defines the world the model is allowed to act in, and it can be checked mechanically, every time, in milliseconds.
The pattern is simple: the LLM proposes, the ontology disposes. The agent stays flexible in how it understands a request. It loses the freedom to commit an action that contradicts the domain.
A guardrail you can read
The W3C standard for this is SHACL (Shapes Constraint Language). Here is the refund rule as a shape:
ex:RefundShape a sh:NodeShape ;
sh:targetClass ex:Refund ;
sh:property [
sh:path ex:forOrder ;
sh:class ex:DeliveredOrder ;
sh:minCount 1 ; sh:maxCount 1 ;
sh:message "A refund must reference exactly one delivered order." ] ;
sh:property [
sh:path ex:amount ;
sh:datatype xsd:decimal ;
sh:minExclusive 0 ] .
The "amount can't exceed the order total" rule is a few more lines, written as a SPARQL constraint inside the same shape. Notice who can review this. A compliance officer who will never read your Python can read "a refund must reference exactly one delivered order" and tell you whether it's right.
Closing the loop
The guardrail sits between the agent's decision and the side effect:
- The agent proposes an action as structured data (JSON-LD maps cleanly to the graph).
- A validator such as pySHACL checks it against the shapes. No model call, no tokens.
- If it conforms, the action executes.
- If not, the validation report goes back to the agent as a tool result: which shape failed, on which field, with the human-written message.
That last step matters. "Action rejected" teaches the agent nothing. "A refund must reference exactly one delivered order; order 8812 has status Shipped" gives it something to reason about. In practice it either corrects itself or escalates to a human, which is exactly the behavior you wanted the prompt to produce.
What it buys you past the demo
One artifact does three jobs that teams usually build separately:
- Context. The same ontology that validates actions can generate the agent's tool descriptions and entity definitions. The model sees the domain as it actually is, in a few hundred tokens instead of a few thousand.
- Guardrails. Rules are declarative, deterministic, and versioned in git. A rule change is a reviewed diff, not a prompt tweak someone made on a Friday.
- Evals. Every rejection is a labeled failure: shape, field, timestamp. Count violations by shape across a test suite and you know exactly where the agent misunderstands your domain, instead of reading an LLM-judge score of 6 out of 10.
The trade-off is real. Someone has to model the domain, and that is slow, careful work. But the knowledge already exists in your database schema, your policy documents, and your senior people's heads. The ontology is where it stops being tribal.
Where to start
Don't model your whole enterprise. Pick the one agent action that would hurt most if it went wrong — the refund, the record deletion, the outbound email. Write three shapes for it. Put the validator in front of that one tool call and log every rejection for a week.
You'll learn two things quickly: how often your agent was about to do something your domain forbids, and how many of those cases your prompt had already "handled."
Frank Coyle, PhD, teaches generative AI and LLMs at UC Berkeley's School of Information and taught computer science at SMU for 32 years. He trained in neuroanatomy and neurophysiology at Emory before moving to computer science. He spoke on "Ontologies as Guardrails" at AIE Code Summit.
Top comments (12)
The refund examples landed for me. A prompt that says never refund more than the order total competes with forty other rules once it grows to 4,000 tokens, so it is easy to see how an undelivered order still gets refunded. I liked the loop where a failed SHACL shape goes back to the agent as a tool result with the exact field, instead of a bare rejection. Starting with the one action that would hurt most if it went wrong feels like the right way to try this without modelling a whole domain first.
Those refund and expense examples are exactly where prompt-only rules break down: a natural-language guideline competes with every other instruction for attention, while a schema or ontology lives outside the model and can be enforced deterministically before an action fires. This mirrors how we treat trusted boundaries elsewhere — the model proposes, but cardinality, referential integrity, and business invariants should be checked in code or a rule layer, not trusted to the model to remember. The practical split is to let the LLM handle intent and selection, and make the schema the last gate that rejects an inconsistent or invalid action before it reaches a tool.
A prompt asks the model to follow a rule; a validator on the tool call refuses the call when the rule breaks. The refund limit belongs in the validator, because the model can lose a sentence from the prompt, while the check still runs each time the call is made.
The piece I'd pin down is what a validation event consists of, because "a refund must reference exactly one delivered order" is really two checks wearing one message. The shape check is legality: does this action reference exactly one order of class DeliveredOrder. The truth check is freshness: is order 8812's status in the graph current with the order system right now. pySHACL can only see the graph you hand it, so a stale snapshot makes the guardrail confidently enforce last Tuesday's world. The ontology defines what's legal; the data defines what's true; the report should name which snapshot it judged against.
Worth storing three pins per validation, pass or fail: the shapes version, the data snapshot identifier, and the action as proposed. That makes every rejection replayable - same action, same shapes, same snapshot, same verdict - which is what turns "rules are versioned in git" from a hygiene note into an audit property. A rejection you can replay is evidence; one you can't is a log line.
Completely agree that deterministic schemas beat prompt tweaks every time when building production agents. Without strict JSON schema enforcement at the tool call interface, prompt drift and hallucinated parameter structures quickly snowball into unhandled runtime errors. Well articulated!
the rejection message going back as a tool result is the part most people skip, a bare 'action rejected' just gets you the same call again. same thing in travel, a model will confidently pick a hotel that doesnt fit the dates or the group, which is why at searchspot we'd rather that check sit outside the prompt
"Prompts ask. Ontologies define" is the line that makes this click, and "the LLM proposes, the ontology disposes" is going straight into my notes.
One thing I'd push on: the validation report is written for a compliance officer to read, but it's delivered to the model as a tool result. Those are different readers with different needs. A shape with five failed constraints could easily produce a wall of text a human audits fine and a model drowns in. Have you found that one message per shape works better than one message per violation in practice?
The "how many of those cases your prompt had already handled" line is the one I'd put in bold — that week of logging is the whole argument.
the part that stuck with me is the validation report going back to the agent as a tool result with the real order status in the message. a bare action rejected teaches it nothing, agreed. we did something close with typed errors on a refund tool and retries got way cleaner once the error named the field and the current value. my gotcha tbh is shape drift, the shapes slowly disagree with the database after a few migrations and nobody notices until the validator rejects good actions. how do you keep the ontology in sync with the schema, generate it or review it by hand?
I would version the shapes in the same change as the migration, and fail the build if a stored example no longer validates. That way a schema change cannot leave the checker describing a world the database has already left.
Interesting read! I recently built a small AI agent from scratch in Python as a learning exercise, and separating the LLM from tool validation was one of the most useful things I learned.
One thing I'm curious about is how you would approach this in smaller projects. Would you introduce something like SHACL from the start, or begin with simple Python validation and move to an ontology only when the domain becomes more complex?
Strong framing - "the LLM proposes, the ontology disposes" is the right shape. One extension worth making explicit: validity constraints and budget constraints are different classes, and both belong in the disposal layer.
"A refund must reference exactly one delivered order" is checkable against a static shape. "Refunds today must not exceed $500" is also mechanical, but it's stateful - it can't live in the shape itself, it needs a ledger the validator queries. We ended up with three separate checks between proposal and execution: a per-action cap, a periodic cap, and a prepaid balance for metered actions, each refusing independently.
The nice properties you describe carry over: checks run in milliseconds with zero tokens, a compliance person can read the rule, and the refusal path is auditable - you can show exactly which constraint disposed of which proposal. The awkward part we hit: enforcement has to live server-side. A cap checked in client code is a suggestion.