Your LLM Isn't Bad At Math. It Was Never Doing Math In The First Place.

Your LLM Isn't Bad At Math. It Was Never Doing Math In The First Place.

In a tutorial, an LLM call looks like a function: pass in text, get back an answer, move on. It's easy to start treating the model like it's evaluating your business logic the same way a function would, deterministically, the same input always producing the same output.

In production, that assumption breaks in a specific, predictable way, and it's worth naming precisely why.

Traditional software runs on deterministic logic: if input A meets condition B, output C happens, every time, with a proof you could write on a whiteboard. An LLM runs on statistical probability: it predicts the most likely next token given the patterns in its training data. Those are two different kinds of "correct." The friction shows up exactly where a team asks the second kind to do the first kind's job.

Below are the places that friction actually shows up in a live system. These are illustrative scenarios based on the shape this failure takes, not one specific incident.

Asking a model to do arithmetic is asking it to guess what arithmetic usually looks like

Say a pipeline extracts line items from a batch of invoices, then asks the same model call to also compute the subtotal, apply tax, and return a total. On short, simple invoices, it's right almost every time, because short simple math is heavily represented in training data and easy to pattern-match. On a 40-line invoice with mixed tax rates and a rounding rule, it starts being right most of the time, which is a different and much worse thing than right. Nothing throws an error. The number is just plausible instead of correct, and "plausible instead of correct" is invisible until someone reconciles the books.

A statistically-applied business rule is not the same rule as a logically-applied one

Say the business rule is "refunds are allowed within 30 days of purchase." Put that rule in a prompt and ask the model to decide eligibility case by case, and it will get the easy cases right: a purchase from six months ago, obviously no; a purchase from yesterday, obviously yes. Where it gets interesting is the boundary: day 29, day 30, day 31, across time zones, with a purchase timestamp in one format and today's date passed in another. A deterministic date comparison gets this right every single time by construction. A model is producing its best guess at what "a refund decision near the boundary" looks like, based on how those decisions were phrased in its training data, and that's a meaningfully different operation even when it happens to output the correct answer nine times out of ten.

The output still looks like a normal response, which is exactly the problem

This is what makes the failure mode dangerous rather than just annoying: a wrong statistical answer doesn't look different from a right one. It's the same JSON shape, the same confident tone, the same absence of an exception. There's no signal in the response itself that tells you this was a guess rather than a computation. You find out later, from a customer service escalation or a finance reconciliation, not from anything in your logs.

Where the line actually goes

None of this means don't use the model near your business logic. It means be precise about which half of the job you're handing it.

What the model is good at: turning unstructured input, an email, a contract clause, a support ticket, a scanned form, into structured data. That's genuinely ambiguous work: language is fuzzy, humans phrase the same request ten different ways, and a statistical model that's seen millions of phrasings is the right tool for mapping "hey can I send this back, it's been like a month" into structured fields like intent and stated days since purchase.

What the model should never be the last word on: the validation, the calculation, and the rule enforcement that runs on that structured data once you have it. That's deterministic code's job, on purpose, with a strict schema at the boundary so a malformed or out-of-range extraction fails loudly instead of quietly flowing downstream as a confident-looking guess.

We lean on this split in how Cyclopt's automated checks work: the model interprets unstructured signals in a codebase or a pull request, but the actual rule enforcement, the pass or fail line, runs through deterministic logic against a defined schema, not through the model re-deciding the rule each time. The interpretation layer changes. The rule layer doesn't get to.

The reframe

Statistical vs. logical correctness isn't a model-quality problem that gets solved by a better model. It's an architecture decision, and better models make it easier to ignore, not less necessary to make. Let the model handle the ambiguity. Let your code handle the rules.

Where do you actually draw that line in a system you've shipped? What's the one thing you learned the hard way should never have been the model's call to make?