Do You Actually Need an LLM?

I spent a weekend running two small language models against three jobs I would normally hand to a language model without thinking about it. For one of those jobs I also wrote a regular expression, as a control.

The regex won. Thirty test cases out of thirty, in zero milliseconds, from a pattern that took an afternoon to write. The better of the two models scored twenty-eight, took five seconds a call, and the two it missed are both mistakes a regex is structurally incapable of making.

That is the kind of result I was looking for, and the question behind it is one I keep avoiding: what is the smallest amount of computation that reliably does this job? Not the cheapest model. Not the best small model. The smallest thing that works, which means the ladder has four rungs and the bottom one is not a model at all.

1. Deterministic code. A regex, a lookup table, an if statement.
2. A small local model. Runs on hardware you already own.
3. A larger local model. Needs a GPU.
4. A remote frontier model. Someone else's hardware, metered.

Most projects start at rung four and never ask about the others. This is a weekend spent asking.

The hardware was a Raspberry Pi 4, 4 GB of RAM, heat sinks and a fan, a hundred and seventy dollars most of which I already owned. The models were smollm2:360m and qwen2.5:1.5b through Ollama. The test cases are ninety-five questions about a fictional small winery, so nothing can be looked up and every answer can be judged by reading. Sixteen of the fifty classification messages are deliberately ambiguous, because real inboxes are mostly ambiguous.

Four verdicts, assigned per workflow, to force a result you can act on instead of a number:

• Unnecessary: deterministic code already does this.
• Sufficient: a small local model does it well enough to run unattended.
• Marginal: usable with a person checking the output.
• Insufficient: the error profile or the latency is not acceptable.

And one rule, which is the part I would insist on if you copy nothing else: I wrote predictions for every workflow and committed them before I installed Ollama. Twenty-two of them, each marked confident, leaning, or coin flip. Once there is a timestamped expectation there is a defined moment when you are done, and no room to decide afterward that you knew all along. I will come back to how they scored, because it is not flattering.

Extraction: pulling an order number out of a message

Find ORD- followed by six digits, or report that there isn't one. Thirty cases: twenty-two with an identifier, five without, and three decoys. The decoys are a membership number, a UPS tracking number, and a phone number, all of which look like identifiers and none of which are order numbers.

The regex won. Thirty out of thirty, in zero milliseconds, at zero watts above idle.

That result is boring until you read the two the best model missed.

On ORD-999000: wrong address, please hold. it answered 999000, dropping the prefix. Annoying, recoverable, the kind of thing you fix with a post-processing step you should not need.

On ORD-111222 please refund it answered NONE.

That is the shortest and least decorated input in the whole set. Almost the entire message is the order number. And the model returned a confident, correctly formatted assertion that there is no order number present. In a pipeline, that is a refund request for an order that exists arriving at a system that has decided it does not.

A regex cannot make that mistake. Not "rarely does" — cannot. The two errors separating 28 from 30 are both of a class that the deterministic baseline is structurally incapable of producing, which is a different and much stronger statement than a two-point accuracy gap.

The 360M result needs two numbers, not one

By the contract, smollm2:360m scored 8 of 22 on the cases that contained an identifier. That reads like a model that cannot find things.

It is not. All fourteen of its wrong answers are the correct six digits with ORD- stripped. ORD-556120 came back as 556120, fourteen times out of fourteen.

It located the right identifier in 22 of 22 and formatted it correctly in 8. Those are two different skills and the single score hides one of them. The model understood the task perfectly and could not obey the output contract, which is a completely different engineering problem than "the model is too small," and it has a completely different fix. (Constrained decoding. I have not tested it. It is the obvious follow-up and it is not in this post.)

The decoys follow from the same gap. It handed back MEM-443322, 1Z999AA10123456784 and 503-555-0142 verbatim, all three. It has no model of the format at all. It returns whatever in the message looks most like an identifier.

Which suggests a use for it that is not extraction

A model with perfect recall and no precision is not a bad extractor. It is a candidate generator that has been asked to do the wrong job.

Twenty-two of twenty-two located, eight of twenty-two formatted. Put that output through the regex as a validator rather than trusting it directly, and you get the model's recall with the pattern's precision. The model proposes anything that looks like an identifier, including format variants the pattern was never written for. The pattern rejects anything malformed.

Crucially, neither half of that pipeline can produce the error the 1.5B model produced alone. The model cannot cause a false negative because it does not decide, and the regex cannot invent because it only matches.

I have not tested that composition and it is not in these numbers. But it is the one place in the extraction result where a small model has a job worth giving it, and it is a job where being sloppy is free.

Verdict for extraction itself: Unnecessary. Write the regex.

Classification: routing a message to one of six queues

Appointment, billing, availability, complaint, question, escalate. Fifty messages. The instruction said to use escalate when the message needs a person, including when you cannot tell what it is about.

The 360M model answered escalate to 43 of 50 messages, which is not classification, and there is nothing further to say about it here. The rest of this section is the 1.5B model.

It is classifying, and it is not good at it. Forty-six percent, with twenty-six messages routed confidently to a wrong queue that was not the safe one.

Read those twenty-six and the verdict writes itself:

• Someone slipped on the steps by the barrel room and hurt their wrist → complaint
• A formal notice regarding a trademark → complaint
• "I have a severe sulfite allergy, what is actually in the wine" → complaint
• A reporter asking about the 2024 harvest → question
• A distributor wanting to carry the wine in forty stores → question

An injury report, a legal notice and an allergy question, all routed to the complaints queue, confidently, with no escalation. All five billing messages about money problems also went to complaints.

Forty-six percent sounds like something you could work with. Twenty-six confident misroutes including an injury and a legal notice is not something you can work with.

And then I broke the score down by class

This is the part I nearly missed, and it is the most useful thing in the experiment.

Two classes near perfect. Four poor. One at zero.

The split is not difficulty, and it is not class size. It is what kind of label each one is.

complaint and question are topics. You can tell what a message is from the words in it. Someone complaining sounds like someone complaining.

escalate is a routing decision. Nothing in the content of a message makes it an escalation. It is an escalation because of what should happen next. "I have a severe sulfite allergy, what is actually in the wine" and "what time do you close" are both questions about the business, with the same grammar and the same tone, and one of them is a liability event.

The model did topics at 90 to 100 percent and routing decisions at zero. And billing sat in the middle for exactly the same reason: a message about being charged twice is a complaint, as text. The label is about which department owns the problem, which is not a property of the sentence.

Small models answer "what is this about." They do not answer "what should we do about it."

That reframes the five examples above. They are not five unlucky misroutings. All five are escalations, and escalate scored zero of twelve. The model was never going to find them, because finding them requires knowing a consequence that the text does not contain.

Two caveats on how hard to lean on this. The classes hold between six and twelve cases each, so "six of six" is six cases and the individual numbers are thin. And this is one model. The pattern across all six classes is consistent enough that I believe the explanation, but I am offering it as the best account of the data rather than as a measurement.

Verdict: Insufficient. And the actionable version: if one of your labels means "a human needs to see this," a small model will not find it. Build that one as a rule.

Grounded QA: answering a question from four documents

Fifteen questions against a 650-token corpus: club policy, tasting room, production notes, shipping. Eight are answerable from one passage, three require combining two, and four have no answer in the material at all.

I read all thirty answers by hand against the corpus. That is the only way to score this, and I will say why in a moment.

Three boundaries separate cleanly here, and they are the useful result.

Retrieval works

The 1.5B model answered all eight single-passage questions correctly. Six bottles in the Vineyard tier. Twenty dollars per person. September 9th. Thirty-four states. No dogs indoors. 21.8 Brix on the 12th. Twenty-five dollar restocking fee. Ten people on the barrel room tour.

Asked what happens if you join the club and immediately want out, it returned the two sentences that answer the question and stopped:

Cancellation requires one completed shipment. Members who cancel before receiving a shipment are charged for that shipment.

That is competent lookup across four documents, on a $170 computer, with the wifi switched off. I did not expect it and it is the most encouraging thing in the weekend.

Combination does not

One club member brings five guests on a Saturday. Club policy waives the tasting fee for the member plus three guests. The tasting room charges twenty dollars a head. Six people, four waived, two paying: forty dollars.

Both models answered twenty.

Neither could hold "plus three guests" against a party of six and subtract. The 1.5B model did get the second half right, correctly working out that a party of six is not a group larger than six and therefore needs no reservation, which is the only piece of multi-document reasoning either model performed all weekend.

The second two-passage question is the one I keep thinking about. Asked why someone would not find a 2024 rose and what they would be offered instead, the 1.5B model quoted the corpus exactly, and then:

...the reference material states that "We did not bottle a rose in 2024. The fruit that would normally go to rose was needed to make volume on the estate Pinot Noir." Therefore, the 2024 rose would be replaced with the fruit that would typically go into the estate Pinot Noir production.

The quotation is verbatim and right. The next sentence reverses the only relationship in it. The fruit went into the Pinot Noir, so what you would be offered is the Pinot Noir. You cannot be offered fruit.

Correct retrieval and inverted inference, in consecutive sentences, with the evidence on screen between them.

The 360M model's multi-document failures are worse than wrong. They are contaminated. It took the fourteen-days-notice rule, which governs skipping a club shipment, and applied it to booking a tasting room reservation. It reached into the shipping document to answer a cancellation question. Given more documents to search, it finds more wrong ones.

Knowing the answer is absent is the weakest link

Four questions have no answer in the corpus, chosen to be things a reader would reasonably expect a winery to have written down: a wine's alcohol percentage, who the winemaker is, what a club tier costs, what yeast was used.

The 1.5B model abstained on two and fabricated two. The 360M model fabricated all four.

And the fabrications are a more dangerous species than invention. Asked for the alcohol percentage of the 2024 estate Pinot Noir, the 1.5B model answered:

The 2024 estate Pinot Noir had a Brix range of 23.1 to 24.6 at pick.

Every component of that is true. Right wine, right vintage, right numbers, right document. Brix is sugar at harvest and has nothing to do with alcohol in the bottle. It reached for the nearest real fact instead of making one up, and produced something that reads as expertise while answering a question nobody asked. If you do not know what Brix is, that answer is indistinguishable from a correct one.

Both models made the same mistake on what the Cellar tier costs: both answered "twelve bottles per quarter." Grounding gave them the right tier and they handed back the quantity in place of the price.

The one answer the evidence did not change

The 360M model, asked whether a dog can come inside the tasting room:

Yes, you can bring a dog inside the tasting room.

The corpus says the patio is dog friendly and the indoor room is not, service animals excepted. I have the same answer from a run where the model had no corpus at all, and it is identical byte for byte. Having the right answer directly in the prompt changed nothing.

In a category where being wrong means a family drives out to a winery and gets turned around at the door.

Verdict: Marginal. Usable with a person reading the output. Not usable without one.

What it costs

Two things in here surprised me.

The compute tax does not track parameter count. The 1.5B model is slower than the 360M on extraction, 4.3 against 5.9 tokens per second, and faster on classification, 7.5 against 6.2, because on classification it answers in two tokens where the smaller model takes six. Four times the parameters, and which one wins depends entirely on how much each one decides to say.

A bigger prompt is cheap. A longer answer is what you pay for. I have a clean comparison here by accident: an earlier QA run went out with 97-token prompts and the real one with 708. Seven times the prompt added fifteen percent to the wait before output started, because prompt evaluation parallelizes. The latency increase came from answers three times longer, plus a twenty-one percent drop in per-token generation speed from attending over a longer context. Median latency went from 3.6 to 6.9 seconds and the slowest single call took ninety-eight.

If you are sizing a local system, that is the inversion to internalize. Stuffing more context in is affordable. Letting the model be chatty is not.

And check your thermals, which is not optional advice. My first attempt at this ran with heat sinks and no fan. Thirty calls to a 360M model was enough to hit the Pi 4's soft temperature limit and start capping the clock. vcgencmd get_throttled returned 0xe0000. I threw those latency numbers away.

Nothing in Ollama's output, my scorer, or the result files said a word about it. Every number on screen was internally consistent and quietly describing a machine that was slowing down. The only way to find out was to ask the firmware a question I had to already suspect the answer to. Buy the fan.

And it does run with the network off. I switched the radio off, proved there was no route out, and ran one call from each workflow. All three worked. The claim that supports is narrow and worth stating carefully: this
configuration, on this afternoon, ran with no route to the internet. It is not a claim that local models are private or that this setup is secure, and a Pi can be misconfigured as thoroughly as anything in a data center. But for the specific question of whether the inference needs the network, it does not.

So: do you actually need an LLM?

Three workflows that all sound like language tasks, three different answers, and not one of them is "yes, use a model."

And the same line runs through all three. Extraction asks what this string is. Topic labels ask what this message is about. Single-passage lookup asks what the document says. Those went well, or well enough. Escalate asks what we should do, the fee question asks what the customer owes, and the unanswerable questions ask whether we know. Zero of twelve, both models wrong, and two of four.

A small model will tell you what something is. It will not tell you what to do about it. Every result here falls on one side or the other of that sentence, and I did not have it written down when I started.

Which gives four properties worth checking before you reach for one:

1. The answer is already present in the input. Not inferable from it. Present in it.
2. There is exactly one of it. The moment two facts have to be combined, this stops working, and the arithmetic in that tasting fee question is the proof.
3. The output is short. Longer answers are where the latency lives on this hardware, and where the fabrication lives everywhere.
4. Something other than the model decides whether the output is acceptable. A regex, a schema, a person, an index that either returns rows or does not.

Take away any one of those four and the experiment says do not do it. The 46 percent and the zero of twelve are what it looks like when you ask a small model to be the thing that decides.

And if you have all four, you may not need the model either. That is worth checking first, because it was true for one of my three workflows and I did not expect it to be.

Where I was wrong

Twenty-two predictions, committed before I installed anything. Ten right, eleven wrong, one unresolvable.

Two of my seven confident calls broke. I was sure the 360M model would get most of the easy extraction cases, and it got 8 of 22 by failing at format rather than at finding. I was sure it would produce invalid output often, and it did so once in fifty, then emitted the same valid word forty-three times.

But the number that bothers me is not in the confident column.

My hunches did worse than my coin flips. Two of ten on the calls where I had a feeling but not much behind it, three of five on the ones I explicitly labeled as guesses.

That ordering is backwards and it is the most useful thing on the sheet. On the coin flips I knew I was guessing, so I guessed cleanly. On the leaning calls I was pattern-matching against intuitions about model behavior that turned out not to describe these models at all, and the feeling of having a reason was the problem. If I had marked all twelve of those as coin flips I would have scored better.

I will take that over a clean result. The honest version of this exercise is not "here are the numbers," it is "here is what I believed, here is what happened, and here is the gap."

The gap, this time, was mostly in the hunches.

---

The harness that produced these numbers had five measurement flaws in it, none of which manufactured an entire finding out of a missing directory and nearly got published. That is a separate post.

Everything here is reproducible: fixtures, runner, scorer, the committed predictions and the run record are at
github.com/kenwalger/right-size-the-model.
Fixture set v1. Read LIMITATIONS.md before citing any of these numbers:
two models, one board, one quantization, ninety-five cases. This is an illustration of a method, not a benchmark.