A team ships an AI feature. A few weeks later, users start reporting that the answers aren't good. Some responses are inaccurate. Some questions return irrelevant information. An agent occasionally chooses the wrong tool. Latency is inconsistent.
Someone asks the question that has become almost automatic: “Should we switch models?” Maybe. But I wouldn't start there.
The model is usually the most visible component in an AI system, which makes it an easy thing to blame. It's also one of the easiest things to change. Replace Model A with Model B, run a few prompts, compare the outputs, and it feels like you're making progress.
The problem is that you're often changing the component without understanding the failure. An AI application is a system. The model is one component inside it. If the system is producing bad results, the first job isn't to replace the model. It's to find out where the system is actually failing.
Start With Retrieval
Take a RAG system designed to answer questions about internal company policies. A user asks: “Can enterprise customers cancel before renewal?” The retriever returns:
- Pricing policy
- Renewal documentation
- Marketing FAQ
- Cancellation policy
- Product documentation
The model produces an incorrect answer. Is the model the problem? Not necessarily. Maybe the relevant cancellation exception was never retrieved. Maybe the retriever found the right document but ranked it below irrelevant documents. Maybe the chunk containing the actual cancellation condition was separated from the rest of the policy during ingestion. Maybe the retrieval threshold is too low.
In each case, the model is receiving inadequate evidence. Replacing the model might change the final answer, but it doesn't fix the retrieval pipeline. A stronger debugging process starts by inspecting what the model actually received.
- What documents were retrieved?
- What were their scores?
- Was the required evidence present?
- Did the correct chunk survive reranking?
If the answer to those questions is no, you have a retrieval problem.
Then Look at Context
Now assume retrieval is working correctly. The relevant policy was retrieved.
But the prompt contains 15 retrieved documents, the entire conversation history, previous tool results, system instructions, and several other pieces of context. The model now has the right information somewhere inside a large amount of information.
That's a different problem. It's a context management problem. More context isn't automatically better context.
Irrelevant documents can compete with relevant evidence. Long conversation histories can introduce stale information. Tool outputs can consume context without contributing to the current task.
The model might have everything it needs and still produce a poor answer because the useful information isn't being presented effectively. Again, changing models might improve the result. But you're treating the symptom rather than the system.
Before switching models, I'd want to know how much context is being sent, where it comes from, what gets discarded, and which pieces are actually necessary for the current task.
Tool Design Can Look Like a Model Problem
The same thing happens with AI agents. Imagine an agent has three tools:
search_web()
search_docs()
search_customer_db()
The agent keeps selecting the wrong one. The immediate reaction might be: “This model isn't good at tool calling.” But look at the tools.
Do their descriptions clearly explain their boundaries? Does search_web() explicitly say when it should be used? Does search_docs() say that it searches internal documentation? Does search_customer_db() clearly explain what data it can access? Or are the tools poorly named, overlapping, and vaguely described?
If the model has to infer the difference between three ambiguous tools, you've created a routing problem. A better tool definition can sometimes improve agent behavior without changing the model at all. The model can only make decisions based on the information and constraints provided to it. Bad tool design can create bad tool selection.
Without Evaluation, Model Comparisons Are Mostly Guesswork
There's another problem that often gets ignored: teams don't have a proper evaluation set. They switch from Model A to Model B and manually test ten prompts. Model B looks better. So they migrate. But what exactly improved? Did retrieval improve? Did tool selection improve? Did factual accuracy improve? Did refusal behavior improve? Did latency get worse? Did the model become more expensive?
Without a consistent evaluation dataset, you don't really know. For an AI system, I'd want a test set containing different types of cases:
Question → Expected answer
Question → Required source
Question → Expected tool
Question → Must refuse
Now you can run the same evaluation against different models and compare actual behavior. You can also identify where a model performs better. Maybe Model B is better at reasoning but worse at tool selection. Maybe Model A has slightly lower answer quality but significantly better latency. Maybe neither model performs well because the retrieval system is returning the wrong documents. Evaluation turns “this model feels better” into an engineering decision.
Observability Tells You Where the Failure Happened
You can't reliably debug an AI system if all you store is the final response. For a failed request, I want to know what happened along the entire path.
For example:
retrieval_latency
retrieved_documents
reranker_scores
prompt_tokens
completion_tokens
selected_tool
tool_latency
model_latency
evaluation_score
Now imagine a request that took three seconds. You discover retrieval took 40ms. The tool call took 2.4 seconds. The model generated the response in 300ms. The problem wasn't model latency. It was the tool.
Or perhaps the model generated a perfectly reasonable response, but the retrieved documents were outdated. Now the problem is the data. Without observability, all you know is: “The AI response was bad.”
With observability, you can start asking: “Which component caused it?” That's the difference between debugging a system and guessing.
Sometimes the Data Is the Problem
This one is particularly easy to miss. Suppose your RAG pipeline retrieves exactly the document it was supposed to retrieve. The model follows the instructions. The context is clean. The answer is still wrong. Then you inspect the source document. It was last updated two years ago.
No model change will fix outdated source data. The model can only reason over the information you give it. If the knowledge layer is incorrect, incomplete, contradictory, or stale, improving the model doesn't solve the underlying problem. This is why AI quality is often closely tied to the quality of the systems surrounding the model.
Sometimes the Requirement Was Never Clear
There's an even more fundamental failure mode: the team doesn't actually know what “good” means. Should the assistant answer from internal documentation only? Should it refuse when evidence is missing? Should it prioritize the latest policy? Should it cite sources? Should an agent use the CRM or the database for customer information? What should happen when two documents contradict each other?
If those rules aren't defined, there is no meaningful way to evaluate the system. You can't reliably optimize something you haven't defined. A model may produce a response that one engineer considers correct and another considers unacceptable. That's not necessarily a model problem. It's a requirements problem.
The Model May Still Be the Problem
None of this means models don't matter. They absolutely do. There are cases where a model has the right context, receives well-designed tools, follows the intended workflow, and still performs poorly. At that point, switching models may be exactly the right decision. The difference is how you arrive at that decision.
If you've established that retrieval is working, context is appropriate, tools are well designed, data is correct, requirements are clear, and evaluation consistently shows that another model performs better, then you have an engineering basis for changing models. That's very different from changing models because the output “feels wrong.”
Trace the Failure Before Replacing the Model
The model is the component everyone sees. That makes it an easy target. But an AI application is more like: Data → Retrieval → Context → Model → Tools → Application Logic → Output
A failure anywhere in that chain can appear to the user as a bad AI response. So when an AI feature underperforms, I wouldn't immediately ask: “Which model should we use?” I'd ask: “Where did the system fail?”
Was the right information retrieved? Was the context relevant? Was the data current? Did the agent select the correct tool? Were the requirements clear? Can we reproduce the failure in an evaluation set? Do the logs show where the latency or error occurred?
Once you can answer those questions, changing the model becomes much more meaningful. The model might be the problem. But don't assume it is simply because it's the easiest variable to change. Trace the system first. Then change the component that's actually failing.
Top comments (0)