An AI agent can execute a valid tool call and still perform the wrong action.
Consider the request: “Cancel my order.”
If the customer has two active orders, the intended action is clear, but the target is unresolved. Selecting the most recent order introduces an assumption that the user has not confirmed.
The challenge is to distinguish requests that require clarification from those that can be handled using the available context. Asking too rarely can lead to incorrect answers or unintended actions. Asking too often creates unnecessary friction.
What the Research Establishes
Research treats ambiguity detection, clarification-question generation, and answering after clarification as related but distinct capabilities.
CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models presents a framework for detecting ambiguous questions, requesting clarification, and answering after receiving additional information. The study reports improved accuracy on its evaluation of mixed ambiguous and unambiguous questions.
Asking Clarification Questions to Handle Ambiguity in Open-Domain QA introduces CAmbigNQ, a dataset containing 5,653 ambiguous questions with relevant passages, possible answers, and clarification questions. It evaluates three stages separately: detecting ambiguity, generating a clarification question, and answering using the clarification.
CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models evaluates ambiguity handling across a structured taxonomy. Its findings identify limitations in both recognizing ambiguity and producing useful clarification questions among the models evaluated.
Together, these studies support evaluating the complete clarification process. Recognizing that a request is ambiguous does not guarantee that the system will ask the right question or correctly use the answer.
These findings apply to the models and benchmarks studied. They do not establish a universal performance level for current AI agents.
When Clarification Is Necessary
A practical decision rule is to ask whether unresolved information would materially change the answer or action. This is an engineering interpretation of the research, rather than a universal threshold established by the papers.
The decision should account for the request, conversation history, and relevant application state.
| Request | Available context | Appropriate response |
|---|---|---|
| “Cancel my order.” | Two eligible orders | Ask which order |
| “Track order ORD-1042.” | A specific order ID | Retrieve the order status |
| “Make it shorter.” | One draft in the preceding message | Shorten that draft |
| “Compare these models.” | No models identified | Ask for the model names |
| “Explain Python decorators.” | No experience level specified | Provide a general explanation |
These examples are illustrative test cases, not examples taken from the cited datasets.
Missing preferences do not always prevent a useful response. An introductory explanation of Python decorators can be provided without first asking about the reader’s experience.
An unresolved action target requires greater care because different interpretations can produce different outcomes.
What Makes a Clarification Question Useful?
A useful question identifies the specific information needed to continue.
“Can you clarify?” leaves the user to determine what is missing.
“Which order should be cancelled: ORD-1042 or ORD-1087?” makes the unresolved choice explicit.
Before asking, the agent should check whether the necessary information is already available through conversation context, application state, or an appropriate read-only lookup.
The objective is to resolve the uncertainty with as little additional effort from the user as possible.
Ambiguity and Missing Evidence Require Different Responses
A clear question can still be unanswerable from the available evidence.
For example, “What is the refund window for product X?” expresses a specific information need. If the relevant policy is unavailable, the problem is an evidence gap.
The appropriate response is to retrieve the policy where possible or explain which information is missing. Asking the user to clarify an already clear question would not resolve the problem.
For RAG systems, unsuccessful retrieval should therefore be distinguished from uncertainty about the user’s intent.
Evaluating Clarification Behavior
A useful evaluation should include requests that require clarification, requests that are directly answerable, and requests whose apparent ambiguity is resolved by context.
Each case should specify the available context, expected behavior, missing information, and any action that must remain pending.
The following measures address different failure modes:
- Clarification recall: The proportion of cases requiring clarification in which the system asks a question.
- Unnecessary clarification rate: The proportion of answerable cases in which the system asks an avoidable question.
- Question usefulness: Whether the question addresses the information needed to resolve the ambiguity.
- Completion after clarification: Whether the system uses the user’s reply to produce the correct answer or intended action.
- Premature action rate: How often the system performs a consequential action before resolving required information.
These are practical evaluation measures proposed for agent testing. They should not be presented as one standardized metric suite shared by all three papers.
Applying Clarification to Agent Execution
When essential information is unresolved, the agent should preserve the pending task and request the missing detail before proceeding with the affected action.
After receiving a reply, it should update the task context and verify that the ambiguity has been resolved.
Application validation should also check required fields and target identifiers before execution. Clarification behavior and tool validation address different parts of the same problem: ensuring that an executable action matches the user’s intended action.
Reliable clarification requires three decisions: whether a question is necessary, which question will resolve the uncertainty, and whether the answer provides enough information to continue.
Top comments (4)
An agent of mine asked whether to rebase and push, and my reply said it could also do step 1, with a request for manual QA. It wrote back "I read 'you can also do step 1' as a yes to my rebase-and-push question, so I'll rebase, push and open the PR." It made your third decision on its own and said yes, though earlier in that session it had put pushing on its list of actions that need my explicit go.
Jackson Ly you know that's a great example of why explicit confirmation matters. Agents should never interpret partial approval as permission for consequential actions like pushing code.
A clarifying question costs a turn, and each turn replays the conversation so far. Asking is worth it when a wrong action costs more than one extra round.
Absolutely! Clarification should be risk-based. An extra turn is worth it when the cost of a wrong action is higher.