An await finishes. The application returns a result. The trace contains a completed tool span. The test is green.
The tool also returned isError: true.
There is no contradiction. JavaScript promise completion and MCP tool success are different signals. Treating them as the same signal creates a surprisingly quiet failure: an agent can continue after a rejected operation while a completion-only test reports success.
I maintain AgentInspect. This article uses a synthetic test against [email protected] and @agent-inspect/[email protected] to separate those signals. It makes no model calls and does not describe a production incident.
Three questions hiding inside “success”
The MCP tools specification distinguishes protocol-level errors from errors reported inside a tool result. A valid result can carry isError: true.
For an application, there is another boundary beyond either of those: whether the requested effect actually happened.
| Question | Evidence | What it does not establish |
|---|---|---|
| Did the call return? | Promise resolution or protocol response | That the tool accepted the operation |
| Did the tool report failure? | MCP result's isError
|
The final state of an external system |
| Did the intended outcome occur? | Independently observed state or receipt | That every earlier execution step was correct |
A booking tool might respond normally to explain that inventory is unavailable. An email tool might accept a message without proving delivery. A database timeout might leave the write outcome uncertain.
Those cases deserve different application behavior. They should not collapse into a single green check.
A returned error is evidence to evaluate, not an instruction to retry automatically.
Start with a deliberately small reproduction
Install matching package versions:
npm install --save-exact [email protected] @agent-inspect/[email protected]
This JavaScript fixture is intentionally not a real MCP server. It isolates the returned-result branch of the wrapper:
import assert from "node:assert/strict";
import { mkdtemp, readdir } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { inspectRun } from "agent-inspect";
import { wrapMcpClient } from "@agent-inspect/mcp";
const traceDir = await mkdtemp(join(tmpdir(), "mcp-result-"));
const response = {
isError: true,
content: [{ type: "text", text: "Synthetic request rejected" }],
};
const client = wrapMcpClient(
{ async callTool() { return response; } },
{ serverName: "fixture" },
);
const returned = await inspectRun(
"returned-tool-error",
() => client.callTool({ name: "lookup_policy", arguments: {} }),
{ traceDir, silent: true },
);
assert.strictEqual(returned, response);
const [filename] = (await readdir(traceDir))
.filter(name => name.endsWith(".jsonl"));
The identity assertion matters. Instrumentation should not silently replace the caller's result with a different object or invent an exception. In this release, the wrapper preserves the result and records a failed outcome observation for the MCP tool error.
The application still owns the response: explain the rejection, choose an authorized fallback, request clarification, or stop. The tracing adapter records evidence; it does not choose the recovery policy.
Test the observation, not just completion
Append this to the same fixture:
import { openTraceFile } from "agent-inspect/readers";
import {
defineTraceContract,
evaluateTraceContractRead,
} from "agent-inspect/checks";
const trace = await openTraceFile(join(traceDir, filename));
const completionOnly = evaluateTraceContractRead(
trace,
defineTraceContract({ run: { requireCompleted: true } }),
);
const withOutcomes = evaluateTraceContractRead(
trace,
defineTraceContract({
run: { requireCompleted: true },
observations: { failOn: ["failed"] },
}),
);
assert.equal(completionOnly.status, "pass");
assert.equal(withOutcomes.status, "fail");
These are measured results from the included test harness, not a hand-drawn CLI transcript. TraceContract is a beta surface; pin the release and keep the fixture beside the contract.
Notice what the second check says: a captured observation reported failure. It does not claim that AgentInspect independently queried a booking system or verified a database transaction. The observation comes from the MCP response. Its provenance determines its evidentiary strength.
Don’t turn every tool error into a retry
isError is not a retryability classification.
A malformed argument, denied permission, unavailable resource, and transient dependency failure can all produce unsuccessful tool results. Retrying the first two without changing anything adds work rather than recovery. Retrying a side-effecting action after an ambiguous response can duplicate an effect.
Prefer an application-owned, bounded policy. Separate read-only recovery from writes. Where a remote service supports idempotency, follow that service's actual contract. Neither the wrapper nor a trace field makes an operation exactly-once.
Also avoid parsing arbitrary error prose into policy decisions. If your server exposes reviewed structured error categories, test those categories. If it does not, preserve uncertainty instead of guessing from the word “timeout.”
Expand the fixture at the integration boundary
The fake client proves a narrow behavior. Before adopting a wrapper, add real SDK tests for normal results, thrown errors, isError, cancellation, method receivers, and lifecycle methods. Verify captured calls against an independently counted server handler.
The companion examples include an installed MCP SDK client/server test using an in-memory transport. That covers SDK method compatibility without implying that network reconnects, authentication, or every transport have been tested.
This distinction is the habit worth keeping: every green result should name the boundary it actually exercised.
If your current agent test only asserts that a promise resolved, add one fixture in which the tool explicitly refuses the request. Does the run still look successful? If so, the missing assertion is probably not about the final sentence.
Implementation references
- MCP wrapper implementation at the reviewed commit
- Versioned TraceContract documentation
- Companion:
examples/probes.test.mjs, testD01.

Top comments (3)
This gets worse one step later, when the agent writes its own summary: a tool result with isError: true followed by "Done, I've updated the record" is exactly the gap between what an agent reports and what actually happened. Asserting on isError and then reading back the real state catches both halves. Does AgentInspect flag a final message that claims success after an errored tool call?
Good catch, @gautamtalksdev. The trace check can fail on the MCP
isErrorobservation, but it does not infer whether a final natural-language claim is true. I would test that at the application boundary: compare the final claim with the tool outcome and, for a write, an independent readback. That mismatch is a useful regression case.Some comments may only be visible to logged-in visitors. Sign in to view all comments.