Silent Failure #5: The Suite Was Green Because It Asserted the Bug

A tool-calling request went out of an OpenAI-compatible gateway with its envelope missing. The upstream answered 200, wrote a normal assistant message, and made zero tool calls. Routing was fine. Quota was fine. Health was fine. The provider's own test file was green too — twelve passing tests, zero failures — which is exactly why nobody looked.

The payload was supposed to look like this:

{ "type": "function", "function": { "name": "find", "parameters": { } } }

and it left the gateway like this:

{ "name": "find", "parameters": { } }

Same name, same schema, no type, no function key. Every server that reads tools[].function.name found nothing to read and quietly behaved as if the caller had supplied no tools at all.

The branch that lost the wrapper

A provider adapter rewrites tool names on the way out, and while it was there it also normalised regex patterns inside the parameter schemas. Two branches, three lines apart:

if (fn !== undefined) {
  const fnOut = { ...fn, name, parameters: completeRequiredSchema(fn["parameters"]) };
  return normalizeRe2Declaration(fnOut);                    // wrapper gone
}
if (custom !== undefined) {
  const customOut = { ...custom, name, format: completeRequiredSchema(custom["format"]) };
  return { ...rawTool, custom: normalizeRe2Declaration(customOut) };  // wrapper kept
}

The custom branch rebuilds the outer object and puts the normalised declaration back inside it. The function branch returns the declaration and drops everything that used to wrap it. One line, and the difference is invisible unless you print the payload.

Why twelve tests stayed green

This is the part worth the post. The suite had two tests covering exactly this rewrite, and both of them read the result like this:

const tool = (payload["tools"] as Array<Record<string, unknown>>)[0]!;
expect(tool["name"]).toBe("find");
const params = tool["parameters"] as Record<string, unknown>;

They reached for tool["name"] — the flattened shape. That is the shape the bug produces. The assertions were not merely agnostic about the envelope; they encoded its absence. Anyone reading that test would conclude the wire format was { name, parameters }, because the test says so in two places.

Measured on a clean checkout of main at 05ffff40, bun 1.4.4:

$ bun test test/providers/buddy-donor-gaps.test.ts
 12 pass
 0 fail
 23 expect() calls

Green, on the commit where tool calling through that provider is broken.

The one change that turned the suite honest

I did not touch the source first. I changed the two assertions to read through the wrapper and added a shape test, then ran the same file again:

$ bun test test/providers/buddy-donor-gaps.test.ts
 10 pass
 4 fail
 24 expect() calls

Two of those failures are the old RE2 assertions, now reading tool["function"]["parameters"]. Two are new envelope tests — one against the normaliser, one through the provider's CN pre-payload path. The failure text is the whole story:

expect(tool["type"]).toBe("function")
Expected: "function"
Received: undefined

Four red tests, one undefined. That undefined is what the upstream had been receiving for however many releases.

Then the fix, and the numbers after it

-      return normalizeRe2Declaration(fnOut);
+      return { ...rawTool, function: normalizeRe2Declaration(fnOut) };

$ bun test test/providers/buddy-donor-gaps.test.ts
 14 pass
 0 fail
 32 expect() calls

Fourteen, not twelve: the two envelope tests are new, and the two RE2 tests still assert the same pattern rewriting — they just read it from the place the wire actually reads it from.

What the broad run was hiding

One file being green is not the same as the tree being green, so I ran every provider test:

$ bun test test/providers
 1644 pass
 18 skip
 13 fail
 1675 tests across 174 files

Thirteen failures, and none of them about tools. They are database-backed suites calling getTestPool, and there is no Postgres in this environment. I did not take that on faith: I ran the same directory with my change stashed, diffed the two failure lists, and got the same thirteen test names before and after. That diff is the only reason I can say "pre-existing" instead of "probably pre-existing" — a claim like that is worth exactly the command behind it.

$ bun run typecheck
$ tsc --noEmit          # exit 0

The general shape of this bug

Three things had to line up, and they line up often:

1. A transform function returned the inner thing instead of the outer thing.
2. The consumer of that payload treats a missing key as "feature not requested" rather than as an error — HTTP 200, no tool_calls, no log line.
3. The test that covered the transform asserted the transformed value, not the value's position in the contract.

Number 2 is the reason the bug survived; number 3 is the reason nobody found it. Health checks, quota counters and telemetry all measured the request, not the response, so all of them were telling the truth about a request that arrived and was answered.

What I'd test first

• Print the serialised payload, not the parsed object. JSON.stringify the outbound body in one test and assert on the raw string. {"type":"function","function":{...}} either appears in that string or it does not, and no amount of shape-drift in the surrounding code can make the assertion agree with a bug.
• Read through the contract, never around it. If the wire format is tools[].function.name, the test reads tools[0].function.name. A test that reaches for tools[0].name is asserting a shape the server does not accept, and it will pass forever.
• Add a negative assertion next to every positive one. In the same test: expect(tool["name"]).toBeUndefined(). That single line is what makes the flattened shape impossible to reintroduce silently — the next person who returns the inner object gets a red test instead of a support ticket that reads "the model ignores my tools".
• Diff failure lists, not failure counts. "13 failures, probably the database" is a guess. Same 13 names with the change stashed and reapplied is evidence.

One question I would genuinely like an answer to, because I have only seen this from one side: if you are on bun 1.4.x or Node 22 and you have an outbound payload transform, do you assert on the serialised string or on the parsed object — and has a shape change ever slipped through the parsed-object version in your repo the way this one slipped through mine?