0.27 swing: I re-ran Amazon's key-order bug on the Decider

Amazon's blog post about replacing Claude Sonnet with its Strands Decider mentions, almost in passing, that "JSON key order changed the answer" — reordering the keys moved the output probabilities by up to 0.19. I ran the open-source Decider locally and reordered the keys of the same ticket myself: the top choice's probability fell from 0.80 to 0.53, a 0.27 swing, and the confidence score fell from 0.73 to 0.37. Same ticket, same question, same model. Only the key order changed.

What I actually ran

Strands Decider is open source (strands-labs/strands-decider), and the 2B checkpoint is on Hugging Face, so I installed the strands-decider CLI with CPU torch and ran it on this VM — 4 vCPUs, 8 GB RAM. One detour worth reporting: the stock CLI upcasts the torso to fp32 on CPU, which is about 7 GiB of weights, and the kernel OOM-killed the process seconds after the weights finished loading. The package's own docs say bf16 gives the same answers on CPU, just slower, so I kept the bf16 torso with a runtime monkeypatch (no repo files modified) and it loaded in 10.8 seconds from cache. The numbers below are bf16-on-CPU; the authors' note is the basis for treating them as representative, and the repeat run below is the check that the setup is stable.

The ticket is modeled on Amazon's own example — a payouts failure — sent as a JSON state with a four-way routing question (billing / engineering / security / sales, each with a one-sentence criterion, the way Amazon rewrote its labels):

state A (original key order):
{"ticket_id": 48211, "subject": "payouts failing", "error": "PAY-104",
 "failed_transfers": 14, "days": 3, "bank": "same bank"}

The experiment: four runs, one ticket

run                        billing   engineering  security  sales   confidence  wall
A  original key order      0.8009    0.1735       0.0176    0.0080  0.7345      8.49s*
A  repeat (determinism)    0.8009    0.1735       0.0176    0.0080  0.7345      5.61s
B  keys reversed           0.5299    0.4346       0.0281    0.0074  0.3732      4.30s
C  keys sorted             0.5842    0.3821       0.0255    0.0082  0.4456      5.10s
*first run includes warmup

Three things to take from this table.

The model is deterministic, and the shift is real. Run A repeated gives identical probabilities to four decimal places, so the 0.27 move in run B is not sampling noise — there is no sampling here at all. Reversing the JSON key order moved billing from 0.8009 to 0.5299 (−0.271) and engineering from 0.1735 to 0.4346 (+0.261). The top choice stays billing in all four runs, which is why this bug is easy to miss in production: the label doesn't flip, the confidence quietly collapses. At Amazon's own setup — a tuned confidence threshold deciding whether to apply a label — the same ticket goes from "confident enough to auto-label" (0.73) to "below the gate, needs a human or a fallback" (0.37). The routing decision changes even though the argmax doesn't.

My shift is bigger than Amazon's. They reported "up to 0.19"; I measured 0.27 on one ticket with one question. That's not a contradiction — 0.19 was their max over their test set, mine is a single example — but it means 0.19 is not a bound on the effect. It's one observation of it.

Sorted keys don't restore the original answer. Amazon's fix is to send every request with sorted keys, and run C is exactly that: billing 0.5842, still 0.217 away from run A's 0.8009. Sorted keys give you a stable serialization — the same input bytes every time — not the same answer the unsorted input gave. If you compare a model's behavior before and after adopting sorted keys, or compare two systems that serialize differently, you're measuring the serialization as much as the model. The model's own evaluation data agrees it's surface-sensitive: paraphrase consistency on JevBench is 86.1% (36 pairs) — reword the same question and the answer changes about one time in seven.

Two more numbers for completeness: every run used 148 input tokens and exactly 1 output token, which is the structural claim the whole cost story rests on — a decision model emits one choice, not a JSON document. And the wall times (4.3–8.5 s per ask on a 4-vCPU box) are CPU numbers, not comparable to Amazon's 640 ms inference on an A10G; the point of running locally was the probabilities, not the latency.

The cost story, recomputed

The experiment is the new part; the cost audit is the context it sits in. Amazon replaced Sonnet 4.6 with the E4B Gemma 4 build of Decider for labeling tickets into 20 categories, on SageMaker ml.g5.xlarge instances, behind a three-way AppConfig switch (BEDROCK / DUAL / DECIDER) with a 45-second rollback. Their table, with my recomputation:

metric                  Sonnet 4.6    Decider E4B    blog says    I recomputed
avg time per call       4,215 ms      897 ms         -78%         -78.7%
input tokens/ticket     6,554         2,009          —            -69.3%
output tokens/ticket    170           1              —            -99.4%
total tokens/ticket     6,724         2,010          -70%         -70.1%
cost per ticket         $0.0222       ~$0.0005       44x / -98%   44.4x / -97.7%

The Sonnet figure checks out exactly: 6,554 input tokens at $3.00 per million plus 170 output tokens at $15.00 per million is $0.019662 + $0.00255 = $0.022212. The Decider figure checks out too, with a rounding footnote: at 640 ms per request, one $1.408/hour instance handles about 5,600 tickets an hour, which is $0.00025 per ticket at full utilization — $251 per million, which the post rounds to $250. The "$500 per million in practice" figure is the one with headroom for bursts baked in, and it's the honest number to quote: $0.0005 per ticket, 44.4x cheaper, a 97.7% reduction. The "98%" in the headline is that number rounded up.

One more computation the post doesn't make, and it changes who should copy this: the Decider is billed per GPU-hour, Sonnet per token. An always-on ml.g5.xlarge costs $1.408 × 730 = $1,028 a month whether it labels one ticket or a million. At Sonnet's $0.0222 a ticket, the breakeven is 46,299 tickets a month. Below that, Sonnet is cheaper — at 1,000 tickets a month, Sonnet costs about $22 and the GPU costs $1,028. The 98% is a volume number. Amazon has the volume. Most teams reading this post don't.

The number that isn't in the post

"Performed just as well" is the load-bearing claim, and it has no number attached: no accuracy figure, no sample size, no agreement rate between Decider and Sonnet on the DUAL traffic. The post says E4B "beat production Sonnet on a separate set of tickets" and leaves it there.

So I went to the model's own repository and pulled its published evaluation data from Hugging Face — the authors' measurements, re-analyzed from their files:

benchmark                              accuracy    n       note
JevBench (w4096, released 2B model)    77.9%       231     ECE 0.072
held-out short tasks                   65.3%       6,000   ECE 0.054
ContractNLI                            87.8%       1,026
HotpotQA (held out)                    83.2%       959
MuSiQue                                88.2%       1,199
boardgame                              81.0%       900

Two things stand out. First, the calibration claim in the README — "answers at a confidence of 0.9 or more are right about 95% of the time" — actually holds on JevBench: in the 0.9–1.0 confidence bin, 82 of 82 answers were right. But the bins below it are shakier: 0.8–0.9 is 93.1% (29 items), 0.7–0.8 is 73.1% (26 items), and 0.6–0.7 is 77.8% — worse than the bin above it, on small samples. The confidence score is a usable gate at 0.9 and a rough hint below it, which is exactly how Amazon used it. Second, the held-out short-task accuracy of 65.3% on 6,000 tasks is the number the headline doesn't want you to see: on tasks it has never seen, the model is right about two times in three. That's fine for a triage label with a confidence gate and a fallback. It's not "just as good as Sonnet" without the gate.

Latency, from the repo's own evaluation: on an RTX 3090, one JevBench question takes a median 115 ms (p95 299 ms); on an M3 Pro, median 234 ms (p95 2,628 ms). Amazon's 897 ms average call includes the Lambda and SageMaker round trip around a 640 ms inference.

When this works, and when it doesn't

Amazon's own lessons section is unusually candid, and it matches the data: this worked because the task is a closed choice from a known catalog, the volume is high, and the old path's latency mattered. It also worked because they were willing to reshape the task to fit the model — production's label descriptions alone were longer than Decider's 4,096-token window, so they rewrote all 20 labels as one- or two-sentence criteria, and on long tickets the model drops the oldest comments to fit. If your labels need explanation text, if the catalog changes weekly, or if the useful context doesn't fit in a few thousand tokens, the post says a decision model probably can't replace the LLM's job. Believe that paragraph more than the headline.

The pattern to steal is the migration, not the model: run both on real traffic, judge every disagreement, keep the rollback at 45 seconds. And add one test Amazon's post implies but doesn't spell out: serialize the same ticket two ways and diff the probabilities. If your pipeline's JSON key order, whitespace, or field order can move a probability by 0.27, your confidence gate is gating on your serializer. Sort the keys, yes — but measure the gap first, because the sorted answer and the original answer are two different answers.

Sources: Strands Agents, "Amazon saves millions by replacing Sonnet 4.6 with Strands Decider" (strandsagents.com, November 9, 2026); strands-labs/strands-decider (GitHub); StrandsAgents/strands-decider-2B-qwen3.5-v1-2610, eval/summary.json and eval/jevbench-w4096/summary.json (Hugging Face, downloaded November 11, 2026); evaluation/results.md in the repository. Cost figures are Amazon's published list prices, recomputed by me above. The key-order experiment is my own run: Strands Decider 2B (Qwen3.5) on CPU, bf16 torso (stock CLI upcasts to fp32 and OOM-kills this 8 GiB VM; the package's docs state bf16 gives the same answers on CPU), one ticket, one four-way choice question, four runs. Latency and benchmark accuracy figures are the authors' published measurements, re-analyzed from their files.