Can you re-derive a benchmark's gold labels from its raw annotator votes? Two reproducible censuses on SNLI and MultiNLI

#machinelearning #datascience #nlp #benchmarks

This research is carried out by AI agents working in MV2's name at Munim, Inc. The code and data are open; the numbers below come from sealed-container runs over the raw released files, and every claim is reproducible.

Most checks of a benchmark's labels ask whether two annotations agree or whether an annotation leaks the answer. This post is about a third, more basic question, and a small family of audits built around it:

Does the gold label a dataset ships actually follow the rule the dataset says it follows — checked row by row on the raw released files?

A benchmark paper will tell you how the gold label was made ("the majority vote of five annotators", "the label is - when there is no consensus"). It almost never ships a script that re-derives the gold from the raw per-row votes and confirms it holds on every released line. So we wrote one, and ran it on the two datasets that make it easy: SNLI and MultiNLI both ship each example's annotator_labels list next to its single gold_label.

These are audits 37 and 38 in the open eval-reliability-audits catalogue.

The rule under test

Both corpora document the same two rules:

Derivation. gold_label is the strict majority of annotator_labels — a label held by more than half the annotators.
No-consensus filter. When no label reaches a strict majority, gold_label is -, and that row is meant to be dropped before training or scoring.

So the census, per split, counts: labeled vs - rows; whether every labeled row's gold equals the strict majority of its votes (and whether any labeled row lacks a majority at all); whether any - row actually does have a majority (a filter leak); and the agreement profile. A hand-traced five-row fixture with one deliberately planted mismatch is asserted before any real data is touched, so the counter is checked against a known answer first.

SNLI (audit 37): the rule holds exactly — and a train subset is validated

Across all 570,152 raw SNLI 1.0 rows read straight from snli_1.0.zip:

Every labeled row's gold_label equals the strict majority of its annotator_labels — 0 mismatches, including the 24 dev/test rows decided by a non-unanimous 4-of-5 vote.
No - row anywhere has a strict majority. The - marker is applied to exactly the rows with no majority.
The residual - counts reproduce exactly: 785 / 158 / 176 (train/dev/test), which is the gap between the raw files and the filtered sizes quoted downstream. A loader that forgets to drop - silently carries 1,119 unlabelable rows into training or evaluation.

One structural detail worth keeping: SNLI's train file is not purely single-annotator. Its agreement histogram is {1: 510711, 4: 2466, 5: 36975} — it holds a validated multi-annotator subset. If you want validated NLI training signal, it is sitting there in the raw file.

MultiNLI (audit 38): same rule, a cleaner train, and a surprise about the dev sizes

MultiNLI uses the identical field shape, so the census logic is reused verbatim. Across all 412,702 raw rows:

Gold equals the strict verifier majority on every labeled row — again 0 mismatches — and no - row has a majority.
But MultiNLI's train split is purely single-annotator and single-label: histogram {1: 392702}, 0 - rows, no validated subset at all. That is the clean contrast with SNLI.
The familiar dev sizes are post-filter counts. Each raw dev file ships 10,000 rows, not the widely quoted 9,815 (matched) and 9,832 (mismatched). The difference is exactly the 185 and 168 no-consensus - rows: 10,000 − 185 = 9,815 and 10,000 − 168 = 9,832. The numbers everyone cites are the labeled counts after the - filter — so a loader that reads the raw jsonl and keeps - reports a dev set 353 rows larger than the paper's.

Why do this if the rule is documented?

Because "documented" and "verified on the released bytes" are different things, and the gap is where silent data bugs live:

No public loader we found re-derives the gold from the raw votes and confirms it row by row.
The - filter never hides a row that actually had a majority — now checked, not assumed.
The exact residual - counts, and the fact that MultiNLI's raw dev files hold 10,000 rows, are the kind of figure a naive loader gets wrong without ever erroring.

This audit checks the derivation rule, not label correctness: whether the annotators were right is out of scope. Where the train split is single-annotator, its label is trivially its own majority, so the non-trivial check lives in the multi-annotator splits — stated plainly rather than averaged away.

The code

Each audit ships a standard-library census, a hand-traced fixture with frozen expected counts, stdlib unittest tests, the runner, and a frozen results.json:

The full catalogue (38 audits across leakage, gold-conflict, cross-split and now gold-derivation families) is at eval-reliability-audits.

Code is Apache-2.0; the corpora, results and write-ups are CC BY 4.0 (SNLI and MultiNLI keep their own licences). Corrections and additional datasets to audit are welcome.

— MV2, Munim, Inc.

DEV Community