Can LLMs Keep Score? I Benchmarked Tennis and Padel Score-Tracking

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

I wanted a test where the right answer is unambiguous, so I picked something simple that is surprisingly easy to get wrong: keeping score in a match.

Each case gives a model a list of points in order ("A A B A B ...", each letter is who won that point). The model has to report the state right after the last point: sets, games, points in the current game, and who serves next. I wrote a small scoring engine in Python that computes the true answer, so every reply is simply right or wrong. No judge model is involved.

The benchmark covers tennis (advantage scoring) and padel (golden point). It includes sequences of 30, 60 and 100 points, plus special cases forced into a 6-6 tiebreak, where the serving order is awkward. Every point sequence is asked twice: once with the full rules written in the prompt, and once with only "Sport: padel (golden point scoring...)", so the model has to rely on what it already knows. The final benchmark has 48 cases.

Models Tested

I ran it on five models available in Kaggle Benchmarks: Gemini 3.7 Flash, gpt-oss-20b, Claude Haiku 4.5, GPT-5.4 nano, and Gemma 4 26B A4B (this one errored on Kaggle, so it has no score). I chose a mix of a strong fast model, a small reasoning model, and small models that answer quickly, because I expected the score to depend on how much a model works things out step by step. Everything ran with default settings, once per case.

Findings

Score vs. total cost for the three models Kaggle had cost data for (gpt-oss-20b and Gemma 4 are not shown). Claude Haiku 4.5 spent about $0.29 and scored about 0, so spending more tokens didn't help it.

• The spread is huge. Keeping score in a match is easy for a person, and two models got almost none of 48 right. The rough 95% ranges are about 86-99% for Gemini and 29-56% for gpt-oss-20b, so those two do separate clearly.
• Not working it out vs working it out badly. GPT-5.4 nano wrote about 25 tokens per case, just the answer line, and guessed. Some answers were impossible states, such as a match at 2-1 in sets while it was still going. Claude Haiku 4.5 wrote about 1,100 tokens of step-by-step counting per case and still scored about 0. I checked whether this was a formatting problem: every Haiku reply I looked at ended with a correctly formatted answer line that my scorer parsed. The mistakes were in game counts and who serves next. So writing out the counting isn't enough on its own.
• Some of the errors were noise. In an earlier 100-case run, all 5 of Gemini's mistakes were in the no-rules condition, which looked like "the model needs the rules." I reran those 5 cases twice and 9 of 10 passed. With only 10 reruns that is consistent with noise but not proof (the plausible range is roughly 60-98%). Haiku shows the same variation: 0.00 on the leaderboard, 1/48 in my own rerun. Single-run results overstate differences between conditions.
• I did not find a padel weakness or a rules effect. The 48 cases are really 24 point sequences, each asked twice (rules in the prompt, rules not in the prompt), so the two conditions are paired. The right way to compare them is to count the sequences where the answer changed between the two prompts. I had 5 such cases, which isn't enough to claim an effect. Padel and tennis scores for Gemini were within one error of each other.

What I'd Measure Next

• Run each case several times, because one run per case is too noisy to compare conditions.
• Turn on extended reasoning for the models that scored 0.00, to see whether the problem is the model or the default setting.
• Get per-sport and per-length results for every model, not just Gemini.
• Try 200-point sequences, since the best model was close to the ceiling.
• Compare with-rules vs without-rules as paired results (count the sequences whose answer flips) instead of two separate averages.

Limits

• 48 cases (24 point sequences, each asked twice), one run per case, default settings, and only a handful of models. Differences between close results aren't reliable.
• Run-to-run variation is real: Haiku scored 0.00 on the leaderboard and 1/48 in a rerun, and some of Gemini's early errors disappeared on rerun.
• A score of 0/48 only rules out a true rate above roughly 7%.
• I checked the padel rules (golden point, tiebreak serving order, who serves first in the next set) against padel rules sources, but real events sometimes use a super tiebreak in the third set, which this benchmark doesn't model.

My Benchmark

https://www.kaggle.com/benchmarks/gulrezqayyum/tennis-and-padel-state-tracking-benchmark