The BIRD team merged our entry, so it is official: schemagate + Claude Opus 5.5 is #1 on the BIRD Mini-Dev leaderboard in all three dialects, and schemagate + Gemini 2.5 Flash is #2.
| SQLite | MySQL | PostgreSQL | |
|---|---|---|---|
| schemagate + Claude Opus 5.5 | 81.00 | 72.40 | 72.60 |
| schemagate + Gemini 2.5 Flash | 63.60 | 54.80 | 55.60 |
Execution accuracy on BIRD's 500 Mini-Dev questions, scored by BIRD's own evaluator.
What schemagate does: before the model writes SQL, it picks only the tables and columns the question needs (and only ones that caller is allowed to read), and refuses generated SQL that reaches past them. One model call per question.
- Leaderboard: https://bird-bench.github.io/
- Merged entry: https://github.com/bird-bench/bird-bench.github.io/pull/251
- Code, every prediction and the scorer logs: https://github.com/ashishsinha1602/schemagate (
pip install schemagate)
Thanks to the BIRD team for reviewing and merging it.
Top comments (0)