A Software Engineer who strongly believes you can add immense value by knowingly implementing Software Engineering best practices even at the age of AI, and puts human life and dignity above all else.
This is one of the few DEV challenges that I regret not being able to join myself. Not enough time!
I'd still love to go through the submissions and complete the challenge on my own pace, when I get time 🙏
So, congrats to everyone who participated! 🥰
And thanks to Kaggle and DEV for arranging such challenges.
Software engineer, backend & AI-focused. Node.js, TypeScript, Go, PHP/Laravel, AWS. I write a lot about reliable systems and AI agents that actually ship. More at nazarboyko.com
Thanks a bunch Nazar, congrats on yours too! Clocks went back in a lot of places on Sunday, not Calgary, and all 19 models still have it on standard time 😎
Software engineer, backend & AI-focused. Node.js, TypeScript, Go, PHP/Laravel, AWS. I write a lot about reliable systems and AI agents that actually ship. More at nazarboyko.com
Thanks a bunch DEV team and Kaggle, super appreciate it! Congrats to the other four winners too, really nice seeing everyone test the models against something real.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Civic technologist building AI tools that help systems explain themselves. Clew Suite builder, NACM 2026 speaker. I write about hackathons, product judgment, agent readiness, and learning in public.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The Calgary one is my favourite — 19 out of 19 models getting the time zone wrong because Alberta stopped changing its clocks is exactly the kind of failure a leaderboard would never catch. And the Ariane 5 rebuild flagging SQL injection in code that never touches a database... that's the test-writing instinct I wish more evals had. Do any of the winners plan to keep their benchmarks running as new models ship, or was this a one-shot?
Developer, maker, and perpetual project starter. I build web apps and dev tools mostly, things that I think are really useful and should exist and be simple to use. I also like to experiment with AI.
Hey, I'm Hema 👋
Developer, writer, and creator of Dev Opportunity Radar, a weekly series published every Friday on DEV, helping people discover opportunities they might otherwise miss.
From 20 years in kitchens to crafting multilayered software architectures Self-taught developer specializing in Rust, Python, ML platforms, and complex system design Creator of SecuredMe. securedme.ca
The line about every submission starting from something the author already knew was hard is also how the paid version of this work gets hired. On 7 Nov, 1,199 open roles on the job boards of AI training platforms asked people to create evaluation tasks, benchmarks or test cases, about 30% of the 4,035 roles whose full description could be read, and most of them sit with domain experts rather than generalists. Their median advertised rate was $65/hr, against $40/hr across all the roles read. (Disclosure: those counts come from AITraining.jobs, where I work, which reads the platforms' public boards; they are posted rates, not what anyone earned.)
The Calgary entry shows why the knowledge matters more than the harness: nobody writes that question without knowing Alberta dropped the clock change in June, and its answer key reruns itself from the time zone database.
A question for the winners: how much of your time went into finding the question every model got wrong, and how much into the runner around it? That split decides whether $65 an hour is a fair price for this kind of work.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
Testing models against real mess instead of clean leaderboards is so real. Over at The Printing World, we see this when automated inventory systems try to guess real-world stock. Huge congrats to the winners!
Starting from a specific, known-hard problem is the best way to build a benchmark. The VeriFactu invoice ledger example alone shows how much more practical these are compared to generic leaderboards. Huge congratulations to all the winners!
We're a place where coders share, stay up-to-date and grow their careers.
Congrats to the winners. 🏆🎉
This is one of the few DEV challenges that I regret not being able to join myself. Not enough time!
I'd still love to go through the submissions and complete the challenge on my own pace, when I get time 🙏
So, congrats to everyone who participated! 🥰
And thanks to Kaggle and DEV for arranging such challenges.
Thanks a lot, DEV team and Kaggle! This was such a fun one to build. Congrats to everyone who took part! 🙌
@jonathansolvesstuff The Calgary time zone one made me laugh and then double-check my own assumptions 😄
Thanks a bunch Nazar, congrats on yours too! Clocks went back in a lot of places on Sunday, not Calgary, and all 19 models still have it on standard time 😎
At this point the models aren't wrong, they’re just loyal to old Calgary 😅😅😅
Ahahaha, true that!
Thanks a bunch DEV team and Kaggle, super appreciate it! Congrats to the other four winners too, really nice seeing everyone test the models against something real.
Congrats All!
Congrats to the winners! Great job, absolutely well deserved!👏
Congrats folks!
Congratulations to all the winners 🏆 👏 🙌
Congratulations!!! to all the winners ....
Congrats to all winners; my pleasure to participate and learn!
Nice Congratulations 🎉 😔 @heyitsjem
Congrats to all the winners!! 😸🍾🎉
The Calgary one is my favourite — 19 out of 19 models getting the time zone wrong because Alberta stopped changing its clocks is exactly the kind of failure a leaderboard would never catch. And the Ariane 5 rebuild flagging SQL injection in code that never touches a database... that's the test-writing instinct I wish more evals had. Do any of the winners plan to keep their benchmarks running as new models ship, or was this a one-shot?
Congrats winners! Great job everyone!
Congrats to the Winners! 👏🎉
congrats to all winners. 🎉
Congrats to the winners. 🏆🎉
Congratulations on your winning endeavors .😁
Big congrats to all the winners 😀
congrat to all winner
The line about every submission starting from something the author already knew was hard is also how the paid version of this work gets hired. On 7 Nov, 1,199 open roles on the job boards of AI training platforms asked people to create evaluation tasks, benchmarks or test cases, about 30% of the 4,035 roles whose full description could be read, and most of them sit with domain experts rather than generalists. Their median advertised rate was $65/hr, against $40/hr across all the roles read. (Disclosure: those counts come from AITraining.jobs, where I work, which reads the platforms' public boards; they are posted rates, not what anyone earned.)
The Calgary entry shows why the knowledge matters more than the harness: nobody writes that question without knowing Alberta dropped the clock change in June, and its answer key reruns itself from the time zone database.
A question for the winners: how much of your time went into finding the question every model got wrong, and how much into the runner around it? That split decides whether $65 an hour is a fair price for this kind of work.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
Testing models against real mess instead of clean leaderboards is so real. Over at The Printing World, we see this when automated inventory systems try to guess real-world stock. Huge congrats to the winners!
Starting from a specific, known-hard problem is the best way to build a benchmark. The VeriFactu invoice ledger example alone shows how much more practical these are compared to generic leaderboards. Huge congratulations to all the winners!