I spent weeks tuning a local OCR pipeline. Downscaling, adaptive thresholding, per-word confidence filtering, all of it.
A vision model beat it in nine seconds on the same photographs.
Here is the constraint that got me there. I built a reading coach for my two kids, and the reading and the writing do not happen on a screen. The child reads a physical book, writes about the chapter by hand, and photographs the page. That photograph is the system's entire input.
Everything interesting in the project falls out of that one decision.
The OCR. Real photographs of a paperback taken by a child have curved pages, bad light and a thumb in the frame. My tuned pipeline produced gibberish on exactly the input it was built for. The vision model read it cold. The local pipeline is still there as a fallback for when the primary is unreachable, which turns out to be the right place for it.
Refusing to answer is a feature. If handwriting cannot be read with confidence the app does not grade it. It names the words it could not resolve and asks for the page again. An app that guesses at illegible writing and then grades its own guess teaches a child that the machine is arbitrary.
One grader does not fit every subject. Every marker wants the final answer correct, except the Fermi one. That passes anything within a factor of ten and fails a bare number with no working behind it, even when the number is right. I tested it with the correct answer and no reasoning, and it refused it.
The bug I could not have predicted. "Draw the next two steps of the pattern, then take a photo" is itself a reading test. My seven-year-old could not get past it. The software was working perfectly and was completely unusable without an adult sitting next to her. Every instruction now has a read-aloud button.
22 sessions. 169 second median turnaround. 3 books and 20,726 words indexed. 566 generated items across nine subjects.
Top comments (2)
The fallback is the part worth hardening. A pipeline tuned on clean scans degrades on curved pages and bad light, so keeping it as the offline path only works if the system can detect the degradation before returning text: a per-word confidence distribution, or a check that the output contains words the page actually shows. Otherwise the fallback is not a fallback, it is a silent gibberish path that only runs when the primary is down, which is exactly when nobody is watching. Refusing to grade when confidence is low is the same idea applied to the output instead of to the model.
The decision to treat refusing to grade as a primary feature is where most commercial pipelines quietly fail. When a system is penalized for admitting uncertainty, it defaults to manufacturing a plausible point estimate. Having the software name the unresolved words and request a fresh photo preserves the feedback loop, because guessing on blurred characters conceals variance behind false precision.
The Fermi grader rule works on the same principle. In any noisy estimation task, stumbling onto the correct terminal number is cheap. If the pipeline cannot verify the intermediate bounds, a correct result is indistinguishable from lucky variance. Scoring the reasoning rather than the scalar output is what keeps the model anchored to the underlying mechanics.