DEV Community

nan nan
nan nan

Posted on

What I learned building a fully offline multi-region OCR tool with PP-OCRv6 and ONNX Runtime

The problem I kept running into
PowerToys Text Extractor (Win+Shift+T) is genuinely good at what it does: grab a block of on-screen text, put it in the clipboard, move on. But it returns a plain text stream, one region at a time. When I needed to pull text from five different parts of a UI and preserve which text came from where, I ended up doing the structuring by hand.


That's the gap I wanted to close: not “can it extract text,” but “what does the output look like when you capture multiple regions and need structured data.”

Why PP-OCRv6, and why offline matters here
I chose PP-OCRv6 because it's a small, Apache-2.0 licensed model family that runs comfortably on CPU via ONNX Runtime. The tiny tier is about 6.6 MB — small enough to bundle inside the MSIX package, which means OCR works on first launch without a download. The accurate tier is about 29.8 MB and downloads once.

The offline constraint wasn't about privacy marketing. It was a latency decision. Every round trip to a cloud OCR API adds network variance to an interaction that should feel instantaneous. Running locally means the bottleneck is model inference, not bandwidth.

What the multi-region workflow actually looks like
After capturing a region, the overlay stays open. You hold Ctrl and drag another region, and it gets auto-numbered. Each region lands in a side drawer with a thumbnail and recognition status. From the drawer you can reorder, delete, undo, and batch-export.

The export contract was the most opinionated part. A Markdown table row carries: index, x, y, w, h, confidence, text. CSV has the same columns plus line_index, mode, elapsed_ms, model_hash. The goal is that the output is immediately usable in an issue tracker, a documentation page, or a spreadsheet — not something you have to reformat first.

The awkward parts
Two things I haven't fully solved:

Tiny tier excludes Japanese. PP-OCRv6 tiny covers 49 languages but not Japanese. I detect Japanese characters and prompt the user to switch to the accurate tier, but it's a visible seam in the experience.

Scrolling capture is still on the roadmap. It's the most requested missing feature in my own testing, but the success rate varies too much across apps for me to ship it as a headline feature yet.

What I'd like feedback on
The drawer interaction is the part I'm least confident about. Does a persistent side drawer feel right for a capture overlay, or would an inline list closer to the captured regions be more natural? I'm also curious whether the Markdown table schema is missing anything you'd need for a real documentation workflow.

The tool is on the Microsoft Store ($2.99 one-time). Source repo and product page are linked below.

Store: https://apps.microsoft.com/detail/9pbp8vbg4rj1

Site: https://lijiannan1241.github.io/CaptureOCR/

Top comments (3)

Collapse
 
nan_nan_d86f644d31091c00f profile image
nan nan •

CaptureOCR is a Windows screenshot OCR workflow for office, research and support teams: capture multiple regions in one session, annotate and beautify them, extract Chinese/mixed-language text locally, and export structured Markdown/CSV—unlike single-shot screen OCR, with no subscription.

Collapse
 
merbayerp profile image
Mustafa ERBAY •

Really interesting approach! 👏

I particularly like that you’re treating OCR output as structured data rather than just extracted text. Including coordinates, confidence scores, and model hashes makes the results much more useful for automation and documentation.

Personally, I’d prefer the persistent drawer for managing multiple regions, especially when reordering or reviewing results.

One suggestion: consider adding a stable region ID and an explicit reading-order field to the export schema. That could make downstream processing more reliable when regions are rearranged.

Also, have you considered preserving the original image dimensions or DPI alongside the coordinates? That would help when mapping OCR results back to screenshots.

Nice work, and thanks for sharing the engineering decisions behind it! 😊

Collapse
 
nan_nan_d86f644d31091c00f profile image
nan nan •

Good job. This is an excellent suggestion; the blogger will give it serious consideration and adopt it.