On-device streaming ASR in 2026: five approaches and when to pick what

Why this topic right now

In 2023 there were two options. Take OpenAI Whisper and live with offline batch inference (3+ GB large-v3, not streaming), or wait for cloud ASR with its privacy, cost, and latency trade-offs. On-device streaming ASR as a production option was fringe.

In 2026 there are at least five viable approaches on the table, each optimized for a different use case and different constraints. This post is an ecosystem overview and decision guide, not a head-to-head benchmark with a single winner.

What is being compared, and what is not

In scope:

• Streaming. The model emits partial transcripts incrementally, not waiting for end of utterance. First-partial latency measured in hundreds of milliseconds.
• On-device. Inference runs locally, no network round-trip, no cloud dependency.
• Production-ready. Shipping in real mobile and edge apps today, not research prototypes.
• Five options that meet all of these criteria.

Out of scope:

• Cloud ASR APIs (Deepgram, Whisper API, Google USM via API). Different trade-off entirely.
• Batch-only on-device models (Whisper offline without a streaming wrapper). Not streaming.
• Wake-word and keyword spotting. Narrower task scope, different SDKs.
• ASR at research stage without a production SDK.
• A few worth naming but excluded for specific reasons: NVIDIA Riva (Jetson-tier and server, not mobile-first), Moonshine from Useful Sensors (recent 2024 release, short production track record), SeamlessStreaming from Meta (speech translation primary, not pure ASR). All legitimate for adjacent use cases.

Methodology for the numbers:
All size, platform, and license claims come from each vendor's own published sources (model cards, release artifacts, official docs). There are no cross-benchmarks of our test subset against someone else's full set. Apples-to-apples comparison is only honest when the same test methodology is applied to every cell. Where specific accuracy numbers are needed, the post names the test set and whose claim it is.

The five options

1. VoxRT

• Model size: 60.4 MB (verified against the voxrt-asr-models v0.1.2 release artifact)
• Architecture: NeMo FastConformer family, RNN-T decoding
• Platforms: iOS, Android, Linux (aarch64, tested on Raspberry Pi)
• License: hybrid. Apache-2.0 on the SDK wrapper, proprietary runtime binary, CC-BY-4.0 for model weights.
• Language: English, open-vocabulary
• Streaming: native, 200 ms chunk size (3200 samples at 16 kHz)
• Source: closed runtime, open SDK wrappers
• Repos: voxrt-asr-linux, voxrt-asr-ios, voxrt-asr-android

Shipping since 2026, targeting iOS, Android, and Linux (no desktop or browser builds today). The hybrid license covers three layers. Apache-2.0 on the wrapper SDK means you can fork and modify integration code. The proprietary runtime binary cannot be audited, forked, patched, or deployed outside VoxRT's distribution. CC-BY-4.0 model weights can be redistributed with attribution.

2. Picovoice Cheetah

• Model size: compact (vendor does not publish an exact figure)
• Architecture: end-to-end neural network (closed-source, specifics not published)
• Platforms: iOS, Android, Linux, macOS, Windows, Raspberry Pi, WebAssembly
• License: commercial. Free tier for personal and non-commercial use, paid tier for production commercial use.
• Language: 8 languages per vendor docs (English, French, German, Italian, Japanese, Korean, Spanish, Portuguese)
• Streaming: native
• Source: closed-source, commercial product
• Site: picovoice.ai/platform/cheetah

Picovoice is a commercial on-device voice AI vendor with wide platform coverage. The company separately offers microcontroller support for wake-word and keyword spotting (Porcupine and Rhino SDKs), but Cheetah streaming ASR itself targets mobile, desktop, Raspberry Pi, and web. Business model is SaaS-style licensing with a free tier for exploration.

3. sherpa-onnx (k2-fsa project)

• Model size: roughly 40 MB (20M-param streaming Zipformer int8) to 340 MB (full fp32), depending on model configuration and quantization
• Architecture: streaming Zipformer, deployed via ONNX runtime
• Platforms: Linux, macOS, Windows, Android, iOS, HarmonyOS, embedded (very wide)
• License: Apache-2.0 (code and most models)
• Language: multi-language. English, Chinese, and many others depending on the model.
• Streaming: native. Zipformer was designed specifically for streaming.
• Source: open-source
• Repo: github.com/k2-fsa/sherpa-onnx

Part of the broader k2-fsa project (Next-generation Kaldi). Very active open-source community under the k2-fsa umbrella. The model zoo keeps growing, with streaming Zipformer models for different languages and different sizes. Using ONNX runtime for inference keeps deployment portable.

4. Vosk

• Model size: small models around 40-50 MB (vosk-model-small-en-us-0.15 = 40 MB), standard models up to ~4.4 GB (largest is vosk-model-de-tuda-0.6-900k) per the Vosk models page
• Architecture: Kaldi-based (TDNN acoustic model plus n-gram or RNNLM language model)
• Platforms: Python, Node.js, Java, Android, iOS, C#, Go, C++ (wide binding support)
• License: Apache-2.0
• Language: 20+ languages with pretrained models
• Streaming: native
• Source: open-source
• Site: alphacephei.com/vosk

Long-running open-source project (Alpha Cephei). Kaldi at the core is proven technology, with pre-conformer architecture (TDNN plus HMM-style decoding). One of the smallest production-ready options in this list through the 40 MB small model.

5. whisper.cpp (with the streaming example)

• Model size: tiny 75 MB, base 142 MB, small 466 MB, medium 1.5 GB, large 2.9+ GB (per the OpenAI Whisper README)
• Architecture: OpenAI Whisper (encoder-decoder transformer), ported to C/C++
• Platforms: Linux, macOS, Windows, iOS, Android, Web (via WASM)
• License: MIT (whisper.cpp code), MIT (Whisper weights from OpenAI)
• Language: 99 languages (multilingual Whisper models)
• Streaming: not native. Whisper is a batch model, and whisper.cpp has a stream example built on a sliding-window pattern.
• Source: open-source
• Repo: github.com/ggml-org/whisper.cpp

An important caveat. Whisper is architecturally a batch model that processes 30-second windows end-to-end. The streaming example in whisper.cpp creates streaming-like behavior through overlapping sliding windows, but first-partial latency is higher than for models designed around streaming, and accuracy is reported to degrade on short chunks. For use cases where accuracy is critical and latency is secondary, this is a valid approach. For real-time dictation with a <500 ms first-partial requirement, the streaming-first models in items 1-4 above are a closer fit.

Comparison matrix

Platform notes: Picovoice separately offers MCU support (Cortex-M) for wake-word and keyword spotting (Porcupine, Rhino), not for Cheetah streaming ASR. sherpa-onnx supports a wide embedded range including RISC-V.

Decision tree by use case

Smallest model size matters above all other considerations:
→ Vosk small (40 MB) or Picovoice Cheetah (compact). Trade-off: Vosk is pre-conformer accuracy, Picovoice is commercial license.

Open-source plus production-ready streaming plus platform maturity:
→ sherpa-onnx (Zipformer well-supported, large community) or Vosk (older but stable).

Multi-language from day one:
→ whisper.cpp (99 languages) or Vosk (20+). VoxRT and Picovoice are English-focused today.

Target is embedded or microcontroller:
→ For embedded Linux (ARM, RISC-V), sherpa-onnx has the widest support. For microcontrollers (Cortex-M), full ASR is not realistic today. Picovoice's Porcupine/Rhino offer wake-word and keyword spotting at that tier, which is a different task than streaming ASR.

Mobile voice input, English, want a simple SDK integration and streaming-first behavior:
→ VoxRT or Picovoice Cheetah. Trade-off: VoxRT has a hybrid license (proprietary runtime), Picovoice uses a commercial subscription.

Whisper quality required, latency secondary:
→ whisper.cpp (streaming is a sliding-window pattern but accuracy is preserved).

Browser inference (WASM):
→ whisper.cpp (WASM supported) or sherpa-onnx (if the WASM path works for your model configuration, verify before integrating). VoxRT does not ship ASR for the browser today.

Nothing fits cleanly:
→ Download your top two candidates based on size, license, and platform fit. Run them on your actual audio data. Measure RTF and accuracy. Decide on measured numbers, not predicted ones.

Honest trade-offs. Where someone else beats us.

I work on VoxRT, but a fair comparison needs an honest picture of where we do not win:

• Vosk small is smaller than our model (around 40 MB versus our 60 MB). If bytes-on-disk is the primary constraint, Vosk small wins on that axis.
• sherpa-onnx has wider platform coverage for embedded Linux targets (ARM, RISC-V). Browser (WASM) is also not something we ship. For those targets we are not an option.
• sherpa-onnx has significantly wider language coverage. Chinese, Japanese, European languages with active model development. We are English-only.
• whisper.cpp handles 99 languages through OpenAI multilingual models. For truly multilingual products, the Whisper family is still the baseline.
• sherpa-onnx and Vosk are fully open-source (Apache-2.0). If your organization will not accept a proprietary runtime binary, our hybrid license does not fit.
• Picovoice has a longer track record as an established commercial vendor. For teams that want a paid support contract and SLA, Picovoice fits traditional procurement better.
• VoxRT is less than a year old. Picovoice has multi-year production use, Vosk and whisper.cpp are each years into shipping. If track record is a procurement gate, we lack it today.
• We do not publish WER numbers currently. Picovoice, Vosk, and sherpa-onnx all publish WER evaluations on standard benchmarks. If your decision gates on specific accuracy numbers from the vendor, that absence is a real gap for us.
• Closed runtime means you cannot patch bugs, fork for custom modifications, or audit the inference path. For teams that require full-source transparency, our hybrid license disqualifies us.

VoxRT was built around a specific trade-off: streaming-first behavior, mobile focus, hybrid license acceptable, English voice input. For that slice we think the engineering is sound. For other slices, the options above are a better fit.

The ecosystem is healthier than it was before 2024

Three years ago this would have been a much shorter post. Whisper had just shipped (2022), on-device streaming ASR was at research stage, Vosk was the best open-source option but had a visible accuracy gap versus cloud, and Picovoice was the commercial choice but with narrower vocabulary coverage.

In 2026 there are five production-ready options, with different architectural choices, different license models, and different target hardware. The one-size-fits-all approach does not work anymore because use cases are not one size.

Pick based on actual constraints:

• License tolerance (open-only? hybrid acceptable? commercial subscription fine?)
• Platform requirements (mobile only? MCU too? browser?)
• Language requirements (English only? multi-language needed?)
• Model size budget (bytes on disk? runtime memory?)
• Latency requirements (real-time dictation? post-utterance OK?)
• Support expectations (self-service open-source? paid vendor contract?)

All five options in this post are production-ready for shipping in 2026. If you cannot pick between options, download your top two candidates by size, license, and platform fit, and run them on your real data. Measure. Decide on the numbers, not the marketing (including mine).