mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
235 lines
15 KiB
Markdown
235 lines
15 KiB
Markdown
# Per-language model recommendations: English and European languages
|
|
|
|
Researched 2026-10-22. Builds on `model-watch.md` (parakeet-ultra, Moonshine,
|
|
Nemotron are not re-argued here). Scope: English, the 25 Parakeet v3 languages
|
|
in `PARAKEET_V3_LANGUAGES`, and the two other European entries Hex lists
|
|
(Turkish `tr`, Macedonian `mk`, which fall into the Qwen branch today).
|
|
|
|
## Verdict
|
|
|
|
```ts
|
|
English
|
|
├── Recommended: keep Parakeet Unified. It is the only Hex English model that streams,
|
|
│ and its offline accuracy is at least on par with ultra (FLEURS en 3.99 vs 4.26)
|
|
├── Most accurate: Cohere's lead is real only on LibriSpeech clean (1.27 vs 1.60/1.80/1.91)
|
|
│ and the Open ASR leaderboard (6.26 vs 6.51 v3); on FLEURS en it is the worst of the set (5.08)
|
|
│ └── Qwen3-ASR 1.7B beats it on both FLEURS en (3.23) and the leaderboard (5.75): evaluate as the replacement
|
|
├── parakeet-ultra for English: no. Same size, but offline-only, so it loses the streaming pipeline
|
|
└── Canary 1B v2 for English: no. 4.47 FLEURS, 1.14 GB, half the speed, no streaming
|
|
|
|
European (Parakeet v3 set)
|
|
├── Recommended: parakeet-ultra replaces v3 everywhere except Greek (lower WER in all 25, same size and speed)
|
|
├── Most accurate: Canary 1B v2 where it beats ultra by more than 1.5 points (12 languages)
|
|
├── Greek: Parakeet is broken for it (32 to 35% WER). Use Cohere (8.96) or Whisper turbo (13.26)
|
|
└── Turkish and Macedonian: Whisper turbo beats Qwen3-ASR 0.6B by 2.4x and 2x. Switch Recommended
|
|
```
|
|
|
|
Two findings matter more than the ultra swap:
|
|
|
|
1. **Greek is routed to a model that gets one word in three wrong.** v3 35.33%,
|
|
ultra 32.25%, Canary v2 26.02%, Whisper turbo 13.26%, Cohere 8.96% (FLEURS).
|
|
2. **Turkish and Macedonian are routed to Qwen3-ASR 0.6B** at 16.74% and 35.09%
|
|
against Whisper turbo's 6.97% and 17.85%.
|
|
|
|
## Evidence and its limits
|
|
|
|
Almost all per-language numbers are FLEURS test WER on Q8_0 GGUFs, from the
|
|
model cards in `handy-computer/transcribe.cpp` `docs/models/*.md`, measured by
|
|
the same project with one pipeline. That is the best like-for-like data
|
|
available, but:
|
|
|
|
- **It is not the HF Open ASR Leaderboard.** The multilingual tracks (FLEURS,
|
|
Common Voice, MLS) are not in `hf-audio/open-asr-leaderboard-results`; that
|
|
dataset only publishes `english_short_latest.csv`. So I could not use a
|
|
leaderboard multilingual ranking. English leaderboard numbers below come from
|
|
that CSV (via `model-watch.md`).
|
|
- **FLEURS is ~860 read utterances per language, one run.** Treat gaps under
|
|
about 0.7 points absolute (or 10% relative) as ties.
|
|
- **Canary 1B v2 was measured at an earlier commit (2026-05-08)** than ultra
|
|
(2026-10-05). The runs should be comparable, but I did not rerun them.
|
|
- Greek and Maltese scores depend heavily on text normalization; Whisper's very
|
|
high Maltese error (70.92%) probably includes script and tokenization issues,
|
|
but it is what Hex users would see.
|
|
- Moondream's own card claims 1.41% on LibriSpeech clean for ultra
|
|
(Open ASR normalizer); transcribe.cpp measures 1.80% with kestrel. The
|
|
FLEURS table is transcribe.cpp's, not vendor-only. The 1.41% is vendor-only.
|
|
- Voxtral Mini 3B, Qwen3-ASR 1.7B, Granite 4.1 numbers are likewise
|
|
transcribe.cpp measurements, not vendor claims.
|
|
|
|
## Runtime constraints (transcribe.cpp)
|
|
|
|
```ts
|
|
model license GGUF Q8 speed M4 Max Metal* languages hints streams crate
|
|
Parakeet Unified EN NVIDIA OML 731 MB 188x en none yes 0.2.4
|
|
Parakeet v3 CC-BY-4.0 740 MB 182x 25 EU none no 0.2.4
|
|
parakeet-ultra CC-BY-4.0 740 MB 207x same 25 none no 0.4.0 (see below)
|
|
Canary 1B v2 CC-BY-4.0 1.14 GB 105x same 25 --prefix only no 0.2.4 (arch/canary present)
|
|
Canary 1B flash CC-BY-4.0 - 118x en de es fr prefix no 0.2.4
|
|
Whisper large-v3-turbo MIT 886 MB 36x ~100 (list below) vocabulary+prompt no 0.2.4 (Hex ships it)
|
|
Cohere Transcribe Apache-2.0 2.41 GB 77x 14 (8 are EU) none no 0.2.4 (Hex ships it)
|
|
Qwen3-ASR 1.7B Apache-2.0 ~2 GB 45x 30 context (unverified in Hex) no 0.2.4 (card exists)
|
|
Qwen3-ASR 0.6B Apache-2.0 850 MB 75x 30 none in Hex no 0.2.4
|
|
Voxtral Mini 3B 2507 Apache-2.0 large 13x 7 EU - no 0.2.4 (arch/voxtral)
|
|
Granite Speech 4.1 2B Apache-2.0 - 76x en de es fr pt keyword biasing (upstream) no newer than 0.2.4
|
|
Nemotron 3.5 streaming OpenMDW-1.1 751 MB 144x 32 prompt yes 0.2.4
|
|
```
|
|
|
|
\* jfk 11 s clip, transcribe.cpp cards. Whisper's 36x is the card's number; Hex's
|
|
catalog says ~19x, so Hex's own measurement is more conservative.
|
|
|
|
- **parakeet-ultra needs a newer crate than Hex's pin.** The port landed in
|
|
transcribe.cpp commit `37273409` (2026-10-05), after v0.3.1 (10-04), so
|
|
v0.4.0 (10-12) is the first release that can contain it; 0.2.4's `parakeet`
|
|
arch has no voice-activity segmentation. A 0.4.0+ bump is required for the
|
|
long-audio behavior and is the safe assumption for the GGUF at all. 0.4.1 is
|
|
current. `orukeet` (CC-BY-SA-4.0, a share-alike license; not recommended) and
|
|
`parakeet-primeline` (German) are also parakeet fine-tunes.
|
|
- **Hints.** Only Whisper has vocabulary and prompt biasing in the GGUF path,
|
|
which is why it stays the Recognition hints pick for every language. Canary
|
|
v2 takes a transcript prefix, which is not vocabulary biasing.
|
|
- **Streaming.** No multilingual model is worth streaming. Nemotron 3.5 streams
|
|
but is far worse than ultra (de 10.33 vs 4.58, fr 10.78 vs 4.66, English
|
|
regressed on the leaderboard to 10.51 vs 6.89). Voxtral Realtime is 4.7 GB
|
|
at 6-7x realtime. Multilingual dictation stays non-streaming.
|
|
- **Canary needs an explicit language.** Hex's per-language picker supplies one;
|
|
it must not be offered under Auto.
|
|
- **No GGUF for newer candidates.** Granite Speech 5.0 2B (leaderboard #2,
|
|
5.06) and TurboCTC multilingual revisions are not in transcribe.cpp;
|
|
Kyutai STT (PyTorch/MLX) and TheWhisper have no GGUF. Excluded.
|
|
|
|
## English
|
|
|
|
```ts
|
|
FLEURS en LS clean Open ASR English avg (current)
|
|
Qwen3-ASR 1.7B 3.23 - 5.75
|
|
Voxtral Mini 3B 3.89 - -
|
|
Parakeet Unified EN 3.99 1.60 (Hex) - (8-set offline 5.91 older board)
|
|
Granite 4.1 2B 4.14 - -
|
|
parakeet-ultra 4.26 1.80 - (older board 5.32 vs v3 6.32, from model-watch)
|
|
Whisper turbo 4.38 - 8.39
|
|
Canary 1B v2 4.47 1.91 -
|
|
Parakeet v3 4.83 1.94 6.51
|
|
Cohere Transcribe 5.08 1.27 6.26
|
|
```
|
|
|
|
Current: Recommended Parakeet Unified, Most accurate Cohere, Hints Whisper.
|
|
Proposed: Recommended Unified (keep), Most accurate Qwen3-ASR 1.7B (medium
|
|
confidence) or keep Cohere until Hex's own English corpus is run, Hints Whisper
|
|
(keep). Fastest is already Unified/ultra territory; no separate row is needed.
|
|
|
|
- Unified vs ultra: ultra is likely a bit better offline on mixed data (older
|
|
eight-set average), but FLEURS en favors Unified and only Unified streams.
|
|
Hex's capture pipeline is built around streaming with a whole-clip fallback
|
|
(AGENTS.md), so switching English to an offline-only model costs latency for
|
|
no demonstrated accuracy gain. Keep Unified. Offer ultra only as an optional
|
|
offline alternative if English users want its long-audio segmentation.
|
|
- Canary v2 and Voxtral 3B do not beat Unified on English. Not proposed.
|
|
- Qwen3-ASR 1.7B: the best English FLEURS and best current-leaderboard open
|
|
model Hex could run (Apache-2.0, 45x on M4 Max). It costs a ~2 GB download and
|
|
no streaming, the same trade Cohere already makes. Confidence medium: the
|
|
FLEURS and leaderboard numbers point the same way, but no one has run it on
|
|
Hex's dictation audio, and Hex marks Qwen as having no recognition hints.
|
|
- Cohere's "most accurate" label rests on LibriSpeech clean (read audiobooks)
|
|
and a 0.25-point leaderboard edge over v3; its FLEURS en is the worst here.
|
|
Medium-low confidence in keeping that label unchanged.
|
|
|
|
## Per-language table (FLEURS test WER %, Q8_0)
|
|
|
|
Columns: ultra, v3, Canary 1B v2 (cv2), Whisper turbo (wt), plus the best
|
|
other available model. Best-in-row is bold. Current for every language below is
|
|
"Recommended v3, Hints Whisper turbo" unless stated.
|
|
|
|
```ts
|
|
lang ultra v3 cv2 wt other proposed (Recommended / Most accurate / Hints) conf
|
|
bg 11.04 12.81 9.22 13.58 - ultra / Canary v2 (-1.8) / Whisper medium
|
|
cs 11.04 12.31 8.56 11.81 Qwen1.7 22.97 ultra / Canary v2 (-2.5) / Whisper medium
|
|
da 15.98 18.64 11.34 13.60 - ultra / Canary v2 (-4.6) / Whisper high
|
|
de 4.58 5.24 4.46 4.54 Qwen1.7 4.25 ultra / (tie, none) / Whisper medium
|
|
el 32.25 35.33 26.02 13.26 Cohere 8.96 Cohere / Whisper / Whisper (hints) high
|
|
en see English section above
|
|
es 3.06 3.65 3.10 3.12 Qwen1.7 3.31 ultra / (tie, none) / Whisper high
|
|
et 14.44 17.96 12.72 18.44 - ultra / Canary v2 (-1.7) / Whisper medium
|
|
fi 11.14 13.30 8.86 8.29 - Whisper / Canary v2 / Whisper (hints) medium
|
|
fr 4.66 5.30 5.09 5.51 Voxtral3B 4.51 ultra / (tie, none) / Whisper medium
|
|
hr 11.30 12.59 8.40 12.54 - ultra / Canary v2 (-2.9) / Whisper medium
|
|
hu 13.09 16.06 13.06 15.07 - ultra / (tie, none) / Whisper medium
|
|
it 2.34 3.02 3.10 2.77 Voxtral3B 2.56 ultra / (none) / Whisper high
|
|
lt 17.06 22.20 13.45 25.11 - ultra / Canary v2 (-3.6) / Whisper medium
|
|
lv 18.04 23.77 10.41 19.53 - Canary v2 / ultra / Whisper high
|
|
mt 17.01 20.63 19.75 70.92 - ultra / (none) / (no hints: Whisper unusable) medium
|
|
nl 6.70 7.66 6.28 5.98 Voxtral3B 6.57 ultra / (tie, none) / Whisper medium
|
|
pl 6.16 7.37 6.88 5.81 Cohere 6.15 ultra / (tie, none) / Whisper medium
|
|
pt 4.16 4.96 4.50 4.17 Voxtral3B 3.84 ultra / (tie, none) / Whisper medium
|
|
ro 10.12 12.62 6.87 10.90 - ultra / Canary v2 (-3.3) / Whisper high
|
|
ru 5.63 6.54 7.83 5.93 Qwen1.7 6.25 ultra / (none) / Whisper high
|
|
sk 7.83 10.19 6.84 10.21 - ultra / Canary v2 (-1.0, borderline) / Whisper low
|
|
sl 18.98 24.30 12.76 20.56 - Canary v2 / ultra / Whisper high
|
|
sv 12.83 15.25 9.74 8.72 - Whisper / Canary v2 / Whisper (hints) medium
|
|
uk 5.71 6.84 10.58 7.31 - ultra / (none) / Whisper high
|
|
tr - - - 6.97 Qwen1.7 9.46 Whisper / (none) / Whisper high
|
|
mk - - - 17.85 Qwen1.7 18.22 Whisper / (none) / Whisper medium
|
|
```
|
|
|
|
`tr` and `mk` currently get Qwen3-ASR 0.6B as Recommended (16.74 and 35.09);
|
|
Qwen3-ASR 1.7B is nearly as good as Whisper there, but Whisper already ships
|
|
and also offers hints.
|
|
|
|
Reading the table:
|
|
|
|
- **ultra vs v3.** Lower in all 25 languages, by 0.5 points on the strong
|
|
Romance languages up to 5 to 6 points on lv, sl, lt, bg, da. Same size,
|
|
languages and API, so the swap is a catalog entry plus the crate bump. Keep
|
|
v3 only as a fallback if the crate bump slips. High confidence for the swap
|
|
(every language moves the same direction, and the card is a like-for-like run
|
|
of the same pipeline).
|
|
- **Canary v2 as "Most accurate"** (CC-BY-4.0, 1.14 GB, 105x on M4 Max, still
|
|
fast): clear wins on bg cs da et hr lt lv ro sk sl, and on fi and sv close to
|
|
Whisper. It is not better on de es fr it nl pl pt ru uk. It adds an explicit
|
|
language requirement, no timestamps, and no vocabulary biasing.
|
|
- **Whisper turbo** wins or ties on fi, nl, pl, sv, tr, mk and is the only
|
|
hint-capable model, but it is the slowest (36x) and collapses on mt (70.92),
|
|
lv, lt, sl, et. For those languages Hex should not advertise it as a general
|
|
option; the Recognition hints row is a trade, not a recommendation.
|
|
- **Qwen3-ASR 0.6B is poor outside the big languages** (cs 44.50, fi 46.49,
|
|
hu 56.24); it should not be Recommended for any European language.
|
|
- **Cohere** covers de es fr it nl pl pt el in Hex, but only wins on Greek.
|
|
Its FLEURS (de 5.06, fr 5.23, es 3.97) is behind ultra.
|
|
|
|
## What to change in `choices_for_runtime`
|
|
|
|
1. Add parakeet-ultra to `MODELS` and make it Recommended in the Parakeet branch
|
|
(requires bumping `transcribe-cpp` past 0.4.0, and the Nix `outputHashes`
|
|
if it touches the build). Keep v3 out of the picker or as a fallback.
|
|
2. Add Canary 1B v2 as Most accurate for bg cs da et hr lt lv ro sk sl (and
|
|
possibly fi sv). Never under Auto.
|
|
3. Route `el` away from Parakeet: Cohere Recommended (add `el` to its branch
|
|
before the Parakeet match), Whisper for hints.
|
|
4. Route `tr` and `mk` to Whisper turbo as Recommended instead of Qwen 0.6B.
|
|
5. English: keep Unified; evaluate Qwen3-ASR 1.7B against Cohere on Hex's own
|
|
English clips before changing the Most accurate label.
|
|
|
|
## Not measured
|
|
|
|
- No local runs: nothing here was rerun on Kit's hardware, and Apple Silicon
|
|
speeds are the transcribe.cpp cards' M4 Max numbers.
|
|
- Real dictation (noisy, short, accented) is not FLEURS. A cheap check is the
|
|
`benchmark-transcription` corpus per language for the changed rows only.
|
|
- Parakeet ultra on 0.2.4 was not tried; the 0.4.0 requirement is inferred from
|
|
dates and the absent VAD code in the 0.2.4 sources.
|
|
- Qwen3-ASR 1.7B Q8 size and Canary v2's behavior under Hex's language-hint
|
|
plumbing need a check before adding.
|
|
|
|
## Sources
|
|
|
|
- transcribe.cpp (MIT) `docs/models/`: `parakeet-ultra`, `parakeet-tdt-0.6b-v3`,
|
|
`parakeet-unified-en-0.6b`, `canary-1b-v2`, `canary-1b-flash`,
|
|
`whisper-large-v3-turbo`, `cohere-transcribe-03-2026`, `qwen3-asr-0.6b`,
|
|
`qwen3-asr-1.7b`, `voxtral-mini-3b-2507`, `granite-speech-4.1-2b`,
|
|
`nemotron-3.5-asr-streaming-0.6b`, `orukeet`, `parakeet-primeline`;
|
|
releases v0.2.4 (2026-09-25), v0.3.1 (10-04), v0.4.0 (10-12), v0.4.1 (10-22);
|
|
commit `37273409` "parakeet ultra" (2026-10-05).
|
|
- Hex's pinned `transcribe-cpp-sys` 0.2.4 sources (`src/arch/`).
|
|
- HF `hf-audio/open-asr-leaderboard-results` (`english_short_latest.csv`) and the
|
|
`huggingface/open_asr_leaderboard` README (multilingual benchmark description).
|
|
- `model-watch.md` for the English leaderboard rows and the ultra, Unified,
|
|
Nemotron, and Moonshine analysis.
|