hex/docs/research/language-recommendations-european.md

15 KiB

Per-language model recommendations: English and European languages

Researched 2026-10-22. Builds on model-watch.md (parakeet-ultra, Moonshine, Nemotron are not re-argued here). Scope: English, the 25 Parakeet v3 languages in PARAKEET_V3_LANGUAGES, and the two other European entries Hex lists (Turkish tr, Macedonian mk, which fall into the Qwen branch today).

Verdict

English
├── Recommended: keep Parakeet Unified. It is the only Hex English model that streams,
│   and its offline accuracy is at least on par with ultra (FLEURS en 3.99 vs 4.26)
├── Most accurate: Cohere's lead is real only on LibriSpeech clean (1.27 vs 1.60/1.80/1.91)
│   and the Open ASR leaderboard (6.26 vs 6.51 v3); on FLEURS en it is the worst of the set (5.08)
│   └── Qwen3-ASR 1.7B beats it on both FLEURS en (3.23) and the leaderboard (5.75): evaluate as the replacement
├── parakeet-ultra for English: no. Same size, but offline-only, so it loses the streaming pipeline
└── Canary 1B v2 for English: no. 4.47 FLEURS, 1.14 GB, half the speed, no streaming

European (Parakeet v3 set)
├── Recommended: parakeet-ultra replaces v3 everywhere except Greek (lower WER in all 25, same size and speed)
├── Most accurate: Canary 1B v2 where it beats ultra by more than 1.5 points (12 languages)
├── Greek: Parakeet is broken for it (32 to 35% WER). Use Cohere (8.96) or Whisper turbo (13.26)
└── Turkish and Macedonian: Whisper turbo beats Qwen3-ASR 0.6B by 2.4x and 2x. Switch Recommended

Two findings matter more than the ultra swap:

  1. Greek is routed to a model that gets one word in three wrong. v3 35.33%, ultra 32.25%, Canary v2 26.02%, Whisper turbo 13.26%, Cohere 8.96% (FLEURS).
  2. Turkish and Macedonian are routed to Qwen3-ASR 0.6B at 16.74% and 35.09% against Whisper turbo's 6.97% and 17.85%.

Evidence and its limits

Almost all per-language numbers are FLEURS test WER on Q8_0 GGUFs, from the model cards in handy-computer/transcribe.cpp docs/models/*.md, measured by the same project with one pipeline. That is the best like-for-like data available, but:

  • It is not the HF Open ASR Leaderboard. The multilingual tracks (FLEURS, Common Voice, MLS) are not in hf-audio/open-asr-leaderboard-results; that dataset only publishes english_short_latest.csv. So I could not use a leaderboard multilingual ranking. English leaderboard numbers below come from that CSV (via model-watch.md).
  • FLEURS is ~860 read utterances per language, one run. Treat gaps under about 0.7 points absolute (or 10% relative) as ties.
  • Canary 1B v2 was measured at an earlier commit (2026-05-08) than ultra (2026-10-05). The runs should be comparable, but I did not rerun them.
  • Greek and Maltese scores depend heavily on text normalization; Whisper's very high Maltese error (70.92%) probably includes script and tokenization issues, but it is what Hex users would see.
  • Moondream's own card claims 1.41% on LibriSpeech clean for ultra (Open ASR normalizer); transcribe.cpp measures 1.80% with kestrel. The FLEURS table is transcribe.cpp's, not vendor-only. The 1.41% is vendor-only.
  • Voxtral Mini 3B, Qwen3-ASR 1.7B, Granite 4.1 numbers are likewise transcribe.cpp measurements, not vendor claims.

Runtime constraints (transcribe.cpp)

model                    license        GGUF Q8   speed M4 Max Metal*   languages           hints            streams   crate
Parakeet Unified EN      NVIDIA OML     731 MB    188x                  en                  none             yes       0.2.4
Parakeet v3              CC-BY-4.0      740 MB    182x                  25 EU               none             no        0.2.4
parakeet-ultra           CC-BY-4.0      740 MB    207x                  same 25             none             no        0.4.0 (see below)
Canary 1B v2             CC-BY-4.0      1.14 GB   105x                  same 25             --prefix only    no        0.2.4 (arch/canary present)
Canary 1B flash          CC-BY-4.0      -         118x                  en de es fr         prefix           no        0.2.4
Whisper large-v3-turbo   MIT            886 MB    36x                   ~100 (list below)   vocabulary+prompt no       0.2.4 (Hex ships it)
Cohere Transcribe        Apache-2.0     2.41 GB   77x                   14 (8 are EU)       none             no        0.2.4 (Hex ships it)
Qwen3-ASR 1.7B           Apache-2.0     ~2 GB     45x                   30                  context (unverified in Hex)  no  0.2.4 (card exists)
Qwen3-ASR 0.6B           Apache-2.0     850 MB    75x                   30                  none in Hex       no        0.2.4
Voxtral Mini 3B 2507     Apache-2.0     large     13x                   7 EU                 -                no        0.2.4 (arch/voxtral)
Granite Speech 4.1 2B    Apache-2.0     -         76x                   en de es fr pt      keyword biasing (upstream)  no   newer than 0.2.4
Nemotron 3.5 streaming   OpenMDW-1.1    751 MB    144x                  32                  prompt           yes       0.2.4

* jfk 11 s clip, transcribe.cpp cards. Whisper's 36x is the card's number; Hex's catalog says ~19x, so Hex's own measurement is more conservative.

  • parakeet-ultra needs a newer crate than Hex's pin. The port landed in transcribe.cpp commit 37273409 (2026-10-05), after v0.3.1 (10-04), so v0.4.0 (10-12) is the first release that can contain it; 0.2.4's parakeet arch has no voice-activity segmentation. A 0.4.0+ bump is required for the long-audio behavior and is the safe assumption for the GGUF at all. 0.4.1 is current. orukeet (CC-BY-SA-4.0, a share-alike license; not recommended) and parakeet-primeline (German) are also parakeet fine-tunes.
  • Hints. Only Whisper has vocabulary and prompt biasing in the GGUF path, which is why it stays the Recognition hints pick for every language. Canary v2 takes a transcript prefix, which is not vocabulary biasing.
  • Streaming. No multilingual model is worth streaming. Nemotron 3.5 streams but is far worse than ultra (de 10.33 vs 4.58, fr 10.78 vs 4.66, English regressed on the leaderboard to 10.51 vs 6.89). Voxtral Realtime is 4.7 GB at 6-7x realtime. Multilingual dictation stays non-streaming.
  • Canary needs an explicit language. Hex's per-language picker supplies one; it must not be offered under Auto.
  • No GGUF for newer candidates. Granite Speech 5.0 2B (leaderboard #2, 5.06) and TurboCTC multilingual revisions are not in transcribe.cpp; Kyutai STT (PyTorch/MLX) and TheWhisper have no GGUF. Excluded.

English

                       FLEURS en   LS clean   Open ASR English avg (current)
Qwen3-ASR 1.7B           3.23        -          5.75
Voxtral Mini 3B          3.89        -          -
Parakeet Unified EN      3.99        1.60 (Hex) -   (8-set offline 5.91 older board)
Granite 4.1 2B           4.14        -          -
parakeet-ultra           4.26        1.80       -   (older board 5.32 vs v3 6.32, from model-watch)
Whisper turbo            4.38        -          8.39
Canary 1B v2             4.47        1.91       -
Parakeet v3              4.83        1.94       6.51
Cohere Transcribe        5.08        1.27       6.26

Current: Recommended Parakeet Unified, Most accurate Cohere, Hints Whisper. Proposed: Recommended Unified (keep), Most accurate Qwen3-ASR 1.7B (medium confidence) or keep Cohere until Hex's own English corpus is run, Hints Whisper (keep). Fastest is already Unified/ultra territory; no separate row is needed.

  • Unified vs ultra: ultra is likely a bit better offline on mixed data (older eight-set average), but FLEURS en favors Unified and only Unified streams. Hex's capture pipeline is built around streaming with a whole-clip fallback (AGENTS.md), so switching English to an offline-only model costs latency for no demonstrated accuracy gain. Keep Unified. Offer ultra only as an optional offline alternative if English users want its long-audio segmentation.
  • Canary v2 and Voxtral 3B do not beat Unified on English. Not proposed.
  • Qwen3-ASR 1.7B: the best English FLEURS and best current-leaderboard open model Hex could run (Apache-2.0, 45x on M4 Max). It costs a ~2 GB download and no streaming, the same trade Cohere already makes. Confidence medium: the FLEURS and leaderboard numbers point the same way, but no one has run it on Hex's dictation audio, and Hex marks Qwen as having no recognition hints.
  • Cohere's "most accurate" label rests on LibriSpeech clean (read audiobooks) and a 0.25-point leaderboard edge over v3; its FLEURS en is the worst here. Medium-low confidence in keeping that label unchanged.

Per-language table (FLEURS test WER %, Q8_0)

Columns: ultra, v3, Canary 1B v2 (cv2), Whisper turbo (wt), plus the best other available model. Best-in-row is bold. Current for every language below is "Recommended v3, Hints Whisper turbo" unless stated.

lang  ultra   v3     cv2    wt     other             proposed (Recommended / Most accurate / Hints)         conf
bg    11.04  12.81   9.22   13.58  -                 ultra / Canary v2 (-1.8) / Whisper                    medium
cs    11.04  12.31   8.56   11.81  Qwen1.7 22.97     ultra / Canary v2 (-2.5) / Whisper                    medium
da    15.98  18.64  11.34   13.60  -                 ultra / Canary v2 (-4.6) / Whisper                    high
de     4.58   5.24   4.46   4.54   Qwen1.7 4.25      ultra / (tie, none) / Whisper                         medium
el    32.25  35.33  26.02   13.26  Cohere 8.96       Cohere / Whisper / Whisper (hints)                    high
en     see English section above
es     3.06   3.65   3.10   3.12   Qwen1.7 3.31      ultra / (tie, none) / Whisper                         high
et    14.44  17.96  12.72   18.44  -                 ultra / Canary v2 (-1.7) / Whisper                    medium
fi    11.14  13.30   8.86   8.29   -                 Whisper / Canary v2 / Whisper (hints)                 medium
fr     4.66   5.30   5.09   5.51   Voxtral3B 4.51    ultra / (tie, none) / Whisper                         medium
hr    11.30  12.59   8.40   12.54  -                 ultra / Canary v2 (-2.9) / Whisper                    medium
hu    13.09  16.06  13.06   15.07  -                 ultra / (tie, none) / Whisper                         medium
it     2.34   3.02   3.10   2.77   Voxtral3B 2.56    ultra / (none) / Whisper                              high
lt    17.06  22.20  13.45   25.11  -                 ultra / Canary v2 (-3.6) / Whisper                    medium
lv    18.04  23.77  10.41   19.53  -                 Canary v2 / ultra / Whisper                           high
mt    17.01  20.63  19.75   70.92  -                 ultra / (none) / (no hints: Whisper unusable)         medium
nl     6.70   7.66   6.28   5.98   Voxtral3B 6.57    ultra / (tie, none) / Whisper                         medium
pl     6.16   7.37   6.88   5.81   Cohere 6.15       ultra / (tie, none) / Whisper                         medium
pt     4.16   4.96   4.50   4.17   Voxtral3B 3.84    ultra / (tie, none) / Whisper                         medium
ro    10.12  12.62   6.87   10.90  -                 ultra / Canary v2 (-3.3) / Whisper                    high
ru     5.63   6.54   7.83   5.93   Qwen1.7 6.25      ultra / (none) / Whisper                              high
sk     7.83  10.19   6.84   10.21  -                 ultra / Canary v2 (-1.0, borderline) / Whisper        low
sl    18.98  24.30  12.76   20.56  -                 Canary v2 / ultra / Whisper                           high
sv    12.83  15.25   9.74    8.72  -                 Whisper / Canary v2 / Whisper (hints)                 medium
uk     5.71   6.84  10.58    7.31  -                 ultra / (none) / Whisper                              high
tr     -      -      -      6.97   Qwen1.7 9.46      Whisper / (none) / Whisper                            high
mk     -      -      -     17.85   Qwen1.7 18.22     Whisper / (none) / Whisper                            medium

tr and mk currently get Qwen3-ASR 0.6B as Recommended (16.74 and 35.09); Qwen3-ASR 1.7B is nearly as good as Whisper there, but Whisper already ships and also offers hints.

Reading the table:

  • ultra vs v3. Lower in all 25 languages, by 0.5 points on the strong Romance languages up to 5 to 6 points on lv, sl, lt, bg, da. Same size, languages and API, so the swap is a catalog entry plus the crate bump. Keep v3 only as a fallback if the crate bump slips. High confidence for the swap (every language moves the same direction, and the card is a like-for-like run of the same pipeline).
  • Canary v2 as "Most accurate" (CC-BY-4.0, 1.14 GB, 105x on M4 Max, still fast): clear wins on bg cs da et hr lt lv ro sk sl, and on fi and sv close to Whisper. It is not better on de es fr it nl pl pt ru uk. It adds an explicit language requirement, no timestamps, and no vocabulary biasing.
  • Whisper turbo wins or ties on fi, nl, pl, sv, tr, mk and is the only hint-capable model, but it is the slowest (36x) and collapses on mt (70.92), lv, lt, sl, et. For those languages Hex should not advertise it as a general option; the Recognition hints row is a trade, not a recommendation.
  • Qwen3-ASR 0.6B is poor outside the big languages (cs 44.50, fi 46.49, hu 56.24); it should not be Recommended for any European language.
  • Cohere covers de es fr it nl pl pt el in Hex, but only wins on Greek. Its FLEURS (de 5.06, fr 5.23, es 3.97) is behind ultra.

What to change in choices_for_runtime

  1. Add parakeet-ultra to MODELS and make it Recommended in the Parakeet branch (requires bumping transcribe-cpp past 0.4.0, and the Nix outputHashes if it touches the build). Keep v3 out of the picker or as a fallback.
  2. Add Canary 1B v2 as Most accurate for bg cs da et hr lt lv ro sk sl (and possibly fi sv). Never under Auto.
  3. Route el away from Parakeet: Cohere Recommended (add el to its branch before the Parakeet match), Whisper for hints.
  4. Route tr and mk to Whisper turbo as Recommended instead of Qwen 0.6B.
  5. English: keep Unified; evaluate Qwen3-ASR 1.7B against Cohere on Hex's own English clips before changing the Most accurate label.

Not measured

  • No local runs: nothing here was rerun on Kit's hardware, and Apple Silicon speeds are the transcribe.cpp cards' M4 Max numbers.
  • Real dictation (noisy, short, accented) is not FLEURS. A cheap check is the benchmark-transcription corpus per language for the changed rows only.
  • Parakeet ultra on 0.2.4 was not tried; the 0.4.0 requirement is inferred from dates and the absent VAD code in the 0.2.4 sources.
  • Qwen3-ASR 1.7B Q8 size and Canary v2's behavior under Hex's language-hint plumbing need a check before adding.

Sources

  • transcribe.cpp (MIT) docs/models/: parakeet-ultra, parakeet-tdt-0.6b-v3, parakeet-unified-en-0.6b, canary-1b-v2, canary-1b-flash, whisper-large-v3-turbo, cohere-transcribe-03-2026, qwen3-asr-0.6b, qwen3-asr-1.7b, voxtral-mini-3b-2507, granite-speech-4.1-2b, nemotron-3.5-asr-streaming-0.6b, orukeet, parakeet-primeline; releases v0.2.4 (2026-09-25), v0.3.1 (10-04), v0.4.0 (10-12), v0.4.1 (10-22); commit 37273409 "parakeet ultra" (2026-10-05).
  • Hex's pinned transcribe-cpp-sys 0.2.4 sources (src/arch/).
  • HF hf-audio/open-asr-leaderboard-results (english_short_latest.csv) and the huggingface/open_asr_leaderboard README (multilingual benchmark description).
  • model-watch.md for the English leaderboard rows and the ultra, Unified, Nemotron, and Moonshine analysis.