hex/docs/plans/language-recommendations.md
2026-10-31 16:12:40 -07:00

195 lines
28 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Per-Language Model Recommendations Plan
**Status:** Approved 2026-10-22; Phase A implemented, Phase B waits for the `transcribe-cpp` bump. Prompting existing users about a better model is deferred until the Rune port is done (see `ROADMAP.md` "Recommend Better Models To Existing Users"); Phase A adds no new UI. Synthesizes [`docs/research/language-recommendations-european.md`](../research/language-recommendations-european.md) (`ebd9fc8`), [`docs/research/language-recommendations-other.md`](../research/language-recommendations-other.md) (`fde2bb3`), and [`docs/research/model-watch.md`](../research/model-watch.md), and integrates with [`docs/plans/transcription-quality.md`](transcription-quality.md) (which owns the `transcribe-cpp` `0.2.4 -> 0.4.1` bump, shared vocabulary, and `parakeet-ultra` runtime work).
## Summary
Hex's current `choices_for_runtime` (`src/transcription_models.rs:494-567`) routes 40 language entries (`auto` plus 39 languages) through four coarse buckets. Apples-to-apples FLEURS test evaluations on the exact Q8_0 GGUF artifacts in `handy-computer/transcribe.cpp` expose three classes of fixes:
```ts
Per-language recommendation updates
├── Phase A: Ship now on pinned transcribe-cpp 0.2.4 (0.5d, zero new models)
│ ├── Greek (el): Parakeet v3 gets 1 in 3 words wrong (35.33% WER) -> route to shipped Cohere (8.96%) + Whisper (13.26%)
│ ├── Qwen 0.6B tail failures: Whisper turbo replaces Qwen 0.6B as Recommended on tr, mk, ar, ms, fil, fa, ja (-3.8 to -23.4 pts)
│ ├── Nordic Parakeet gap: Whisper turbo becomes Recommended on fi (8.29 vs 13.30) and sv (8.72 vs 15.25), Parakeet as Fastest
│ ├── Auto & CJK cleanup: drop 5-language SenseVoice from Auto, yue (37.44% CER), and ko (8.27%); Qwen 0.6B becomes Auto Fastest
│ └── Maltese (mt): drop Whisper turbo (70.92% WER) from Recognition hints
└── Phase B: Ship with transcribe-cpp 0.4.1 in transcription-quality.md Phase 3 (+1d)
├── ParakeetUltra (740 MB): replaces Parakeet v3 across 24 EU languages (-0.6 to -6.2 pts everywhere, 30 s pause cuts)
├── Canary1BV2 (1.14 GB): Recommended on lv, sl (-6.2 to -7.6 pts vs ultra); Most accurate on bg, cs, da, et, hr, lt, ro (-1.7 to -4.6 pts)
└── Qwen3Asr17B (2.19 GB): Most accurate on auto, yue, ko, hi, vi, th, id, fa (and en pending ses_e98f7b094ffe1Q3KtZFS6Np4KL)
```
---
## 1. Final Per-Language Table
**Rules applied across all 40 entries in `LANGUAGES` (`src/transcription_models.rs:437-478`):**
- **Noise floor (`~1.5 points`):** Differences under ~1.5 WER/CER points on FLEURS test (~860–900 read utterances per language, bootstrap 95% CI ~±0.9 pts) are treated as ties. We do not churn an existing recommendation or add a `Most accurate` card for a `<1.5 pt` edge (e.g., `zh` Qwen 0.6B 7.57% vs 1.7B 7.14%; `ko` Qwen 0.6B 5.82% vs Whisper 5.24%; `id` Qwen 0.6B 8.49% vs Whisper 7.20%; `sk` Parakeet Ultra 7.83% vs Canary v2 6.84%; `de`/`es`/`fr`/`hu`/`it`/`nl`/`pl`/`pt`/`ru`/`uk` Ultra vs Canary/Whisper/Voxtral).
- **Roles:** `Rec` = `Recommended` (first entry is also `recommended_selection(lang)` for first-run setup), `Fast` = `Fastest`, `Acc` = `Most accurate`, `Hint` = `Recognition hints`. When `Whisper turbo` is `Rec`, no duplicate `Hint` row is shown because it already supports hints.
- **Metrics:** CER % for `zh`, `yue`, `ja`, `ko`, `th`; WER % for all other languages (Q8_0 GGUF on FLEURS test via `transcribe.cpp`).
| Code | Language | Current (`0.2.4`) | Proposed (`Phase A: 0.2.4` → `Phase B: 0.4.1`) | Evidence (FLEURS Q8_0 WER/CER %) | Conf. | Runtime |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| `auto` | Auto | Rec: Whisper turbo<br>Fast: SenseVoice<br>Rec: Qwen 0.6B | **A:** Rec: Whisper turbo; Fast: Qwen 0.6B (drop SenseVoice)<br>**B:** + Acc: Qwen 1.7B | 30-lang median: Qwen 1.7B **7.6**, Whisper turbo **8.6**, Qwen 0.6B 13.3 (mean 21.4). SenseVoice covers only 5 langs. | Med | `0.2.4` (`1.7B` in B) |
| `en` | English | Rec: Parakeet Unified<br>Acc: Cohere<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Unified; Acc: Qwen 1.7B (or Cohere, gated on audit); Hint: Whisper turbo | FLEURS: Qwen 1.7B **3.23**, Unified **3.99**, Ultra 4.26, Whisper 4.38, Canary 4.47, v3 4.83, Cohere 5.08. Open ASR avg: Qwen 1.7B **5.75**, Cohere 6.26, v3 6.51. | High (Rec)<br>Med (Acc) | `0.2.4` |
| `zh` | Mandarin Chinese | Rec: Qwen 0.6B<br>Fast: SenseVoice<br>Hint: Whisper turbo | **Keep current** (no churn; top models within 0.43 pt noise band) | Qwen 1.7B 7.14, Fun-ASR-MLT 7.33, Qwen 0.6B **7.57**, Whisper turbo 8.50, SenseVoice 10.12 (245x). | Med | `0.2.4` |
| `yue` | Cantonese | Rec: Qwen 0.6B<br>Fast: SenseVoice<br>Hint: Whisper turbo | **A:** Rec: Qwen 0.6B; Hint: Whisper turbo (drop SenseVoice Fast)<br>**B:** + Acc: Qwen 1.7B (Qwen hints enabled in `0.4.1`) | Qwen 1.7B **6.13**, Qwen 0.6B **7.91** vs Whisper turbo 34.62, SenseVoice 37.44 (+29.5 pts; verify script norm). | High | `0.2.4` (`1.7B` in B) |
| `ja` | Japanese | Rec: Qwen 0.6B<br>Fast: SenseVoice<br>Hint: Whisper turbo | **A/B:** Rec: Whisper turbo; Fast: SenseVoice | Whisper turbo **4.82**, Cohere 5.13, Qwen 1.7B 5.29, SenseVoice **7.63** (245x), Qwen 0.6B 8.61 (-3.79 pts). (Fun-ASR-MLT 2.10 outlier deferred). | Med-High | `0.2.4` |
| `ko` | Korean | Rec: Qwen 0.6B<br>Fast: SenseVoice<br>Hint: Whisper turbo | **A:** Rec: Qwen 0.6B; Hint: Whisper turbo (drop SenseVoice Fast)<br>**B:** + Acc: Qwen 1.7B | Qwen 1.7B **4.60**, Whisper turbo 5.24, Qwen 0.6B **5.82** (0.58 pt gap vs Whisper is noise; Qwen 2x faster), SenseVoice 8.27 (+2.45 pts worse). | Med | `0.2.4` (`1.7B` in B) |
| `es` | Spanish | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Ultra **3.06**, Canary v2 3.10, Whisper 3.12, Qwen 1.7B 3.31, v3 3.65 (all top models within 0.25 pt). | High | `0.4.1` (Ultra) |
| `fr` | French | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Ultra **4.66**, Canary v2 5.09, v3 5.30, Whisper 5.51 (Voxtral 3B 4.51 is 0.15 pt noise at 13x speed). | High | `0.4.1` (Ultra) |
| `de` | German | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Qwen 1.7B 4.25, Canary v2 4.46, Whisper 4.54, Ultra **4.58**, v3 5.24 (all within 0.33 pt noise). | High | `0.4.1` (Ultra) |
| `pt` | Portuguese | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Ultra **4.16**, Whisper 4.17, Canary v2 4.50, v3 4.96 (Voxtral 3B 3.84 is 0.32 pt noise). | High | `0.4.1` (Ultra) |
| `it` | Italian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Ultra **2.34**, Whisper 2.77, v3 3.02, Canary v2 3.10. | High | `0.4.1` (Ultra) |
| `ar` | Arabic | Rec: Qwen 0.6B<br>Hint: Whisper turbo | **A/B:** Rec: Whisper turbo; Acc: Cohere Transcribe | Cohere (shipped) **13.60**, Qwen 1.7B 14.91, Whisper turbo **15.48** vs Qwen 0.6B 24.51 (-9.03 to -10.91 pts). | High | `0.2.4` |
| `hi` | Hindi | Rec: Qwen 0.6B<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Qwen 0.6B; Acc: Qwen 1.7B; Hint: Whisper turbo | Qwen 1.7B **7.84** (-4.84 pts), Qwen 0.6B **12.68**, Whisper turbo 18.85. | High | `0.2.4` (`1.7B` in B) |
| `vi` | Vietnamese | Rec: Qwen 0.6B<br>Hint: Whisper turbo | **A:** Rec: Qwen 0.6B; Acc: Cohere; Hint: Whisper turbo<br>**B:** Rec: Qwen 0.6B; Acc: Qwen 1.7B; Hint: Whisper turbo | Qwen 1.7B **6.15** (-3.17 pts), Cohere (shipped) **7.39** (-1.93 pts), Qwen 0.6B **9.32**, Whisper turbo 9.48. | Med | `0.2.4` (`1.7B` in B) |
| `ru` | Russian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Ultra **5.63**, Whisper 5.93, Qwen 1.7B 6.25, v3 6.54, Canary v2 7.83. | High | `0.4.1` (Ultra) |
| `uk` | Ukrainian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Ultra **5.71**, v3 6.84, Whisper 7.31, Canary v2 10.58. | High | `0.4.1` (Ultra) |
| `nl` | Dutch | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Whisper 5.98, Canary v2 6.28, Ultra **6.70**, v3 7.66 (0.72 pt Whisper edge is within 1.5 pt noise; Whisper already shown as Hint). | Med | `0.4.1` (Ultra) |
| `pl` | Polish | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Whisper 5.81, Cohere 6.15, Ultra **6.16**, Canary v2 6.88, v3 7.37 (0.35 pt gap is noise). | Med | `0.4.1` (Ultra) |
| `tr` | Turkish | Rec: Qwen 0.6B<br>Hint: Whisper turbo | **A/B:** Rec: Whisper turbo | Whisper turbo **6.97**, Qwen 1.7B 9.46 vs Qwen 0.6B 16.74 (**-9.77 pts**). | High | `0.2.4` |
| `sv` | Swedish | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Rec: Whisper turbo; Fast: Parakeet v3<br>**B:** Rec: Whisper turbo; Fast: Parakeet Ultra | Whisper turbo **8.72**, Canary v2 9.74 (within 1.02 pt of Whisper), Ultra **12.83**, v3 15.25 (**-6.53 pts** vs v3, **-4.11 pts** vs Ultra). | Med-High | `0.2.4` (Ultra in B) |
| `da` | Danish | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Rec: Whisper turbo; Fast: Parakeet v3<br>**B:** Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo | Canary v2 **11.34** (-4.64 pts vs Ultra), Whisper **13.60** (-5.04 pts vs v3), Ultra **15.98**, v3 18.64. | High | `0.2.4` / `0.4.1` |
| `fi` | Finnish | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Rec: Whisper turbo; Fast: Parakeet v3<br>**B:** Rec: Whisper turbo; Fast: Parakeet Ultra | Whisper turbo **8.29**, Canary v2 8.86 (within 0.57 pt of Whisper), Ultra **11.14**, v3 13.30 (**-5.01 pts** vs v3, **-2.85 pts** vs Ultra). | Med-High | `0.2.4` (Ultra in B) |
| `cs` | Czech | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo | Canary v2 **8.56** (**-2.48 pts** vs Ultra), Ultra **11.04**, Whisper 11.81, v3 12.31. | Med | `0.4.1` |
| `el` | Greek | Rec: Parakeet v3<br>Hint: Whisper turbo | **A/B:** Rec: Cohere Transcribe; Hint: Whisper turbo | Cohere (shipped) **8.96**, Whisper turbo **13.26** vs Canary v2 26.02, Ultra 32.25, v3 35.33 (**-26.37 pts**). | High | `0.2.4` |
| `ro` | Romanian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo | Canary v2 **6.87** (**-3.25 pts** vs Ultra), Ultra **10.12**, Whisper 10.90, v3 12.62. | High | `0.4.1` |
| `hu` | Hungarian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Canary v2 13.06, Ultra **13.09** (0.03 pt tie), Whisper 15.07, v3 16.06. | Med | `0.4.1` (Ultra) |
| `id` | Indonesian | Rec: Qwen 0.6B<br>Hint: Whisper turbo | **A:** Keep current (1.29 pt gap is <1.5 pt noise)<br>**B:** Rec: Qwen 0.6B; Acc: Qwen 1.7B; Hint: Whisper turbo | Qwen 1.7B **5.37** (**-3.12 pts** vs 0.6B), Whisper turbo 7.20, Qwen 0.6B **8.49**. | Med | `0.2.4` (`1.7B` in B) |
| `ms` | Malay | Rec: Qwen 0.6B<br>Hint: Whisper turbo | **A/B:** Rec: Whisper turbo | Whisper turbo **8.64**, Qwen 1.7B 10.42 vs Qwen 0.6B 17.18 (**-8.54 pts**). | Med-High | `0.2.4` |
| `th` | Thai | Rec: Qwen 0.6B<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Qwen 0.6B; Acc: Qwen 1.7B; Hint: Whisper turbo | Qwen 1.7B **6.89** (**-1.92 pts** vs 0.6B), Qwen 0.6B **8.81**, Whisper turbo 13.15. | Med | `0.2.4` (`1.7B` in B) |
| `fa` | Persian | Rec: Qwen 0.6B<br>Hint: Whisper turbo | **A:** Rec: Whisper turbo<br>**B:** Rec: Whisper turbo; Acc: Qwen 1.7B | Qwen 1.7B **28.29** (-2.27 pts vs Whisper), Whisper turbo **30.56** vs Qwen 0.6B 50.30 (**-19.74 pts**). | High | `0.2.4` (`1.7B` in B) |
| `bg` | Bulgarian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo | Canary v2 **9.22** (**-1.82 pts** vs Ultra), Ultra **11.04**, v3 12.81, Whisper 13.58. | Med | `0.4.1` |
| `hr` | Croatian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo | Canary v2 **8.40** (**-2.90 pts** vs Ultra), Ultra **11.30**, Whisper 12.54, v3 12.59. | Med | `0.4.1` |
| `et` | Estonian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo | Canary v2 **12.72** (**-1.72 pts** vs Ultra), Ultra **14.44**, v3 17.96, Whisper 18.44. | Med | `0.4.1` |
| `lv` | Latvian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Canary 1B v2; Fast: Parakeet Ultra; Hint: Whisper turbo | Canary v2 **10.41** (**-7.63 pts** vs Ultra, **-13.36 pts** vs v3), Ultra **18.04**, Whisper 19.53, v3 23.77. | High | `0.4.1` |
| `lt` | Lithuanian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Acc: Canary 1B v2; Hint: Whisper turbo | Canary v2 **13.45** (**-3.61 pts** vs Ultra), Ultra **17.06**, v3 22.20, Whisper 25.11. | Med-High | `0.4.1` |
| `mt` | Maltese | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Rec: Parakeet v3 (drop Whisper Hint)<br>**B:** Rec: Parakeet Ultra | Ultra **17.01**, Canary v2 19.75, v3 **20.63** vs Whisper turbo 70.92 (unusable on Maltese). | Med | `0.2.4` / `0.4.1` |
| `sk` | Slovak | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Parakeet Ultra; Hint: Whisper turbo | Canary v2 6.84 vs Ultra **7.83** (-0.99 pt is inside 1.5 pt noise floor; omit Canary v2), v3 10.19, Whisper 10.21. | Med | `0.4.1` (Ultra) |
| `sl` | Slovenian | Rec: Parakeet v3<br>Hint: Whisper turbo | **A:** Keep current<br>**B:** Rec: Canary 1B v2; Fast: Parakeet Ultra; Hint: Whisper turbo | Canary v2 **12.76** (**-6.22 pts** vs Ultra, **-11.54 pts** vs v3), Ultra **18.98**, Whisper 20.56, v3 24.30. | High | `0.4.1` |
| `mk` | Macedonian | Rec: Qwen 0.6B<br>Hint: Whisper turbo | **A/B:** Rec: Whisper turbo | Whisper turbo **17.85**, Qwen 1.7B 18.22 vs Qwen 0.6B 35.09 (**-17.24 pts**). | High | `0.2.4` |
| `fil` | Filipino | Rec: Qwen 0.6B<br>Hint: Whisper turbo | **A/B:** Rec: Whisper turbo | Whisper turbo **12.08** (`tl` runtime hint) vs Qwen 1.7B 24.29, Qwen 0.6B 35.43 (**-23.35 pts**). | High | `0.2.4` |
---
## 2. New Catalog Entries (Small-Catalog Rule)
A model earns a slot in `MODELS` (`src/transcription_models.rs`) only if it is `Recommended` or `Most accurate` across several languages by `>1.5 points` or materially better for a major language. Three models qualify; seven candidates are rejected.
### Qualifying additions (Phase B, `transcribe-cpp` `0.4.1`)
| `TranscriptionModelId` | Wire name | Display name | Size (Q8_0) | Speed (M4 Max) | License | Pinned artifact repository / filename | Architecture / Variant | Role |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| `ParakeetUltra` | `parakeet_ultra` | `Parakeet Ultra` | ~740 MB | ~207x | CC-BY-4.0 | `handy-computer/parakeet-ultra-gguf`<br>`parakeet-ultra-Q8_0.gguf` | `parakeet` / `ultra-0.6b` | Replaces `ParakeetV3` in picker across 24 EU languages (`Rec` for 20, `Fast` for 4); 30 s pause-aware VAD cuts (`0.4.1`). |
| `Canary1BV2` | `canary_1b_v2` | `Canary 1B v2` | ~1.14 GB | ~105x | CC-BY-4.0 | `handy-computer/canary-1b-v2-gguf`<br>`canary-1b-v2-Q8_0.gguf` | `canary` / `canary-1b-v2` | `Rec` for `lv`, `sl` (-6.2 to -7.6 pts vs Ultra); `Acc` for `bg`, `cs`, `da`, `et`, `hr`, `lt`, `ro` (-1.7 to -4.6 pts). Explicit language required (`supports_language_detection: false`). |
| `Qwen3Asr17B` | `qwen3_asr17_b` | `Qwen3-ASR 1.7B` | 2.19 GB | ~45x | Apache-2.0 | `handy-computer/Qwen3-ASR-1.7B-gguf`<br>`Qwen3-ASR-1.7B-Q8_0.gguf` | `qwen3_asr` / `qwen3-asr-1.7b` | `Acc` for `auto`, `yue`, `ko`, `hi`, `vi`, `th`, `id`, `fa` (and `en` pending audit); supports vocabulary hints in `0.4.1` without `0.6B`'s dictionary echo. |
*(Note: `MoonshineStreamingSmall` is tracked separately in Phase 6 of [`docs/plans/transcription-quality.md`](transcription-quality.md) as the internal voice-command recognizer replacing `libmoonshine`, not a dictation-picker entry.)*
### Already-shipped model unlocked in Phase A (`0.2.4`)
- **`CohereTranscribe` (`cohere_transcribe`, 2.41 GB, Apache-2.0):** Already compiled into `MODELS` with `"el"`, `"ar"`, and `"vi"` in `COHERE_LANGUAGES` (`src/transcription_models.rs:243-245`), but currently offered only for `"en"`. Exposing it as `Recommended` for Greek (`el`, 8.96% vs Parakeet v3 35.33%) and `Most accurate` for Arabic (`ar`, 13.60% vs Qwen 0.6B 24.51%) requires zero new catalog entries.
### Rejected candidates
- **`Fun-ASR-MLT-Nano-2512` (891 MB, FunASR Model License v1.1):** Only stands out on Japanese (`2.10%` CER vs Whisper turbo `4.82%`), which looks like FLEURS training-set overlap, carries a custom license with a non-disparagement termination clause, and enables ITN by default. Deferred unless non-FLEURS Japanese benchmarks confirm a real gain.
- **`Fun-ASR-Nano-2512` (891 MB, FunASR license):** Trails `Qwen 0.6B` on `zh` (8.59% vs 7.57%) and `Whisper turbo` on `ja` (8.50% vs 4.82%).
- **`Cohere Transcribe Arabic` (`cohere-transcribe-arabic-07-2026`, 2.41 GB, Apache-2.0):** Single-language 2.41 GB download from a gated upstream repo for a 2.54 pt gain over the already-shipped 14-language `CohereTranscribe` (`11.06%` vs `13.60%` on `ar`).
- **`Whisper large-v3` (1.67 GB, Apache-2.0):** 2x the size and slower than `Whisper turbo`; only materially beats turbo where `Qwen3-ASR` already wins (`yue`, `th`) or ties (`mk`).
- **`Voxtral Mini 3B 2507` (Apache-2.0, 13x Metal):** Within 0.15–0.32 pts of `ParakeetUltra` on `fr`/`it`/`nl`/`pt` at 1/15th the speed.
- **`Nemotron 3.5 ASR Streaming 0.6B` (751 MB, OpenMDW-1.1):** Regressed accuracy on English (10.51%) and European languages (`de` 10.33%, `fr` 10.78%).
- **`Granite Speech 4.1 2B` / `5.0 2B` (Apache-2.0):** `4.1` does not beat Unified/Whisper; `5.0 2B` (5.06% Open ASR Leaderboard) is not yet ported to `transcribe.cpp`.
---
## 3. The English Decision
`ParakeetUnifiedEnglish` (`parakeet_unified_en`, 731 MB, ~188x Metal, 3.99% FLEURS `en`) **stays the default (`Recommended`) model for English**. It is the only model in Hex's catalog that streams on GPU during hotkey capture (`streams: true`), so switching the default to an offline-only model (`parakeet-ultra` or `Qwen3-ASR 1.7B`) would sacrifice release-to-paste latency on every dictation clip.
Two decisions depend on Kit's in-progress own-recordings audit (`ses_e98f7b094ffe1Q3KtZFS6Np4KL`, comparing `ParakeetUnifiedEnglish`, `Qwen3-ASR 1.7B`, and `parakeet-ultra`):
1. **Default (`Recommended`) threshold:** Keep `ParakeetUnifiedEnglish` as `Recommended` unless `parakeet-ultra` or `Qwen3-ASR 1.7B` beats `ParakeetUnifiedEnglish` by **>= 1.5 WER points** on Kit's real dictation corpus *after* accounting for shared vocabulary (Phase 2/4 of `transcription-quality.md`) **and** keeps p95 post-release transcription latency **<= 150 ms** on a 10 s clip.
2. **`Most accurate` slot (`Qwen3-ASR 1.7B` vs `CohereTranscribe`):** `CohereTranscribe` leads only on read audiobooks (LibriSpeech clean 1.27%), while `Qwen3-ASR 1.7B` beats it on both FLEURS `en` (3.23% vs 5.08%, **-1.85 pts**) and the 9-dataset Open ASR Leaderboard (5.75% vs 6.26%) at a slightly smaller size (2.19 GB vs 2.41 GB) and adds vocabulary hint support in `0.4.1`. Replace `CohereTranscribe` with `Qwen3-ASR 1.7B` as English `Most accurate` in Phase B if `Qwen3-ASR 1.7B` ties or beats `CohereTranscribe` on `ses_e98f7b094ffe1Q3KtZFS6Np4KL`.
---
## 4. Verification Before Shipping
Run targeted local checks via `cargo run --release -- benchmark-transcription` on 30–50 clip subsets before locking disputed rows:
1. **Cantonese (`yue`) script normalization (`0.2.4`, before Phase A):**
- *Claim to check:* Whether `SenseVoice Small` (37.44% CER) and `Whisper turbo` (34.62% CER) fail on Cantonese acoustics or merely emit Simplified characters against Traditional FLEURS references.
- *Check:* Run 30 `yue` clips through `Qwen3Asr06B`, `SenseVoiceSmall`, and `WhisperLargeV3Turbo` and score both raw and Traditional/Simplified-normalized CER. Even if script-normalized CER improves, check whether users dictating Cantonese receive the expected Traditional script.
2. **Greek (`el`) and Arabic (`ar`) with shipped `CohereTranscribe` (`0.2.4`, before Phase A):**
- *Claim to check:* Confirm `CohereTranscribe` loads and decodes `el` and `ar` cleanly under Hex's 35 s chunk cap (`max_audio_chunk_samples` in `src/transcription_models.rs:597-609`) and matches the ~9% (`el`) / ~13.6% (`ar`) FLEURS WER on a 30-clip subset.
3. **`Canary 1B v2` language conditioning (`0.4.1`, before Phase B):**
- *Claim to check:* Verify `canary-1b-v2-Q8_0.gguf` honors `RunOptions` language hints for `lv`, `sl`, `da`, `ro`, `cs`, `hr`, `lt`, `bg`, `et` without requiring a `--prefix` workaround, and measure M-series Metal load/prewarm time and RSS.
4. **`Qwen3-ASR 0.6B` vs `1.7B` vocabulary hints (`0.4.1`, Phase B / `transcription-quality.md` Phase 3):**
- *Claim to check:* `transcribe.cpp` notes that `0.6B` echoes `--vocabulary` dictionary terms into the output more often than `1.7B`. Run the Phase 1 term-free guard set (`transcription-quality.md`) on both models before setting `supports_recognition_hints: true` on `0.6B` vs `1.7B` only.
5. **`Fun-ASR-MLT-Nano-2512` Japanese outlier (optional, only if considering adding it later):**
- *Claim to check:* Test `Fun-ASR-MLT` vs `Whisper turbo` on 30 non-FLEURS Japanese clips (Common Voice `ja`) to see if `2.10%` CER is real or FLEURS training contamination.
---
## 5. Migration
Changing `choices_for_runtime` and adding catalog entries must never disrupt existing installs:
1. **Never auto-switch or auto-download:**
- `AppSettings::load_from` (`src/app_settings.rs:1086`) validates saved selections through `transcription_models::validate(&settings.transcription)` (`src/transcription_models.rs:569-591`), which checks `model.available()` and `model.supports_language(&selection.language)` — **not** `choices_for_runtime()`.
- A user whose saved selection is `(ParakeetV3, "de")`, `(ParakeetV3, "el")`, or `(Qwen3Asr06B, "tr")` keeps their exact model and language on upgrade with zero background downloads or migrations.
- `ParakeetV3` (`"parakeet_v3"`), `CohereTranscribe` (`"cohere_transcribe"`), `Qwen3Asr06B` (`"qwen3_asr06_b"`), and `SenseVoiceSmall` (`"sense_voice_small"`) remain `available() == true` in `MODELS` with stable wire names.
2. **Menu-bar picker stays intact:**
- `src/macos_rune/status_item.rs:340` lists all `MODELS` that are `available()` and `is_installed()`, remembering each model's last language via `transcription_recents`. Users with `ParakeetV3` installed can still switch to and from it in the menu bar.
3. **Keep active/installed legacy models visible and deletable in the Settings picker (`src/rune_ui/picker.rs`):**
- Today `picker::modal` (`src/rune_ui/picker.rs:170`) only renders cards from `choices_for_runtime(&language)`. If a user upgrades with `ParakeetV3` active on French (or installed on disk using 740 MB) after `ParakeetUltra` replaces `ParakeetV3` in `choices_for_runtime("fr")`, `ParakeetV3` would disappear from the French picker modal, hiding both its `Active` indicator and its `Delete` button.
- **Fix in `picker.rs`:** Make `ModelChoice::recommendation` an `Option<Recommendation>` (or append a tail iterator in `picker::modal`) for any model in `MODELS` that supports `language`, is not already in `choices_for_runtime(&language)`, and is either the user's active selection for `language` or `is_installed()`. Render its card without a recommendation badge so the user sees `Active` and can click `Delete` to reclaim the 740 MB / 850 MB after switching to the new `Recommended` model.
---
## 6. Phasing and Integration with `transcription-quality.md`
```ts
Rollout timeline
├── Phase A (now, on transcribe-cpp 0.2.4, 0.5d)
│ └── Pure routing & picker cleanup in src/transcription_models.rs + src/rune_ui/picker.rs
└── Phase B (with transcription-quality.md Phase 3, transcribe-cpp 0.4.1, +1d)
└── Add ParakeetUltra, Canary1BV2, Qwen3Asr17B + full European/Asian table
```
### Phase A: Immediate Routing Fixes on `0.2.4` (Effort: 0.5 days)
Ships immediately before any crate bump or new model downloads:
- **`src/transcription_models.rs` (`choices_for_runtime`):**
- Route `"el"` (Greek) to `[Rec: CohereTranscribe, Hint: WhisperLargeV3Turbo]` before the `PARAKEET_V3_LANGUAGES` arm.
- Route `"fi"` and `"sv"` to `[Rec: WhisperLargeV3Turbo, Fast: ParakeetV3]`.
- Route `"da"` to `[Rec: WhisperLargeV3Turbo, Fast: ParakeetV3]` (until `ParakeetUltra` + `Canary1BV2` land in Phase B).
- Route `"ja"` to `[Rec: WhisperLargeV3Turbo, Fast: SenseVoiceSmall]`.
- Drop `SenseVoiceSmall` (`Fastest`) from `"yue"` and `"ko"`.
- Split the `QWEN_LANGUAGES` arm: keep `Qwen3Asr06B` `Recommended` for `"hi" | "vi" | "th" | "id"` (adding `Acc: CohereTranscribe` on `"vi"`), and route `"ar" | "tr" | "ms" | "fil" | "fa" | "mk"` to `Rec: WhisperLargeV3Turbo` (adding `Acc: CohereTranscribe` on `"ar"`).
- In `AUTO_LANGUAGE`: drop `SenseVoiceSmall` and change `Qwen3Asr06B` from a second `Recommended` to `Fastest` (`[Rec: WhisperLargeV3Turbo, Fast: Qwen3Asr06B]`).
- In `"mt"` (Maltese): omit `WhisperLargeV3Turbo` (`RecognitionHints`) where it gets 70.92% WER.
- **`src/rune_ui/picker.rs`:**
- Show installed or active models supporting `language` that are no longer in `choices_for_runtime(&language)` without a recommendation badge so users can see their active model and delete old weights.
- Update unit tests in `src/transcription_models.rs`.
### Phase B: New Models with `transcribe-cpp` `0.4.1` (Effort: +1 day inside [`docs/plans/transcription-quality.md`](transcription-quality.md) Phase 3)
Integrates directly into Phase 3 of `docs/plans/transcription-quality.md` (which already budgets 2 days for the `0.2.4 -> 0.4.1` bump and `ParakeetUltra`):
- Pin `ParakeetUltra`, `Canary1BV2`, and `Qwen3Asr17B` in `MODELS` with verified byte counts and SHA-256 checksums.
- Swap the `PARAKEET_V3_LANGUAGES` branches from `ParakeetV3` to `ParakeetUltra`, add `Canary1BV2` as `Recommended` on `lv`, `sl` (with `ParakeetUltra` as `Fastest`) and as `Most accurate` on `bg`, `cs`, `da`, `et`, `hr`, `lt`, `ro`.
- Add `Qwen3Asr17B` as `Most accurate` on `auto`, `yue`, `ko`, `hi`, `vi`, `th`, `id`, `fa`, and (if confirmed by `ses_e98f7b094ffe1Q3KtZFS6Np4KL`) `en`.
- Enable `supports_recognition_hints: true` on `Qwen3Asr17B` (and `Qwen3Asr06B` if it passes the Phase 4 dictionary-echo check).
---
## Decisions for Kit
1. **Ship Phase A (`el`, `tr`, `mk`, `ar`, `ms`, `fil`, `fa`, `ja`, `fi`, `sv`, `auto` fixes on `0.2.4`) now, or batch everything into the `0.4.1` bump?**
- *Recommendation:* **Ship Phase A now (0.5d)** — it fixes broken defaults like Greek (35.3% -> 9.0% WER), Filipino (35.4% -> 12.1%), Persian (50.3% -> 30.6%), Macedonian (35.1% -> 17.9%), and Turkish (16.7% -> 7.0%) using models Hex already ships.
2. **Which new models earn a catalog slot in Phase B alongside `ParakeetUltra`: add both `Canary 1B v2` (1.14 GB) and `Qwen3-ASR 1.7B` (2.19 GB), or only one?**
- *Recommendation:* **Add both `Canary 1B v2` and `Qwen3-ASR 1.7B` (3 new models total with `ParakeetUltra`), and reject the other 7 candidates** — `Canary 1B v2` cuts WER by 1.7–7.6 points across 9 European languages, while `Qwen3-ASR 1.7B` is the `Most accurate` tier for `Auto`, Asian/Middle Eastern languages, and potentially English.
3. **On Latvian (`lv`) and Slovenian (`sl`), where `ParakeetUltra` is 18–19% WER and `Canary 1B v2` is 10.4–12.8% WER, should `Canary 1B v2` be `Recommended` (1.14 GB default download) or `Most accurate`?**
- *Recommendation:* **Make `Canary 1B v2` `Recommended` and `ParakeetUltra` `Fastest` on `lv` and `sl`** — a 6.2 to 7.6 point WER drop (nearly cutting errors in half at 105x realtime) is worth the extra 400 MB on first download.