# Transcription Quality, Custom Vocabulary, and Runtime Unification **Status:** Plan, 2026-10-22. Synthesizes [`docs/research/model-watch.md`](../research/model-watch.md) and [`docs/research/custom-vocabulary.md`](../research/custom-vocabulary.md) against the current tree (`2.3.1`). ## Summary Hex's default model (`ParakeetUnifiedEnglish`) is fast (~188x realtime on Metal) and accurate on general English (5.91% 8-set WER), but it has no vocabulary input today and misrecognizes domain terms and names into common English words (`"Hex"` -> `"Hanks"`, `"Cappuccino"` -> `"Capuchin"`, `"OpenCode"` -> `"open code"`). Blind find-and-replace cannot fix real-word collisions like `"Hanks"` or `"Capuchin"` without corrupting legitimate sentences. Meanwhile, in-call meeting transcripts are moving onto rolling Parakeet chunks on the shared `TranscriptionEngine` (with Moonshine being dropped from `src/meeting_live.rs`), leaving voice commands (`src/moonshine.rs`) as the last consumer of the ONNX Runtime + `libmoonshine` stack. ```ts Transcription quality plan (ordered by value / effort) ├── Phase 1: Term & false-replacement eval harness (2d) │ └── Extends `hex benchmark-transcription` + corpus scripts with `expected_terms` and a term-free guard set ├── Phase 2: Shared vocabulary, Rune UI, and replacement fixes (4d) │ └── App-wide vocabulary + "heard as" aliases in Settings, History 1-click add, #139/#177 fixes, OpenCode & meeting wiring ├── Phase 3: `transcribe-cpp` 0.2.4 -> 0.4.1 & catalog refresh (2d) │ └── Unlocks `RunOptions::vocabulary`, Qwen3-ASR hints (#129), and `parakeet-ultra` (replaces Parakeet v3) ├── Phase 4: Parakeet greedy-decode phrase boost in `transcribe.cpp` (5d spike + 5d ship) │ └── Host CPU logit biasing + SentencePiece tokenizer + margin gate (`"Hanks"` -> `"Hex"`, `"Capuchin"` -> `"Cappuccino"`) ├── Phase 5: In-call meeting transcripts & optional Apple live tail (3d) │ └── Tightens rolling Parakeet pause cuts (6-10 s) with shared vocabulary; optional ~2% CPU `SpeechTranscriber` draft tail └── Phase 6: Moonshine Streaming GGUF for voice commands (5d) └── Replaces `libmoonshine` + ONNX Runtime + `download.moonshine.ai` with 199 MB Q8 GGUF on `transcribe-cpp` ``` ### Hard invariants preserved across all phases - **Dictation is never delayed by meetings:** meeting chunks run on `TranscriptionEngine` at utility QoS only while `!parakeet::dictation_busy()`, never interrupt a live stream, and finish within the 300 ms hold threshold + 200 ms release tail. - **One loaded dictation model, one active stream per `Model`:** `transcribe-cpp` 0.4.1 still enforces a single active compute/stream lease per `Model` (`Session::stream` returns `Error::Busy` on concurrent use). No second Parakeet `Model` (+0.74 GB RAM) is ever loaded for meetings or commands. - **Local-only by default, strict GPU where required:** Metal on macOS (`src/parakeet.rs`) and Vulkan on Linux / Windows x64 (`src/transcription/local_transcriber.rs`). --- ## Phase 1: Eval Corpus and Benchmark Gate - **Goal:** Measure term recall and false-replacement regressions before shipping any vocabulary, post-processing, or decoder change. - **Changes:** - Extend `ClipDefinition` in `src/transcription_benchmark.rs` (`hex benchmark-transcription`) with optional `expected_terms: Vec` and add `--vocabulary ` to load a vocabulary/alias list into `Parakeet::load_for_benchmark`. - Emit two new summary metrics in `src/transcription_benchmark.rs` JSONL output: - `term_recall`: fraction of `expected_terms` present in the output (case- and spacing-normalized). - `false_replacements`: count and rate of vocabulary terms appearing in transcripts when absent from `expected`. - Build a two-part local benchmark manifest reusing the corpus infrastructure landed in `f4755e4` (`scripts/dictation-corpus.sh` and `src/dictation_corpus_tests.rs`, merged from `test/dictation-corpus`): 1. **Target-term set (~100 clips):** `say` voices, ElevenLabs clips, and Kit's real recordings containing `Hex` (including the real `"Hanks"` miss), `Cappuccino` (the `"Capuchin"` miss), `OpenCode`, `Rune`, `Scrawl`, `Effect`, `Anomaly`, and teammate names. 2. **Term-free guard set (200 LibriSpeech `test-clean` clips + Kit's term-free dictation clips):** includes legitimate sentences using common words like `"Tom Hanks"` and `"Avatar"` to catch over-eager biasing. - **Measurement & gate:** Harness runs end-to-end via `cargo run --release -- benchmark-transcription` and reports baseline `ParakeetUnifiedEnglish` WER, `term_recall`, `false_replacements`, `median_rtf`, `p95_clip_ms`, and `/usr/bin/time -l` peak resident memory. Every subsequent phase gates on: **WER delta <= +0.3 pt on the term-free set, false replacements <= 0.5%, and p95 latency delta < 20 ms on a 10 s clip.** - **Effort:** 2 days. - **Risk:** Low. Synthetic TTS clips alone overestimate phonetic regularity; mitigated by requiring real microphone and meeting-miss clips in the manifest. - **Unblocks:** Objective go/no-go gates for Phases 2, 3, and 4. --- ## Phase 2: Shared Vocabulary List, Replacement Fixes, and Rune UI - **Goal:** Give users one place to define names, jargon, and known misrecognitions that works across every model, mode, dictation, and meeting transcript. - **Changes:** - **Fix `src/dictation/text_replacements.rs` bugs (#139, #177):** - `#177` (CJK never matches): `phrase_boundary` currently requires `!word_character` (`^\w$`) on both sides, which rejects CJK matches inside unspaced sentences. Require word-character boundaries only on edges where the matched phrase itself starts or ends with a space-delimited script (Latin/Cyrillic/Greek/digits); CJK characters self-delimit. - `#139` (overlapping matches skipped): `matcher.find_iter(text)` skips overlapping occurrences of the same rule if an earlier match failed `phrase_boundary`. Scan from the next UTF-8 character boundary after any rejected match. - **Data model (`src/app_settings.rs`):** - Add `pub vocabulary: Vec` (`{ term: String, aliases: Vec }`) to `AppSettings` in `settings.json`. - Add a one-time migration in `MIGRATIONS` that imports any non-empty `transcription.recognition_hints` and `transcription_recents[*].recognition_hints` (comma/newline-split) into `vocabulary`, and removes the `!model.supports_recognition_hints` error from `transcription_models::validate` so switching models never fails when hints are present. - **Pipeline wiring (`src/parakeet.rs`, `src/transcription/local_transcriber.rs`, `src/dictation_processor.rs`, `src/meeting_transcript.rs`):** - **Model hints:** Pass `vocabulary` terms (capped at 64 terms to prevent Whisper silence hallucinations, #108/#193) to models with `supports_recognition_hints`. - **Global alias replacements:** Compile unambiguous `aliases -> term` pairs ( plus multi-word compound joins like `"open code" -> "OpenCode"`) into a shared `ReplacementSet` that runs before mode-specific `replacements` in `dictation_processor.rs`, `dictation_modes.rs`, and on rolling/final chunks in `meeting_transcript.rs`. - **OpenCode rewrite prompt (`src/dictation_processor.rs`):** When a mode enables OpenCode rewriting, append a compact `Vocabulary` block (`"Prefer these exact spellings when a word sounds like one: Hex (heard as: Hanks), Cappuccino (heard as: Capuchin), OpenCode, Rune, Scrawl, Effect…"`). Because the LLM sees sentence context, it resolves real-word collisions (`"Hanks"`, `"Capuchin"`) without corrupting ordinary prose. - **Rune UI (`src/rune_ui/settings.rs`, `src/rune_ui/history.rs`, `src/desktop_i18n.rs`):** - Follow [`docs/design-system.md`](../design-system.md): replace the Whisper-only `Recognition hints` row in `settings::dictation_section` with a `Vocabulary` `section` card visible for every model. - Compose existing `rune_ui/controls.rs` pieces: `card_header` with a Ghost `PLUS` `IconButton` (`"Add term"`) and a Secondary `"Import…"` / `"Export…"` action (`#160`); rows with `TextInput` for `Term (e.g. Cappuccino)` and optional `Heard as (e.g. Capuchin)`, plus `remove_button`. - In `src/rune_ui/history.rs`, add a Secondary `"Add to vocabulary…"` action in `detail_header` when `raw_text` differs from `final_text` or is selected, prefilling the misheard phrase. - Add all five language translations in `src/desktop_i18n.rs` and update `--snap` scenes (`mac-settings`, `history`) in light and dark. - **Measurement & gate:** - Unit tests in `src/dictation/text_replacements.rs` covering CJK phrases (`#177`) and overlapping candidates (`#139`). - Phase 1 benchmark: 100% recall on unambiguous aliases (`"open code" -> "OpenCode"`, `"scrawl"`, `"rune"`) and LLM-rewritten `"Hanks"` / `"Avatar"` sentences; `cargo test --bin hex rune_ui` green with updated light/dark goldens. - **Effort:** 4 days. - **Risk:** Low. Users could enter a common word as an unconditional alias (e.g. `"Hanks" -> "Hex"`); mitigate in Phase 4 by routing single-word real-word collisions through acoustic margin gating when Parakeet is active, or warning inline when an alias is a common dictionary word. - **Unblocks:** Feeds a single user vocabulary into Qwen3-ASR hints (Phase 3), Parakeet decoder boosting (Phase 4), and meeting transcripts (Phase 5). --- ## Phase 3: `transcribe-cpp` Bump (`0.2.4` -> `0.4.1`) and Catalog Refresh - **Status (2026-10-26):** The dependency bump is done with behavior unchanged: same transcripts on the vocabulary corpus, weights now mmap'd (peak footprint about 0.9 GB to 0.15 GB on Metal). Hints, the catalog refresh, and Vulkan prewarm verification remain. - **Goal:** Upgrade `transcribe-cpp` to `0.4.1` to unlock generic `RunOptions::vocabulary` / `prompt`, enable Qwen3-ASR recognition hints (`#129`), and replace `ParakeetV3` with `parakeet-ultra`. - **Changes:** - **Dependency bump (`Cargo.toml`, `src/transcription/gguf_session.rs`, `src/parakeet.rs`, `src/transcription/local_transcriber.rs`):** - Bump `transcribe-cpp` from `=0.2.4` to `=0.4.1` at all three target entries in `Cargo.toml` (default, Linux `vulkan`, Windows x64 `vulkan`). - Update `model.capabilities()` call sites (`src/transcription/gguf_session.rs:110`, `src/parakeet.rs:1073`, `src/transcription/local_transcriber.rs:154`) to handle `Result`. - Replace the Whisper-only `RunExtension::Whisper` hint wiring with `0.4.1`'s generic `RunOptions::vocabulary` / `prompt` capability bits (`docs/prompting.md`), and set `supports_recognition_hints: true` on `Qwen3Asr06B` in `src/transcription_models.rs` (`#129`). - **Model catalog (`src/transcription_models.rs`):** - Add **`ParakeetUltra`** (`handy-computer/parakeet-ultra-gguf`, `parakeet-ultra-Q8_0.gguf`, ~740 MB, CC-BY-4.0, 25 languages): Moondream's post-trained v3 that drops LibriSpeech clean WER from 1.94% to 1.80% (8-set leaderboard 5.32% vs v3's 6.32%) and cuts long audio at natural pauses (30 s segments). - Recommend `ParakeetUltra` in `choices_for_runtime` for the 24 non-English European languages where `ParakeetV3` is recommended today; keep `ParakeetV3` deserializable in `TranscriptionModelId` so existing `settings.json` files continue to work or migrate cleanly. - **Measurement & gate:** - `hex benchmark-transcription` comparing `0.2.4` vs `0.4.1` on `ParakeetUnifiedEnglish`: no WER regression, `median_rtf` and `p95_clip_ms` within 5%, and 8 s stream prewarm (`STREAM_PREWARM_SAMPLES`) passing on Metal and Vulkan. - Verify `Qwen3Asr06B` with Phase 2 vocabulary hints on English/French/Mandarin product-name clips (`#129`). - Compare `ParakeetUltra` vs `ParakeetV3` on the benchmark manifest (WER and pause segmentation on >30 s clips). - **Effort:** 2 days. - **Risk:** Low. `0.4.1` changes C++ internals across architectures; prewarm and peak RSS must be verified on macOS Metal before pushing. - **Unblocks:** Provides the `0.4.1` `RunOptions::vocabulary` plumbing and C++ base for Phase 4. --- ## Phase 4: Parakeet Greedy-Decode Phrase Boost - **Goal:** Bias Parakeet's acoustic decoder toward the user's vocabulary during inference so `"Hex"`, `"Cappuccino"`, `"OpenCode"`, `"Rune"`, `"Scrawl"`, `"Effect"`, and people's names win over phonetically similar common words (`"Hanks"`, `"Capuchin"`) without an LLM call or a second model. - **Why this works in Parakeet:** - In `transcribe.cpp` (`src/arch/parakeet/decoder.cpp`), `decode_rnnt_greedy` (Unified English), `decode_tdt_greedy` (v2/v3/ultra), and streaming RNN-T all run on the host CPU (`n_threads = 1` on GPU devices in `src/transcription/gguf_session.rs`). - At every step, `joint_step` copies the joint network output into a host `std::vector logits`, applies `log_softmax`, and selects `argmax_range(token_logits, n_token_cls)` with `blank_id = vocab`. - **Changes:** - **Step 4a: 5-day spike in a local `transcribe.cpp` 0.4.1 checkout:** 1. **SentencePiece encoder:** Implement tokenization for Parakeet's 1,024-piece SentencePiece vocabulary stored in the GGUF metadata (replacing the `INT_MIN` stub in `transcribe_tokenize`), generating casing variants (`"Hex"`, `" hex"`, `"Cappuccino"`, `" cappuccino"`). 2. **Aho-Corasick / prefix trie shallow fusion (GPU-PB style, arXiv 2508.07014):** Build a token-level prefix trie with failure links from `RunOptions::vocabulary`. 3. **Acoustic margin gate + prefix rollback:** At each `joint_step`, only apply `boost` (e.g. +1.5 to +3.0 nats) to a trie-advancing token `t` when `logits[best_non_blank] - logits[t] <= margin` (the acoustics already support `t`). To prevent a multi-token term (`"Cappuccino"`, `"OpenCode"`) from stranding a broken first subword if a later subword fails the acoustic margin gate, keep a single shadow predictor state while inside an active multi-token trie prefix and commit it only when the full term completes (zero extra compute outside active prefixes). - **Step 4b: Ship upstream + fork bridge (`Cargo.toml`, `src/parakeet.rs`, `src/transcription/local_transcriber.rs`):** - Wire `Parakeet` and `LocalTranscriber` to pass `settings.vocabulary` terms into `RunOptions::vocabulary` (both offline `run` and `with_stream`). - **Upstream vs. fork strategy:** Open an upstream PR to `handy-computer/transcribe.cpp` filling Parakeet's currently empty `vocabulary` slot in `docs/prompting.md` and `transcribe_tokenize`. While upstream reviews, carry a narrow `[patch.crates-io]` git patch pointing to `anomalyco/transcribe.cpp` (pinned rev, updating `nix/package.nix` hashes if needed) so Hex ships immediately, then return to the crates.io pin once upstream releases. - **Fallback if greedy spike misses the gate:** Confidence-gated phonetic snap in Rust using `Transcript::tokens` (`Token::p`) and word timestamps already returned by `transcribe_segments` (`src/parakeet.rs:1190`), with FluidAudio-style guards (>=80% similarity for <=4-char words, >=85% with stop words). - **Measurement & gate:** - Must pass the Phase 1 gate on `ParakeetUnifiedEnglish`: **term recall >= 70% (including `"Hanks" -> "Hex"` and `"Capuchin" -> "Cappuccino"` real clips), false-replacement rate <= 0.5% on the 200-clip LibriSpeech + term-free guard set, overall WER delta <= +0.3 pt, and p95 decode overhead < 5 ms per clip.** - **Effort:** 5 days (spike) + 5 days (streaming path, `[patch.crates-io]`, upstream PR). - **Risk:** Medium. Over-boosting short 1-token words (`"Hex"`) can cause false replacements on similar rhymes (`"checks"`, `"text"`, `"necks"`). Mitigated by scaling the required acoustic margin tighter for short words (<=4 chars) and allowing per-entry `"heard as"` aliases to restrict when a short word boosts. - **Unblocks:** Zero-latency custom vocabulary in hotkey dictation and rolling meeting transcripts without requiring OpenCode. --- ## Phase 5: In-Call Meeting Transcripts and Live Drafts - **Goal:** Make meeting transcripts accurate and up-to-date *during* the call (both in the Meetings pane and `Option-Control-V` meeting paste) without loading a second GGUF model or delaying dictation. - **Coordination with in-flight work (`ses_e992a1790ffej3W41HZ62GVxQb`):** - The other session is switching in-call copies/views to rolling Parakeet chunks on the shared `TranscriptionEngine` and removing Moonshine from `src/meeting_live.rs`. - **Changes:** - **Faster pause-aware rolling chunks (`src/meeting_transcript.rs`):** - Today `CHUNK_MIN = 10 s`, `CHUNK_TARGET = 20 s`, `CHUNK_MAX = 30 s` (`src/meeting_transcript.rs:36-40`), which leaves in-call rolling transcripts 20-30 s behind live speech. - Lower `CHUNK_MIN` to `5 s` and `CHUNK_TARGET` to `8 s` (keeping `CHUNK_MAX = 25 s` and `CHUNK_OVERLAP = 2 s` for continuous talkers). At ~190x realtime on Metal, an 8 s chunk takes ~42 ms on the shared engine thread, finishing well inside dictation's 300 ms hold threshold + 200 ms release tail. - **Vocabulary in rolling and final meeting chunks (`src/meeting_transcript.rs`):** - Apply Phase 2's shared vocabulary replacements and Phase 4's `RunOptions::vocabulary` to rolling and final meeting chunks so `"Hex"`, `"Cappuccino"`, `"OpenCode"`, and participant names are right on the first rolling pass. - **Optional sub-second volatile tail via Apple `SpeechTranscriber` (`src/apple_speech.rs`, `src/meeting_live.rs`, macOS 26+):** - Once Moonshine is removed from `meeting_live.rs`, the only gap is the 0-8 s sentence currently being spoken before a pause cut lands. - If Kit wants live word-by-word captions for that active sentence, stream 100 ms Int16 buffers per track into Apple's on-device `SpeechAnalyzer` / `SpeechTranscriber` (`docs/plans/meeting-live-transcription.md`), emitting `TranscriptPhase::Updated` / `Completed` into `transcript.live.ndjson` after `rolling.covered_ms`. Rolling Parakeet overwrites the tail as each 5-8 s chunk completes. - **Measurement & gate:** - Replay a recorded speaker-mode call through `meeting_echo_replay`: - Rolling Parakeet turn latency after a pause drops from ~20 s to **p50 <= 6 s, p95 <= 10 s**. - Zero increase in speaker-echo leaks (`You` false positives) or dropped local `You` turns. - Dictation latency while a meeting records stays at baseline (**p50 ~70 ms, p95 < 110 ms** on an 8 s dictation clip). - Peak resident memory drops by **~400 MB** compared to the old dual-stream Moonshine live worker (and stays at 0 MB extra model RAM if Apple `SpeechTranscriber` (~2% CPU) is enabled). - **Effort:** 3 days (1 day for rolling cadence + vocabulary; 2 days if adding the `SpeechTranscriber` volatile tail). - **Risk:** Low. More frequent 5-8 s chunks slightly increase Metal wakeups (~15 chunks/min across two tracks = ~0.6 s of GPU work per minute, ~1% duty cycle), still yielding immediately whenever `parakeet::dictation_busy()` is true. - **Unblocks:** Complete removal of Moonshine from meetings, freeing Phase 6 to simplify `src/moonshine.rs` around commands alone. --- ## Phase 6: Moonshine Streaming GGUF for Voice Commands - **Goal:** Replace the ONNX Runtime + `libmoonshine` C library (`src/moonshine.rs`) and `download.moonshine.ai` downloader with `moonshine-streaming-small` Q8_0 GGUF through `transcribe-cpp`. - **Why Moonshine GGUF instead of streaming Parakeet for commands:** - Always-on commands cannot share the dictation model's single `Session::stream` lease without blocking dictation, and a second Parakeet `Model` costs +740 MB RAM and continuous 5.6 s left-context GPU recomputation. - At command latency (<300 ms), `moonshine-streaming-small` (199 MB Q8_0, 7.84% 8-set WER, 2.54% LibriSpeech clean) beats Parakeet Unified at 160 ms (8.44% WER) and runs at 129x realtime on Metal or 64x on CPU. - **Changes:** - Add `moonshine-streaming-small-Q8_0.gguf` (199 MB, `handy-computer/moonshine-streaming-small-gguf`, pinned revision and SHA-256) to Hex's Hugging Face model downloader (`src/transcription_models.rs`). - Rewrite `src/moonshine.rs` on top of `transcribe-cpp`'s `moonshine_streaming` architecture: - Because `transcribe-cpp`'s Moonshine stream returns committed/tentative text and caps a single stream at 81.9 s without built-in VAD line events, drive line lifecycle with Hex's existing 16 kHz `PauseSegmenter` (or `transcribe-cpp`'s built-in Silero VAD v6.2). - Start a stream on speech onset (`TranscriptPhase::Started`), emit `TranscriptPhase::Updated` when tentative/committed text changes (preserving low-latency voice-delimited dictation triggers), and finalize on pause (`TranscriptPhase::Completed`) for ordinary command resolution in `src/recognition.rs`. - Delete `libmoonshine.dylib` dynamic loading (`libloading`), `MOONSHINE_ORT_SINGLE_THREAD` in `src/main.rs`, the 8-file `.ort` manifest, and the `download.moonshine.ai` host requirement. - **Measurement & gate:** - Replay the `moonshine_lab` command fixture suite (`cargo run --profile moonshine-lab -- ...`) comparing ONNX Moonshine vs GGUF Moonshine small: - Command match accuracy and wake-phrase recall >= ONNX baseline. - Update latency **p50 <= 35 ms, p95 <= 70 ms**. - Active command CPU <= **20% of one core** (vs 22% single-threaded ONNX today) and resident memory <= **300 MB** (down from ~400 MB). - **Effort:** 5 days. - **Risk:** Medium. Command responsiveness depends on endpointing hangover when mapping VAD segments onto `Started`/`Updated`/`Completed` lines; tuning against `moonshine_lab` fixtures before deleting the ONNX path prevents regressions. - **Unblocks:** One inference engine (`transcribe.cpp`) and one model downloader (Hugging Face with SHA-256 receipts) across the entire codebase. --- ## Decisions for Kit 1. **Where should the shared Vocabulary UI live?** - *Recommendation:* Put the shared **Vocabulary** card in **Settings -> Dictation** (visible for every model, replacing the Whisper-only `Recognition hints` row) plus a 1-click **"Add to vocabulary"** button on History entries, while keeping per-mode **Corrections** in **Modes** for mode-specific text rules. 2. **Carry a temporary `anomalyco/transcribe.cpp` `[patch.crates-io]` fork for Phase 4 Parakeet phrase boosting, or wait for upstream merge?** - *Recommendation:* **Carry a pinned `[patch.crates-io]` fork while the upstream PR is in review** so Hex gets Parakeet vocabulary boosting immediately once the 5-day spike clears the benchmark gate. 3. **In-call meetings after Moonshine is dropped: is 5-8 s rolling Parakeet enough, or also add the Apple `SpeechTranscriber` (~2% CPU, 0 MB RAM) volatile tail?** - *Recommendation:* **Ship 5-8 s pause-cut rolling Parakeet with shared vocabulary first (1 day)**; add the Apple `SpeechTranscriber` volatile tail only if you miss seeing words appear mid-sentence before a pause. 4. **Replace `Parakeet v3` in the picker with `parakeet-ultra` in Phase 3, or list both?** - *Recommendation:* **Replace `Parakeet v3` with `parakeet-ultra` in the picker** (keeping `parakeet_v3` working in existing `settings.json` files), since `parakeet-ultra` is the same 740 MB Q8 size and 25-language footprint with strictly better WER and pause-aware long-audio segmentation.