23 KiB
Transcription Quality, Custom Vocabulary, and Runtime Unification
Status: Plan, 2026-10-22. Synthesizes docs/research/model-watch.md and docs/research/custom-vocabulary.md against the current tree (2.3.1).
Summary
Hex's default model (ParakeetUnifiedEnglish) is fast (~188x realtime on Metal) and accurate on general English (5.91% 8-set WER), but it has no vocabulary input today and misrecognizes domain terms and names into common English words ("Hex" -> "Hanks", "Cappuccino" -> "Capuchin", "OpenCode" -> "open code"). Blind find-and-replace cannot fix real-word collisions like "Hanks" or "Capuchin" without corrupting legitimate sentences. Meanwhile, in-call meeting transcripts are moving onto rolling Parakeet chunks on the shared TranscriptionEngine (with Moonshine being dropped from src/meeting_live.rs), leaving voice commands (src/moonshine.rs) as the last consumer of the ONNX Runtime + libmoonshine stack.
Transcription quality plan (ordered by value / effort)
├── Phase 1: Term & false-replacement eval harness (2d)
│ └── Extends `hex benchmark-transcription` + corpus scripts with `expected_terms` and a term-free guard set
├── Phase 2: Shared vocabulary, Rune UI, and replacement fixes (4d)
│ └── App-wide vocabulary + "heard as" aliases in Settings, History 1-click add, #139/#177 fixes, OpenCode & meeting wiring
├── Phase 3: `transcribe-cpp` 0.2.4 -> 0.4.1 & catalog refresh (2d)
│ └── Unlocks `RunOptions::vocabulary`, Qwen3-ASR hints (#129), and `parakeet-ultra` (replaces Parakeet v3)
├── Phase 4: Parakeet greedy-decode phrase boost in `transcribe.cpp` (5d spike + 5d ship)
│ └── Host CPU logit biasing + SentencePiece tokenizer + margin gate (`"Hanks"` -> `"Hex"`, `"Capuchin"` -> `"Cappuccino"`)
├── Phase 5: In-call meeting transcripts & optional Apple live tail (3d)
│ └── Tightens rolling Parakeet pause cuts (6-10 s) with shared vocabulary; optional ~2% CPU `SpeechTranscriber` draft tail
└── Phase 6: Moonshine Streaming GGUF for voice commands (5d)
└── Replaces `libmoonshine` + ONNX Runtime + `download.moonshine.ai` with 199 MB Q8 GGUF on `transcribe-cpp`
Hard invariants preserved across all phases
- Dictation is never delayed by meetings: meeting chunks run on
TranscriptionEngineat utility QoS only while!parakeet::dictation_busy(), never interrupt a live stream, and finish within the 300 ms hold threshold + 200 ms release tail. - One loaded dictation model, one active stream per
Model:transcribe-cpp0.4.1 still enforces a single active compute/stream lease perModel(Session::streamreturnsError::Busyon concurrent use). No second ParakeetModel(+0.74 GB RAM) is ever loaded for meetings or commands. - Local-only by default, strict GPU where required: Metal on macOS (
src/parakeet.rs) and Vulkan on Linux / Windows x64 (src/transcription/local_transcriber.rs).
Phase 1: Eval Corpus and Benchmark Gate
- Goal: Measure term recall and false-replacement regressions before shipping any vocabulary, post-processing, or decoder change.
- Changes:
- Extend
ClipDefinitioninsrc/transcription_benchmark.rs(hex benchmark-transcription) with optionalexpected_terms: Vec<String>and add--vocabulary <path>to load a vocabulary/alias list intoParakeet::load_for_benchmark. - Emit two new summary metrics in
src/transcription_benchmark.rsJSONL output:term_recall: fraction ofexpected_termspresent in the output (case- and spacing-normalized).false_replacements: count and rate of vocabulary terms appearing in transcripts when absent fromexpected.
- Build a two-part local benchmark manifest reusing the corpus infrastructure landed in
f4755e4(scripts/dictation-corpus.shandsrc/dictation_corpus_tests.rs, merged fromtest/dictation-corpus):- Target-term set (~100 clips):
sayvoices, ElevenLabs clips, and Kit's real recordings containingHex(including the real"Hanks"miss),Cappuccino(the"Capuchin"miss),OpenCode,Rune,Scrawl,Effect,Anomaly, and teammate names. - Term-free guard set (200 LibriSpeech
test-cleanclips + Kit's term-free dictation clips): includes legitimate sentences using common words like"Tom Hanks"and"Avatar"to catch over-eager biasing.
- Target-term set (~100 clips):
- Extend
- Measurement & gate: Harness runs end-to-end via
cargo run --release -- benchmark-transcriptionand reports baselineParakeetUnifiedEnglishWER,term_recall,false_replacements,median_rtf,p95_clip_ms, and/usr/bin/time -lpeak resident memory. Every subsequent phase gates on: WER delta <= +0.3 pt on the term-free set, false replacements <= 0.5%, and p95 latency delta < 20 ms on a 10 s clip. - Effort: 2 days.
- Risk: Low. Synthetic TTS clips alone overestimate phonetic regularity; mitigated by requiring real microphone and meeting-miss clips in the manifest.
- Unblocks: Objective go/no-go gates for Phases 2, 3, and 4.
Phase 2: Shared Vocabulary List, Replacement Fixes, and Rune UI
- Goal: Give users one place to define names, jargon, and known misrecognitions that works across every model, mode, dictation, and meeting transcript.
- Changes:
- Fix
src/dictation/text_replacements.rsbugs (#139, #177):#177(CJK never matches):phrase_boundarycurrently requires!word_character(^\w$) on both sides, which rejects CJK matches inside unspaced sentences. Require word-character boundaries only on edges where the matched phrase itself starts or ends with a space-delimited script (Latin/Cyrillic/Greek/digits); CJK characters self-delimit.#139(overlapping matches skipped):matcher.find_iter(text)skips overlapping occurrences of the same rule if an earlier match failedphrase_boundary. Scan from the next UTF-8 character boundary after any rejected match.
- Data model (
src/app_settings.rs):- Add
pub vocabulary: Vec<VocabularyEntry>({ term: String, aliases: Vec<String> }) toAppSettingsinsettings.json. - Add a one-time migration in
MIGRATIONSthat imports any non-emptytranscription.recognition_hintsandtranscription_recents[*].recognition_hints(comma/newline-split) intovocabulary, and removes the!model.supports_recognition_hintserror fromtranscription_models::validateso switching models never fails when hints are present.
- Add
- Pipeline wiring (
src/parakeet.rs,src/transcription/local_transcriber.rs,src/dictation_processor.rs,src/meeting_transcript.rs):- Model hints: Pass
vocabularyterms (capped at 64 terms to prevent Whisper silence hallucinations, #108/#193) to models withsupports_recognition_hints. - Global alias replacements: Compile unambiguous
aliases -> termpairs ( plus multi-word compound joins like"open code" -> "OpenCode") into a sharedReplacementSetthat runs before mode-specificreplacementsindictation_processor.rs,dictation_modes.rs, and on rolling/final chunks inmeeting_transcript.rs. - OpenCode rewrite prompt (
src/dictation_processor.rs): When a mode enables OpenCode rewriting, append a compactVocabularyblock ("Prefer these exact spellings when a word sounds like one: Hex (heard as: Hanks), Cappuccino (heard as: Capuchin), OpenCode, Rune, Scrawl, Effect…"). Because the LLM sees sentence context, it resolves real-word collisions ("Hanks","Capuchin") without corrupting ordinary prose.
- Model hints: Pass
- Rune UI (
src/rune_ui/settings.rs,src/rune_ui/history.rs,src/desktop_i18n.rs):- Follow
docs/design-system.md: replace the Whisper-onlyRecognition hintsrow insettings::dictation_sectionwith aVocabularysectioncard visible for every model. - Compose existing
rune_ui/controls.rspieces:card_headerwith a GhostPLUSIconButton("Add term") and a Secondary"Import…"/"Export…"action (#160); rows withTextInputforTerm (e.g. Cappuccino)and optionalHeard as (e.g. Capuchin), plusremove_button. - In
src/rune_ui/history.rs, add a Secondary"Add to vocabulary…"action indetail_headerwhenraw_textdiffers fromfinal_textor is selected, prefilling the misheard phrase. - Add all five language translations in
src/desktop_i18n.rsand update--snapscenes (mac-settings,history) in light and dark.
- Follow
- Fix
- Measurement & gate:
- Unit tests in
src/dictation/text_replacements.rscovering CJK phrases (#177) and overlapping candidates (#139). - Phase 1 benchmark: 100% recall on unambiguous aliases (
"open code" -> "OpenCode","scrawl","rune") and LLM-rewritten"Hanks"/"Avatar"sentences;cargo test --bin hex rune_uigreen with updated light/dark goldens.
- Unit tests in
- Effort: 4 days.
- Risk: Low. Users could enter a common word as an unconditional alias (e.g.
"Hanks" -> "Hex"); mitigate in Phase 4 by routing single-word real-word collisions through acoustic margin gating when Parakeet is active, or warning inline when an alias is a common dictionary word. - Unblocks: Feeds a single user vocabulary into Qwen3-ASR hints (Phase 3), Parakeet decoder boosting (Phase 4), and meeting transcripts (Phase 5).
Phase 3: transcribe-cpp Bump (0.2.4 -> 0.4.1) and Catalog Refresh
- Status (2026-10-26): The dependency bump is done with behavior unchanged: same transcripts on the vocabulary corpus, weights now mmap'd (peak footprint about 0.9 GB to 0.15 GB on Metal). Hints, the catalog refresh, and Vulkan prewarm verification remain.
- Goal: Upgrade
transcribe-cppto0.4.1to unlock genericRunOptions::vocabulary/prompt, enable Qwen3-ASR recognition hints (#129), and replaceParakeetV3withparakeet-ultra. - Changes:
- Dependency bump (
Cargo.toml,src/transcription/gguf_session.rs,src/parakeet.rs,src/transcription/local_transcriber.rs):- Bump
transcribe-cppfrom=0.2.4to=0.4.1at all three target entries inCargo.toml(default, Linuxvulkan, Windows x64vulkan). - Update
model.capabilities()call sites (src/transcription/gguf_session.rs:110,src/parakeet.rs:1073,src/transcription/local_transcriber.rs:154) to handleResult<Capabilities>. - Replace the Whisper-only
RunExtension::Whisperhint wiring with0.4.1's genericRunOptions::vocabulary/promptcapability bits (docs/prompting.md), and setsupports_recognition_hints: trueonQwen3Asr06Binsrc/transcription_models.rs(#129).
- Bump
- Model catalog (
src/transcription_models.rs):- Add
ParakeetUltra(handy-computer/parakeet-ultra-gguf,parakeet-ultra-Q8_0.gguf, ~740 MB, CC-BY-4.0, 25 languages): Moondream's post-trained v3 that drops LibriSpeech clean WER from 1.94% to 1.80% (8-set leaderboard 5.32% vs v3's 6.32%) and cuts long audio at natural pauses (30 s segments). - Recommend
ParakeetUltrainchoices_for_runtimefor the 24 non-English European languages whereParakeetV3is recommended today; keepParakeetV3deserializable inTranscriptionModelIdso existingsettings.jsonfiles continue to work or migrate cleanly.
- Add
- Dependency bump (
- Measurement & gate:
hex benchmark-transcriptioncomparing0.2.4vs0.4.1onParakeetUnifiedEnglish: no WER regression,median_rtfandp95_clip_mswithin 5%, and 8 s stream prewarm (STREAM_PREWARM_SAMPLES) passing on Metal and Vulkan.- Verify
Qwen3Asr06Bwith Phase 2 vocabulary hints on English/French/Mandarin product-name clips (#129). - Compare
ParakeetUltravsParakeetV3on the benchmark manifest (WER and pause segmentation on >30 s clips).
- Effort: 2 days.
- Risk: Low.
0.4.1changes C++ internals across architectures; prewarm and peak RSS must be verified on macOS Metal before pushing. - Unblocks: Provides the
0.4.1RunOptions::vocabularyplumbing and C++ base for Phase 4.
Phase 4: Parakeet Greedy-Decode Phrase Boost
- Goal: Bias Parakeet's acoustic decoder toward the user's vocabulary during inference so
"Hex","Cappuccino","OpenCode","Rune","Scrawl","Effect", and people's names win over phonetically similar common words ("Hanks","Capuchin") without an LLM call or a second model. - Why this works in Parakeet:
- In
transcribe.cpp(src/arch/parakeet/decoder.cpp),decode_rnnt_greedy(Unified English),decode_tdt_greedy(v2/v3/ultra), and streaming RNN-T all run on the host CPU (n_threads = 1on GPU devices insrc/transcription/gguf_session.rs). - At every step,
joint_stepcopies the joint network output into a hoststd::vector<float> logits, applieslog_softmax, and selectsargmax_range(token_logits, n_token_cls)withblank_id = vocab.
- In
- Changes:
- Step 4a: 5-day spike in a local
transcribe.cpp0.4.1 checkout:- SentencePiece encoder: Implement tokenization for Parakeet's 1,024-piece SentencePiece vocabulary stored in the GGUF metadata (replacing the
INT_MINstub intranscribe_tokenize), generating casing variants ("Hex"," hex","Cappuccino"," cappuccino"). - Aho-Corasick / prefix trie shallow fusion (GPU-PB style, arXiv 2508.07014): Build a token-level prefix trie with failure links from
RunOptions::vocabulary. - Acoustic margin gate + prefix rollback: At each
joint_step, only applyboost(e.g. +1.5 to +3.0 nats) to a trie-advancing tokentwhenlogits[best_non_blank] - logits[t] <= margin(the acoustics already supportt). To prevent a multi-token term ("Cappuccino","OpenCode") from stranding a broken first subword if a later subword fails the acoustic margin gate, keep a single shadow predictor state while inside an active multi-token trie prefix and commit it only when the full term completes (zero extra compute outside active prefixes).
- SentencePiece encoder: Implement tokenization for Parakeet's 1,024-piece SentencePiece vocabulary stored in the GGUF metadata (replacing the
- Step 4b: Ship upstream + fork bridge (
Cargo.toml,src/parakeet.rs,src/transcription/local_transcriber.rs):- Wire
ParakeetandLocalTranscriberto passsettings.vocabularyterms intoRunOptions::vocabulary(both offlinerunandwith_stream). - Upstream vs. fork strategy: Open an upstream PR to
handy-computer/transcribe.cppfilling Parakeet's currently emptyvocabularyslot indocs/prompting.mdandtranscribe_tokenize. While upstream reviews, carry a narrow[patch.crates-io]git patch pointing toanomalyco/transcribe.cpp(pinned rev, updatingnix/package.nixhashes if needed) so Hex ships immediately, then return to the crates.io pin once upstream releases. - Fallback if greedy spike misses the gate: Confidence-gated phonetic snap in Rust using
Transcript::tokens(Token::p) and word timestamps already returned bytranscribe_segments(src/parakeet.rs:1190), with FluidAudio-style guards (>=80% similarity for <=4-char words, >=85% with stop words).
- Wire
- Step 4a: 5-day spike in a local
- Measurement & gate:
- Must pass the Phase 1 gate on
ParakeetUnifiedEnglish: term recall >= 70% (including"Hanks" -> "Hex"and"Capuchin" -> "Cappuccino"real clips), false-replacement rate <= 0.5% on the 200-clip LibriSpeech + term-free guard set, overall WER delta <= +0.3 pt, and p95 decode overhead < 5 ms per clip.
- Must pass the Phase 1 gate on
- Effort: 5 days (spike) + 5 days (streaming path,
[patch.crates-io], upstream PR). - Risk: Medium. Over-boosting short 1-token words (
"Hex") can cause false replacements on similar rhymes ("checks","text","necks"). Mitigated by scaling the required acoustic margin tighter for short words (<=4 chars) and allowing per-entry"heard as"aliases to restrict when a short word boosts. - Unblocks: Zero-latency custom vocabulary in hotkey dictation and rolling meeting transcripts without requiring OpenCode.
Phase 5: In-Call Meeting Transcripts and Live Drafts
- Goal: Make meeting transcripts accurate and up-to-date during the call (both in the Meetings pane and
Option-Control-Vmeeting paste) without loading a second GGUF model or delaying dictation. - Coordination with in-flight work (
ses_e992a1790ffej3W41HZ62GVxQb):- The other session is switching in-call copies/views to rolling Parakeet chunks on the shared
TranscriptionEngineand removing Moonshine fromsrc/meeting_live.rs.
- The other session is switching in-call copies/views to rolling Parakeet chunks on the shared
- Changes:
- Faster pause-aware rolling chunks (
src/meeting_transcript.rs):- Today
CHUNK_MIN = 10 s,CHUNK_TARGET = 20 s,CHUNK_MAX = 30 s(src/meeting_transcript.rs:36-40), which leaves in-call rolling transcripts 20-30 s behind live speech. - Lower
CHUNK_MINto5 sandCHUNK_TARGETto8 s(keepingCHUNK_MAX = 25 sandCHUNK_OVERLAP = 2 sfor continuous talkers). At ~190x realtime on Metal, an 8 s chunk takes ~42 ms on the shared engine thread, finishing well inside dictation's 300 ms hold threshold + 200 ms release tail.
- Today
- Vocabulary in rolling and final meeting chunks (
src/meeting_transcript.rs):- Apply Phase 2's shared vocabulary replacements and Phase 4's
RunOptions::vocabularyto rolling and final meeting chunks so"Hex","Cappuccino","OpenCode", and participant names are right on the first rolling pass.
- Apply Phase 2's shared vocabulary replacements and Phase 4's
- Optional sub-second volatile tail via Apple
SpeechTranscriber(src/apple_speech.rs,src/meeting_live.rs, macOS 26+):- Once Moonshine is removed from
meeting_live.rs, the only gap is the 0-8 s sentence currently being spoken before a pause cut lands. - If Kit wants live word-by-word captions for that active sentence, stream 100 ms Int16 buffers per track into Apple's on-device
SpeechAnalyzer/SpeechTranscriber(docs/plans/meeting-live-transcription.md), emittingTranscriptPhase::Updated/Completedintotranscript.live.ndjsonafterrolling.covered_ms. Rolling Parakeet overwrites the tail as each 5-8 s chunk completes.
- Once Moonshine is removed from
- Faster pause-aware rolling chunks (
- Measurement & gate:
- Replay a recorded speaker-mode call through
meeting_echo_replay:- Rolling Parakeet turn latency after a pause drops from ~20 s to p50 <= 6 s, p95 <= 10 s.
- Zero increase in speaker-echo leaks (
Youfalse positives) or dropped localYouturns. - Dictation latency while a meeting records stays at baseline (p50 ~70 ms, p95 < 110 ms on an 8 s dictation clip).
- Peak resident memory drops by ~400 MB compared to the old dual-stream Moonshine live worker (and stays at 0 MB extra model RAM if Apple
SpeechTranscriber(~2% CPU) is enabled).
- Replay a recorded speaker-mode call through
- Effort: 3 days (1 day for rolling cadence + vocabulary; 2 days if adding the
SpeechTranscribervolatile tail). - Risk: Low. More frequent 5-8 s chunks slightly increase Metal wakeups (~15 chunks/min across two tracks = ~0.6 s of GPU work per minute, ~1% duty cycle), still yielding immediately whenever
parakeet::dictation_busy()is true. - Unblocks: Complete removal of Moonshine from meetings, freeing Phase 6 to simplify
src/moonshine.rsaround commands alone.
Phase 6: Moonshine Streaming GGUF for Voice Commands
- Goal: Replace the ONNX Runtime +
libmoonshineC library (src/moonshine.rs) anddownload.moonshine.aidownloader withmoonshine-streaming-smallQ8_0 GGUF throughtranscribe-cpp. - Why Moonshine GGUF instead of streaming Parakeet for commands:
- Always-on commands cannot share the dictation model's single
Session::streamlease without blocking dictation, and a second ParakeetModelcosts +740 MB RAM and continuous 5.6 s left-context GPU recomputation. - At command latency (<300 ms),
moonshine-streaming-small(199 MB Q8_0, 7.84% 8-set WER, 2.54% LibriSpeech clean) beats Parakeet Unified at 160 ms (8.44% WER) and runs at 129x realtime on Metal or 64x on CPU.
- Always-on commands cannot share the dictation model's single
- Changes:
- Add
moonshine-streaming-small-Q8_0.gguf(199 MB,handy-computer/moonshine-streaming-small-gguf, pinned revision and SHA-256) to Hex's Hugging Face model downloader (src/transcription_models.rs). - Rewrite
src/moonshine.rson top oftranscribe-cpp'smoonshine_streamingarchitecture:- Because
transcribe-cpp's Moonshine stream returns committed/tentative text and caps a single stream at 81.9 s without built-in VAD line events, drive line lifecycle with Hex's existing 16 kHzPauseSegmenter(ortranscribe-cpp's built-in Silero VAD v6.2). - Start a stream on speech onset (
TranscriptPhase::Started), emitTranscriptPhase::Updatedwhen tentative/committed text changes (preserving low-latency voice-delimited dictation triggers), and finalize on pause (TranscriptPhase::Completed) for ordinary command resolution insrc/recognition.rs.
- Because
- Delete
libmoonshine.dylibdynamic loading (libloading),MOONSHINE_ORT_SINGLE_THREADinsrc/main.rs, the 8-file.ortmanifest, and thedownload.moonshine.aihost requirement.
- Add
- Measurement & gate:
- Replay the
moonshine_labcommand fixture suite (cargo run --profile moonshine-lab -- ...) comparing ONNX Moonshine vs GGUF Moonshine small:- Command match accuracy and wake-phrase recall >= ONNX baseline.
- Update latency p50 <= 35 ms, p95 <= 70 ms.
- Active command CPU <= 20% of one core (vs 22% single-threaded ONNX today) and resident memory <= 300 MB (down from ~400 MB).
- Replay the
- Effort: 5 days.
- Risk: Medium. Command responsiveness depends on endpointing hangover when mapping VAD segments onto
Started/Updated/Completedlines; tuning againstmoonshine_labfixtures before deleting the ONNX path prevents regressions. - Unblocks: One inference engine (
transcribe.cpp) and one model downloader (Hugging Face with SHA-256 receipts) across the entire codebase.
Decisions for Kit
- Where should the shared Vocabulary UI live?
- Recommendation: Put the shared Vocabulary card in Settings -> Dictation (visible for every model, replacing the Whisper-only
Recognition hintsrow) plus a 1-click "Add to vocabulary" button on History entries, while keeping per-mode Corrections in Modes for mode-specific text rules.
- Recommendation: Put the shared Vocabulary card in Settings -> Dictation (visible for every model, replacing the Whisper-only
- Carry a temporary
anomalyco/transcribe.cpp[patch.crates-io]fork for Phase 4 Parakeet phrase boosting, or wait for upstream merge?- Recommendation: Carry a pinned
[patch.crates-io]fork while the upstream PR is in review so Hex gets Parakeet vocabulary boosting immediately once the 5-day spike clears the benchmark gate.
- Recommendation: Carry a pinned
- In-call meetings after Moonshine is dropped: is 5-8 s rolling Parakeet enough, or also add the Apple
SpeechTranscriber(~2% CPU, 0 MB RAM) volatile tail?- Recommendation: Ship 5-8 s pause-cut rolling Parakeet with shared vocabulary first (1 day); add the Apple
SpeechTranscribervolatile tail only if you miss seeing words appear mid-sentence before a pause.
- Recommendation: Ship 5-8 s pause-cut rolling Parakeet with shared vocabulary first (1 day); add the Apple
- Replace
Parakeet v3in the picker withparakeet-ultrain Phase 3, or list both?- Recommendation: Replace
Parakeet v3withparakeet-ultrain the picker (keepingparakeet_v3working in existingsettings.jsonfiles), sinceparakeet-ultrais the same 740 MB Q8 size and 25-language footprint with strictly better WER and pause-aware long-audio segmentation.
- Recommendation: Replace