hex/docs/features/vocabulary.md
Dax c34d9b1e27
fix(transcription): drop Russian Whisper credits and correct vocabulary spellings (#2290)
Whisper large-v3 and large-v3-turbo on Russian audio end a clip with
"Продолжение следует..." or a subtitle community credit nobody said. Add
those to the trailing subtitle-credit list; the existing letters-and-digits
folding is already Unicode-aware, so Cyrillic case folds correctly.

Parakeet's engine bias replaces an exact "sounds like" spelling with its
term, but a prompt-based model (Whisper, Qwen3-ASR) sends only the terms, so
a listed spelling such as "JetHub" reached the transcript unchanged. After a
whole-clip run on a prompt model, replace whole-word, case-insensitive
occurrences with their term while the boost is not Off, reusing the
text-replacement matcher. A protection rule keeps each term's own casing and
wins the longest match, one-letter spellings are ignored like the engine
does, and only single-word spellings touch the timed words.
2026-11-21 19:53:07 -05:00

25 KiB
Raw Permalink Blame History

Vocabulary

One shared list of names Hex should know (people, projects, products), each with optional "sounds like" spellings a transcript may show instead. An empty list (the default) changes nothing. The window edits the list on macOS, Linux, and Windows; the hex vocabulary commands are macOS only for now.

General › Vocabulary   // vocabulary.settings; after Language
├── Row value -> "None", "1 term", "N terms" -> opens the Vocabulary sheet
├── Selected model takes no vocabulary (ModelDefinition::vocabulary_use)
│   -> muted note on the row and under the sheet's description: "SenseVoice
│   Small doesn't use these terms; rewriting and search still do"; Parakeet,
│   Underdog Bark, Whisper, and Qwen3-ASR (QwenCleo too) show none
├── Selected model reads fewer terms than the list holds -> muted line under
│   the sheet's description: "This model reads the first 38 of 52 terms"
│   (dictation::vocabulary::terms_read: Parakeet's 64-term cap or the
│   prompt's character budget, the same count transcription takes; scene
│   vocabulary-budget)
├── Sheet description -> what the list does, then "Terms with spaces match
│   only when said in full" (a term is matched whole; "Dax Rad" doesn't help
│   "Dax" said alone, so a name also said alone gets its own row)
├── Sheet rows -> Term | Sounds like (comma-separated) | ×; scroll past 8
├── Empty list -> Opens with one blank row focused; opening never animates rows
├── Add term -> New empty row, focused; the list glides to its end and the row
│   rises 8 pt as it fades in (SETTLE, bounce 0; Reduce Motion: fade only);
│   saved once the term has text
├── Labs › Vocabulary files (off by default, every OS) -> Import… and Export… join
│   the footer beside Add term; off, the sheet is exactly as before (labs.md)
│   ├── Import… -> one CSV or TXT regular file (at most 1 MiB, UTF-8), Teams' custom
│   │   dictionary columns: term, then comma-separated "sounds like" spellings; a "Term"
│   │   header, blank rows, and further columns are ignored. A folder, device, or FIFO, a
│   │   larger or non-UTF-8 file, and an unclosed quote are refused with that reason.
│   │   Existing terms keep their entry and gain new spellings; new terms append in file
│   │   order; a term this file added never also counts as updated and a term counts as
│   │   updated once. Skipped counts each refused term once (over 64 characters, or a new
│   │   term past 500) and each dropped spelling (over 64 characters, or past the 16th).
│   │   The merged rows save through the sheet's own draft path, then "Added 12 terms",
│   │   "Added 3 terms, updated 2", "Updated 2 terms, skipped 1", "No new terms in that
│   │   file" only when nothing changed, "No terms found in that file", or the error; a
│   │   failed save says so instead of reporting success
│   └── Export… -> a save dialog named hex-vocabulary.csv; appends .csv only when the
│       name has no extension and <name>.csv doesn't exist, writes the four-column
│       header then term, spellings joined by ", ", and two empty cells, quoting what
│       needs it, atomically; "Exported N terms"
├── Edits -> Debounced drafts, like Modes' corrections -> settings.json `vocabulary`
├── × -> Removes that row at any click speed or focus; rows keep an identity,
│   so a late edit, focus change, or second × for a removed row touches nothing
├── Done, Escape, or a press outside -> Saves, drops blank rows, closes
├── Past the limits -> Keep the term (cut to 64 characters) and the spellings
│   that fit; inline danger note    // never drops the term
└── Boost -> no control in the window: Normal (default) unless changed with
    `hex settings set vocabulary_boost off|light|normal|strong`
    (settings.json `vocabulary_boost`)  // vocabulary.boost

Transcription (Parakeet: RNN-T and TDT, Underdog Bark and Nemotron 3.5 ASR too)  // vocabulary.transcription; transcription::gguf_session::with_vocabulary
├── Parakeet takes the list unless the boost is Off (labs.md)
├── Every run and live stream reads the list and the boost at its start:
│   dictation, meetings, captures, the local API
│   └── `hex transcribe` with no Hex running starts `hex service --embedded`,
│       as does the TypeScript SDK's direct child on every start; it reads
│       the saved vocabulary, boost, Labs switches, and speech engine once
│       when it starts (AppSettings::apply_for_embedded_service); a saved
│       change reaches a running child only after it restarts
├── First 64 terms (the engine's cap) and their spellings; empty or Off -> unchanged
├── Whole-clip run on an RNN-T model (Parakeet unified): meetings, captures,
│   the local API, dictation that falls back to the whole clip
│   ├── The greedy words stand; where the audio could also spell a term, the
│   │   decoder follows the term alongside them, and the term replaces them
│   │   when its score plus the boost per term piece beats theirs at the word
│   │   boundary (0.6 × the boost when the words don't sound like it)
│   └── After the tree decode, a separate pass replaces exact "sounds like"
│       spelling matches with their term, with no scoring; a longer spelling
│       is not split by a shorter spelling of the same term inside it
├── Live stream or TDT model -> after the greedy decode, spans of up to 4 words
│   that look or sound like a term are rescored on the same audio frames from
│   the same decoder state: the term replaces the span when its score plus the
│   boost per term piece beats the heard words'
│   └── Exact "sounds like" spellings resolve first with no scoring (longest
│       span, then earliest; a longer spelling is not split by a shorter
│       spelling of the same term inside it; ordinary candidates overlapping
│       one are skipped), and the pass runs even when a stream kept no frames
├── Left alone: case-only differences, inflections ("Cappuccinos"), addresses
│   and paths ("opencode.ai"), and "sounds like" spellings under 2 characters
│   (Unicode code points, counted after lowercasing and dropping ASCII spaces
│   and punctuation), which are ignored entirely; rescoring also skips terms
│   under 5 letters without a spelling, which the RNN-T decoder can still write
│   ("Kip" -> Kit)
└── SenseVoice, Cohere, GigaAM, Granite, Apple Speech -> no vocabulary in transcription

Context prompt (Whisper incl. ivrit.ai, NB-Whisper and KB-Whisper, Qwen3-ASR, QwenCleo)  // vocabulary.prompt-transcription; transcription::gguf_session::with_vocabulary
├── Always on (not Labs): replaced the per-model Recognition hints box (#129)
├── Every whole-clip run reads the list at its start: dictation, meetings,
│   captures, the local API; live streams run without it
├── "Term, Term, …": the first terms in list order, as many as fit 500
│   characters (Whisper: its prompt slot holds 223 tokens and trims an
│   overlong one from the front) or 1,000 (Qwen3-ASR's system message); the
│   64-term cap is Parakeet's bias array only (#1246)
├── The sheet says how many terms the selected model reads when that is
│   fewer than the list; the 64-term vs full-budget hit rate is unmeasured
├── Whisper: transcribe.cpp's generic context prompt in the <|startofprev|>
│   slot; Qwen3-ASR: the system message
├── Output that only repeats the prompt (two or more terms) is dropped, as
│   both models return it on silence or noise (README.md#get-ready)
└── "Sounds like" spellings are not sent; rewriting and search use them (see
    the correction below)

"Sounds like" correction for prompt runs  // vocabulary.sounds-like-correction; dictation::vocabulary::correct_sounds_like
├── Prompt models such as Whisper and Qwen3-ASR, after a whole-clip run only:
│   dictation, meetings, captures, and the local API; never a live stream's
│   text
├── Parakeet does it in the engine (above). Other models and Apple Speech
│   don't: a model without a vocabulary slot never reads the list, so the
│   Settings note and `hex status` stay accurate for them
├── vocabulary_boost Off turns it off, as does an entry with no usable spelling
├── Uses the whole vocabulary, not only the terms that fit in the prompt: a
│   spelling is corrected even when its term did not fit
├── Each listed spelling, whole word and case-insensitive, becomes its term:
│   "push to JetHub" -> "push to GitHub", "гитхаб" -> "GitHub"
├── A spelling shorter than 2 characters (ASCII letters and digits plus every
│   non-ASCII code point, counted after dropping ASCII spaces and punctuation;
│   never lowercased, so `İ` counts once) is ignored, like the engine's own
│   pass; so is one equal to its term in any case
├── Every entry's term is protected, including a term with no spellings, so
│   another entry's spelling cannot rewrite it: "Next.js" with no spellings
│   survives "Nest" [next]. A spelling inside its own term keeps the term,
│   other occurrences keep their original casing ("next.js" and "Open Code
│   Hub" stay as written), and an entry with an empty term is skipped
├── A longer word ("JetHubber") and a differently split one ("Jet Hub") are
│   left alone
├── Text and segments get every spelling; the timed words get only
│   single-word spellings, so a multi-word spelling ("open code") fixes the
│   text and segments but leaves each timed word as the model wrote it
└── Word times stay put

Words on screen (Labs › Words on screen, macOS)  // vocabulary.screen
├── Off by default; off, or for a model that ignores vocabulary, no
│   Accessibility read happens for a dictation
├── On, as a capture becomes real: one background read of the focused
│   window's text, bounded to 1,500 elements, 40 levels, 32 KB, a soft 150 ms
│   deadline checked between elements, and 4 KB per value; a larger text value
│   reads only its AXVisibleCharacterRange, or is skipped when none is named
├── At most 24 terms: mixed-case and CamelCase words, letters with digits,
│   2–6 letter all-caps acronyms, and capitalized words away from a sentence's
│   start; plain lowercase words, sentence-start capitals, common window
│   labels, hex ids, ordinals, times, units, and random-looking tokens are
│   dropped
├── Joins that dictation after the mode's terms and the shared list, inside
│   the same 64-term/character budgets; a model with VocabularyUse::None gets
│   nothing
├── Parakeet reads them only with a boost other than Off; Whisper and
│   Qwen3-ASR read them as a prompt
├── A capture with terms for a model that reads them uses its whole-clip run
│   rather than a live stream's text: a stream fixes its vocabulary when it
│   starts, so the text arrives after release with a little more latency
├── Only that capture: meetings, the local API, and OpenCode rewriting never
│   see the terms, and only the term count and read time are logged, at debug
├── One reader runs at a time; an intend with no pending capture reads nothing
└── Password fields and their subtrees are skipped; Hex's own window is never
    read
    // checks: dictation::screen_vocabulary::tests (Qdrant, sentence starts,
    // acronyms, long all-caps words, stoplist and log levels, hashes, ordinals,
    // times, units, random tokens, the 24-term cap, dedup against the saved
    // list), a_captures_screen_terms_come_last_and_leave_with_it and
    // parakeet_terms_need_a_boost in vocabulary.rs,
    // a_cancelled_capture_leaves_no_stale_screen_terms and
    // one_screen_read_runs_at_a_time in screen_vocabulary.rs; not observed on
    // a real Mac

Learn from corrections (Labs › Learn from corrections, macOS)  // vocabulary.learn
├── Off by default; after a dictation is pasted, Hex watches only the field it
│   pasted into, reading it about every 100 ms for up to 5 minutes
├── The first read retries for up to a second until the app shows the paste;
│   it compares what Hex actually inserted, never the transcript
├── It stops on a focus change, an empty or vanished field, a newer paste, or
│   the timeout; the lab turned off stops it too and learns nothing, and a
│   read a newer paste overtook is discarded
├── Only a real Paste starts a watch: a copy, a Service or Print output, a
│   YOLO send, and Hex's own window start none
├── It compares the final text with what it pasted and adds up to 5 name-like
│   corrected words to Vocabulary: capitalized mid-sentence, all caps, mixed
│   case, letters with digits, and short phrases of those. A word after a
│   line break, colon, or sentence end is never name-like; neither is I'm
├── Password fields, fields over 20,000 characters, text over 400 words,
│   rewrites (more than 3 inserted words and over 60% changed), plain
│   lowercase words, and terms already in Vocabulary are skipped
├── A learned term is saved like `hex vocabulary add` (vocabulary.add on the
│   saved list, then one settings update that applies it, and an open window
│   adopts it; vocabulary.cli)
└── Nothing leaves the Mac; the one logged event holds the term count only
    // checks: learn_from_corrections_is_its_own_switch_and_macos_only in
    // labs.rs; a_tick_stops_in_the_order_the_watch_reads,
    // a_reading_a_newer_paste_overtook_is_discarded,
    // only_the_lab_turning_off_discards_a_correction,
    // only_a_paste_the_person_can_still_fix_is_watched,
    // learned_terms_join_the_list_the_way_hex_vocabulary_add_does, and
    // a_full_list_keeps_its_terms_and_takes_no_more in correction_watch.rs;
    // learns_a_short_correction_past_its_sixty_percent,
    // ignores_a_long_rewrite, the_pasted_text_matches_only_on_word_boundaries,
    // a_punctuation_edge_needs_no_word_boundary, an_im_word_is_never_a_name,
    // a_name_with_an_apostrophe_is_learned, a_line_break_starts_a_sentence,
    // a_colon_starts_a_sentence, and
    // a_closing_quote_after_a_period_still_starts_a_sentence in
    // correction_terms.rs; not observed on a real Mac

Recognition hints, folded in                // vocabulary.hints
├── Loading settings.json -> `recognition_hints` of the selected model and the
│   recent models (newest first), split on commas, semicolons, and line
│   breaks, become terms; names already listed (any case) are skipped, and
│   the rewritten file drops the field
└── `hex settings get|set transcription.recognition_hints` -> deprecated alias:
    reads the terms comma-separated; set adds names (cli.md)

hex vocabulary                           // vocabulary.cli
├── list [--json]
├── add <term> [--sounds-like a,b] -> New term, or new spellings merged into it
│   └── Already there, nothing new -> changed: false
├── remove <term> -> Any case
└── Running Hex saves and reloads settings; an open window adopts the change

A mode's own vocabulary                  // vocabulary.modes; DictationMode::vocabulary
├── settings.json `dictation_processing.modes[].vocabulary`, same entries; no
│   window editor yet (Output lists it as the mode's first step; Edit with
│   agent's prompt explains it)
├── Its terms come first, then the shared list's, each term once (any case),
│   so the 64-term cap keeps them
├── OpenCode rewriting -> every OS, the mode chosen for the dictation
└── Transcription -> macOS dictation only: the mode of the app and page the
    context monitor last saw when the stream or whole-clip run starts. The app
    is polled every 0.5 s; a scriptable browser's page is read when the browser
    comes to the front and once more when a dictation capture is real (held
    past its hold threshold, at a finish that lands earlier, or at the control
    command's start), never on every press. A live stream starts with the page
    cached when the browser came to the front; the refresh applies to the
    finished text. Meetings, captures, the local API, Linux, and Windows hear with the
    shared list

OpenCode rewriting (Modes)               // vocabulary.prompt
└── Prompt gains "Vocabulary (spell these exactly when the speaker means them):
    term (may be transcribed as a, b); …" before the foreground application

Live meeting chapters (experiment live-chapters)  // vocabulary.chapters
└── Chapter passes and the boundary pass start with "Names and terms that may
    appear: a, b, …" (first 30 terms, no spellings), read on each call

Meeting notes (Labs › Vocabulary in meeting notes)  // vocabulary.notes
├── Off (default) -> notes prompts are unchanged
└── On -> the final title and summary prompt (automatic notes and Regenerate
    summary) starts with the same line, read at each generation, ahead of the
    calendar event; it shares the transcript's excerpt budget (#1154)

Search                                   // vocabulary.search
├── hex meetings search, MCP search_transcripts, hex history --search
├── Query word = a one-word term, fuzzy on
│   ├── Its "sounds like" spellings, or the term as two words -> 0.5 (like a prefix)
│   └── A word that sounds like it (≥2 consonant sounds, ≤ half the letters changed,
│       not a stopword or common word) -> 0.3
└── Aliases never count as exact; exact hits always rank first

The list lives in settings.json as vocabulary, each entry a plain term ("Rune") or {term, sounds_like}; both load, and Hex writes a term without spellings as a plain string. hex vocabulary list --json keeps the object form. It is written only when non-empty, so older files load unchanged and are not rewritten. Terms are unique ignoring case; at most 500 terms, 64 characters each, and 16 spellings per term. Read-only commands (hex history, hex meetings, hex mcp) read the field straight from the file and never load or migrate settings.

Short names with few consonant sounds (Rune, Dax) would sound like too many words, so search finds their mis-transcriptions only through "sounds like" spellings, and rescoring changes them only where a spelling matches. An exact match of one of your "sounds like" spellings (at least 2 characters) is replaced by the term in a separate pass after decoding, with no acoustic scoring; other near matches still go in only when the model scores the term close to what it heard on the same audio. Because that pass never asks how the term sounds, a spelling that is also an ordinary word is replaced whenever it's heard, so keep spellings distinctive. Exact spellings count toward the 16 candidate spans per decode and sort first. Both live in Hex's transcribe.cpp fork (branch hex/vocabulary-sounds-like: decode_rnnt_tree and rescore_vocabulary in src/arch/parakeet/decoder.cpp, while vocabulary.cpp finds candidates, classifies spans, and runs the exact-spelling pass, force_vocabulary_spellings); measurements are in vocabulary-boost-results.md.

Provisional defaults, each changed in one place:

hex vocabulary listed in --help          // control/cli.rs: add #[command(hide = true)] to hide it
Prompt wording                           // dictation/processor.rs: vocabulary_lines
Chapter prompt wording                   // meeting/chapters.rs: vocabulary_line
Short-name rule                          // meeting/query.rs: SOUND_ALIKE_MIN_CONSONANTS
Suggested terms                          // dictation/vocabulary.rs: SUGGESTED
Boost strengths (nats per term piece)    // dictation/vocabulary.rs: VocabularyBoost::nats

Sources: dictation/vocabulary.rs, dictation/vocabulary/csv.rs, rune_ui/vocabulary.rs, meeting/query.rs (vocabulary_aliases, sounds_alike), dictation/processor.rs (vocabulary_lines), and control.rs (edit_vocabulary).

Checks: dictation::vocabulary::csv::tests (header skip, quotes and embedded newlines, an unclosed quote, BOM, CRLF, blank rows, merge with existing terms and spellings, duplicate terms, dropped spellings, the caps, a folder, and an export→import round trip), merge_status and the sheet's import and export through its draft path in rune_ui/vocabulary.rs, the_vocabulary_sheet_offers_files_only_with_its_lab_on in rune_ui/tests.rs, and vocabulary_files_is_its_own_switch_and_everywhere in labs.rs. The native open and save dialogs were not opened in a desktop session. dictation::vocabulary::tests (dedupe and merge, clamping, case-insensitive removal, comma splitting, reading the saved list without settings, the first 64 terms and their spellings reaching transcription, the prompt bounded by characters alone (the_prompt_keeps_the_first_terms_that_fit), the sheet's count matching both (terms_read_counts_exactly_what_transcription_takes), the post-run spelling correction (sounds_like_spellings_become_their_terms_after_transcription, sounds_like_corrections_need_a_boost_and_skip_the_terms_own_spelling, a_spelling_inside_its_own_term_keeps_the_term, a_term_without_spellings_still_protects_itself, an_entry_without_a_term_is_skipped, a_term_keeps_its_casing_outside_its_spellings, a_timed_word_keeps_its_casing_and_takes_a_spelling, a_one_letter_spelling_is_ignored, a_transcripts_text_segments_and_words_are_corrected, and a_multi_word_spelling_corrects_text_and_segments_but_not_words), folding hints, boost names; protect_rules_keep_the_matched_text_and_still_win_the_longest_match and a_protect_rule_coexists_with_deletion_rules in dictation/text_replacements.rs cover the protection rule; only_a_prompt_run_corrects_sounds_like_spellings in transcription/gguf_session_tests.rs checks the per-slot wiring; loading_folds_saved_recognition_hints_into_the_vocabulary_once in app_settings.rs; the_embedded_service_reads_the_saved_vocabulary in app_settings/persistence/tests.rs reads the saved list, boost, and Labs switches through the no-write path and checks the bias they produce; publishing them into the embedded service's globals, and run_service calling that, have no automated check, so only a separate hex service --embedded process shows them; recognition_hints_alias_the_vocabulary in control/settings.rs; whisper_and_qwen3_asr_read_the_vocabulary_as_a_prompt_and_parakeet_biases in transcription/models.rs; vocabulary_round_trips_and_stays_out_of_files_without_one in app_settings.rs; a_vocabulary_term_finds_its_mis_transcriptions_below_exact_hits, sounds_like_spellings_and_split_words_alias_a_term, vocabulary_never_pulls_in_common_words, and sound_alikes_need_two_consonant_sounds_and_few_changes in meeting/query_tests.rs; prompt_names_vocabulary_terms_and_their_mistranscriptions; vocabulary_edits_report_whether_they_changed_the_list and the request round trip in control.rs; vocabulary_lists_terms_with_their_spellings in control/cli.rs; rows_project_to_cleaned_merged_entries, rows_past_the_limits_keep_their_term_and_what_fits, the_sheet_starts_empty_lists_with_a_row_and_saves_on_close, removing_a_focused_row_leaves_the_others_and_late_edits_touch_nothing, removing_the_last_row_then_editing_it_is_harmless, rapid_removals_from_one_frame_remove_only_their_own_rows, and a_reload_retires_the_old_rows in rune_ui/vocabulary.rs; the_vocabulary_row_opens_its_sheet, add_term_scrolls_the_new_row_into_view, the_vocabulary_row_says_when_the_model_ignores_the_list, and an_outside_press_on_a_settle_frame_still_exits_the_vocabulary_sheet (an outside press on the first frame after the window comes back still plays the sheet's exit) in rune_ui/tests.rs; only_parakeet_based_models_use_the_vocabulary in transcription/models.rs; transcribe.cpp's tests/parakeet_vocabulary_unit.cpp (fork branch hex/vocabulary-sounds-like) covers the pure spelling pass, overlap order (longest span, then earliest), the 1-character rule, and the span rules, but Hex CI does not run it. Not observed: an end-to-end audio check of a forced spelling replacement. Scenes: mac-settings-vocabulary, mac-settings-vocabulary-empty, vocabulary-unused (Whisper, no golden). Passed on macOS on 2026-10-23; the unused-model note's checks have not been observed running. Not yet observed: the sheet edited in a running Hex, a CLI edit against a running Hex, or a rewrite with a live provider.