hex/docs/research/vocabulary-boost-results.md
Kit Langton fb87ecf28b
Add agent marker and drop internal references (#702)
* Add agent marker and drop internal references

* Format
2026-10-31 18:30:32 -07:00

195 lines
9.4 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Vocabulary boost on real clips
**Status:** Measured, 2026-10-28; whole-clip RNN-T runs moved to decoder
biasing on 2026-10-29 ([below](#decoder-biasing-whole-clip-rnn-t)). Shipped
on by default (no experiment) on 2026-10-29. Follows [`custom-vocabulary.md`](custom-vocabulary.md),
[`vocabulary-real-clips.md`](vocabulary-real-clips.md) and the no-ship
`spike/parakeet-boost` branch.
**Question:** Can Parakeet write the user's Vocabulary terms ("Cappuccino",
"OpenCode", "Scrawl") when it mishears them, without writing them where they
weren't said?
**Answer:** Yes, at the default strength (Normal, 2.5 nats per piece). With
"Cappuccino" in the list, Hex writes it right **12 of 19** times on Kit's real
clips (was 4), and **13 of 19** when streaming (was 6). Across every listed term,
recall goes from 136 to 146 of 157. Word error rate is no worse in any group.
Clips that contain no term are unchanged, and every new term the boost wrote
was right. The cost is time: rescoring adds about 0.05–0.15 s (median) to a clip's
finish when the clip has a candidate, and nothing when it doesn't.
## How it works
The change lives in the `transcribe.cpp` fork (`anomalyco/transcribe.cpp`,
branch `hex/parakeet-vocabulary`, `9badd2a`) under
`src/arch/parakeet/vocabulary.cpp`. Streams and TDT models still use it.
```ts
greedy decode, unchanged ─▶ words with their frames
└─ candidate finder: spans of 1–4 words that look like a term
├─ same letters, a sounds-like spelling, or close letters
├─ or one consonant sound off, only in names with 5+ sounds
└─ skip: case-only, inflections, short terms without a sounds-like,
addresses and paths (. / @ : \ _)
└─ for each candidate: transducer forward score on the same frames,
from the same predictor state
└─ replace when log P(term) + boost × pieces ≥ log P(heard)
```
Hex passes the list at the start of each run or stream
(`transcription::gguf_session::with_vocabulary`), only to Parakeet models that accept it and
only when no Whisper-style hint family is set. `transcription/local_transcriber` shares that
path, so macOS, Linux and Windows all get it. The list is capped at 64 terms and
each term at 12 pieces, with at most 16 candidate spans per decode. Rescoring runs on the transcription engine thread after
decoding, never on the audio thread.
### Why not boost tokens during decoding
The spike added a bonus to term pieces at each greedy step, behind a margin
gate. On real speech the gate was either too tight or too loose. Parakeet is
very sure of "a" at the start of "Cappuccino" (log-prob near 0), so a gate that
protected ordinary speech never let the term start: Cappuccino went 0/11 → 1/11
on the microphone and "Capuchin" stayed Capuchin. A looser gate fired on ordinary
words and raised streaming WER by 0.9 points. Rescoring sees the whole span
before it decides. Where Kit said Cappuccino and Parakeet wrote "capuchin", the
frames score −0.2 as "capuchin" and −9.2 as "Cappuccino". Cappuccino is 5 pieces, so a
2.5 boost (12.5 total) covers that gap and the term wins. Elsewhere "capuchin"
scored −0.1 against −13.7. That gap is wider than the boost, so the word stayed.
The audio decides, not the spelling.
### Rejected: internal language model subtraction
Subtracting the predictor's own text prior (ILM, λ = 0.3–0.5) makes rare names
cheaper. It over-fired: "about" → Cappuccino and "right" → Parakeet. The boost
stays a plain value per piece.
## Method
- **rve** (real-voice-expansion): 168 of Kit's real clips: 20 dictations, 78
meeting microphone, 70 meeting system audio. 19 contain "Cappuccino" and 43
contain no term.
- **tc** (transcription-corpus): 274 clips: 51 synthesized term clips, 18
synthesized look-alikes, 200 ordinary synthesized sentences, 5 private.
- **Vocabulary:** Hex, Cappuccino, OpenCode, Rune, Scrawl, Effect, Anomaly, Dax,
Aiden, Kit, Parakeet, Moonshine, GPUI, Sparkle, Argon, Astra, HiveMind.
- **Model:** `parakeet-unified-en-0.6b-Q8_0`, Metal, on the MacBook Pro. Each
clip runs offline and streaming, with and without the list.
- **WER** is normalized, and "term-normalized" also joins "open code",
"GP UI", "hive mind" and "moon shine" on both sides. Without that, a correct
"OpenCode" against a reference "open code" counts as two errors, which made
raw WER rise even though every change was a fix.
- **False insertions:** a term written where the reference has none. Each one
was checked by hand.
## Results
Recall counts listed terms written correctly. WER is term-normalized.
| rve, offline | Off | Normal (2.5) | Strong (4.0)¹ |
|---|---|---|---|
| Cappuccino | 4/19 | **12/19** | 14/19 |
| All terms | 136/157 | **146/157** | 148/157 |
| WER, all | 5.05 | **4.90** | 4.88 |
| WER, dictation mic | 3.32 | 3.32 | |
| WER, meeting mic | 4.85 | 4.60 | |
| WER, meeting system | 5.63 | 5.59 | |
| WER, no-term clips | 3.14 | **3.14** | 3.24 |
| Wrong insertions | — | **0** | 1 ("Capuchin") |
| rve, streaming | Off | Normal (2.5) |
|---|---|---|
| Cappuccino | 6/19 | **13/19** |
| All terms | 139/157 | **146/157** |
| WER, all | 6.00 | **5.91** |
| WER, dictation mic | 0.26 | 0.26 |
| WER, meeting mic | 6.39 | 6.30 |
| WER, meeting system | 6.41 | 6.32 |
| WER, no-term clips | 2.26 | **2.26** |
| Wrong insertions | — | **0** |
| tc | Offline off | Offline Normal | Streaming off | Streaming Normal |
|---|---|---|---|---|
| Cappuccino | 2/5 | 4/5 | 2/5 | 4/5 |
| All terms | 47/56 | 53/56 | 47/56 | 53/56 |
| WER, all | 1.82 | 1.69 | 1.65 | 1.54 |
| WER, no-term clips | 1.63 | 1.63 | 1.61 | 1.61 |
| Wrong insertions | — | 0 | — | 0 |
¹ Strong was measured at 3.0 per piece. Hex's Strong level is 4.0, a
deliberate step up for users who want it.
The scorer counted four "false" insertions on rve and one on tc at Normal. None
is an error. Two are "Scrawl" where the reference wrote "scroll" but Kit said
Scrawl. Two ("Kit", "Scrawl") are already there with the boost off. The tc one
is "Parakeet" in a clip whose rough reference spells it "parake". At 3.0, the
literal word "Capuchin" became Cappuccino once. That is why Normal is the default.
### Latency
Measured at the final fork commit on the MacBook Pro while it ran other build
jobs, so absolute numbers are noisy; the differences are what matter.
| Per 10 s of audio | Off | Normal |
|---|---|---|
| rve offline, median / p90 | 363 / 670 ms | 356 / 698 ms |
| tc offline, median / p90 | 695 / 957 ms | 763 / 1111 ms |
| rve streaming finish, median / p90 | 175 / 298 ms | 215 / 439 ms |
With this 17-term list, 81 of 168 rve clips had at least one candidate to
rescore. A clip with none pays nothing. Where there are candidates, rescoring
takes a median of 62 ms offline and 122 ms at a stream's finish, and at most
0.55 s. An earlier, looser candidate finder scored 146 clips for the same
results and cost about twice as much.
## Decoder biasing (whole-clip RNN-T)
Measured 2026-10-29 on the same clips and list, offline only. Fork branch
`hex/parakeet-vocabulary-tree`, `7045d1c` (`decode_rnnt_tree` in
`src/arch/parakeet/decoder.cpp`). A whole-clip run on an RNN-T model now biases
while decoding instead of rescoring afterwards:
```ts
greedy decode ─▶ the words, exactly as before
└─ where a term's first piece is plausible at a word start
└─ shadow hypotheses spell the term through the vocabulary trie
├─ boost per piece: the full boost if the greedy words sound like
│ the term (classify_vocabulary_span), else 0.6 × boost
├─ same skips as rescoring: case-only, inflections, addresses
└─ at the word boundary: the term replaces the greedy span only
if it scores higher; otherwise the shadows are dropped
```
A plain RNN-T beam search with the trie (shallow fusion) was tried first and
lost to rescoring: this model gives blank so much weight that summed paths drop
whole phrases, and it ran about 2.5× slower than greedy. Keeping the greedy
path and branching only inside terms fixes both.
| | Off | Rescoring, Normal | Decoder, Light (2.0) | Decoder, Normal (2.5) | Decoder, Strong (4.0) |
|---|---|---|---|---|---|
| rve Cappuccino | 4/19 | 12/19 | 11/19 | **14/19** | 18/19 |
| rve all terms | 136/157 | 146/157 | 148/157 | **151/157** | 155/157 |
| rve WER, all | 5.05 | 4.90 | 4.87 | **4.80** | 4.80 |
| rve WER, no-term clips | 3.14 | 3.14 | 3.14 | **3.14** | 3.24 |
| tc all terms | 47/56 | 53/56 | 53/56 | **54/56** | 55/56 |
| tc WER, all | 1.82 | 1.69 | 1.69 | **1.67** | 1.65 |
| tc WER, no-term clips | 1.63 | 1.63 | 1.63 | **1.63** | 1.63 |
| Decoder time vs greedy | 1× | +rescoring pass | 1.07–1.09× | 1.04–1.08× | 1.17–1.20× |
Strong again turns one ordinary word into a term on a no-term clip, as
rescoring did at 3.0. Rescoring's same-run baseline was 147/157 and 4.88; the
table keeps the published 2026-10-28 figures. Live streams keep rescoring:
decoding the whole utterance again at a stream's finish would cost the latency
streaming exists to save.
## Settings
Vocabulary › Boost: Off (0), Light (2.0), Normal (2.5, default), Strong (4.0).
CLI: `hex settings set vocabulary_boost normal`.
## Next
- Rescoring cost grows with candidates per clip. If the streaming finish
matters, score candidates in one batched joint pass instead of one at a time.
- Remaining misses are mostly "about"/"other" for Cappuccino, where the audio
really sounds like the common word. A user's sounds-like spelling ("a bat
war") fixes those without a stronger boost.