hex/docs/research/transcription-benchmark.md
Dax 3434eb5f35
Rename voice-control to hex everywhere, without migrations (#293)
* Rename voice-control to hex everywhere, without migrations

The Cargo package and binary are now hex, every OS keeps its data in a
hex folder, and VOICE_CONTROL_* variables are HEX_*. Hex is a fresh
app: the macOS voice-control rename, its symlink, and the Windows and
Linux beta settings imports are gone. The Windows installer gets a new
AppId so it never takes over an old voice-control install's entry.

* Format

* Drop the Windows paste-last default only the beta import used
2026-10-28 13:08:44 -04:00

7.4 KiB

Transcription Runtime Benchmark

Status: Historical benchmark method and provisional M2 Max results from 2026-07-16. These measurements are not a current model-selection decision.

The later September 4 reconnaissance measures current-model inference stages and an exploratory thread-count sweep. It did not change production performance defaults.

The commands below run the current harness, not a pinned historical runtime. As of 2026-09-01, omitting --model selects the compiled default GGUF model, Parakeet Unified English, rather than the app's saved selection or the Parakeet v2 model measured here. Pass --model to select the intended GGUF explicitly; the output filename does not select a model. Reproducing the archived comparison also requires its original runtime revisions, model artifacts, corpus, and hardware. See the project README for current model setup.

Run every command from the repository root.

Goal

Compare Hex transcription runtimes on the same M2 Max inputs without changing the production backend based on published results from other machines.

The primary performance metric is median warm real-time factor (median_rtf), where lower is better. Secondary metrics are model load time, median and p95 per-clip latency, peak memory measured externally, and aggregate word error rate.

Corpus

The vocabulary corpus adds a committed synthetic term/distractor set, a public ordinary-speech guard fetcher, and private manifest composition. Optional manifest vocabulary, per-clip expected_terms and group fields enable recall and false-insertion scoring. vocabulary is a scoring list only; it does not change recognition or imply Parakeet hint support.

The corpus is local and ignored by Git because Hex does not retain dictation audio by default. Create perf/transcription-corpus/manifest.json and keep the WAV files beside it:

{
  "clips": [
    {
      "id": "short-product-name",
      "audio": "short-product-name.wav",
      "expected": "OpenCode should keep this sentence concise."
    }
  ]
}

Record each clip once, transcribe the spoken ground truth manually, and then keep both fixed across backend runs. Do not derive references from either model. Use mono or multichannel PCM WAV files; the harness downmixes and resamples them to the same 16 kHz mono input before timing inference.

Build a representative dictation corpus of at least 12 clips covering:

  • one-phrase, one-sentence, and multi-sentence dictation
  • clean speech and ordinary room or keyboard noise
  • immediate speech onset and a natural pause before speech
  • product names, technical prose, numbers, and punctuation requests
  • corrections, hesitations, and repeated words
  • clip durations from roughly 1 to 20 seconds

The production runtime pads clips shorter than 1.5 seconds. The harness applies the same padding but uses the original spoken duration when calculating RTF.

Command

Use a release build and save the production transcribe.cpp JSONL output:

cargo run --release -- benchmark-transcription \
  perf/transcription-corpus/manifest.json \
  > perf/transcription-corpus/transcribe-cpp-metal-v2-q8.jsonl

The command defaults to the installed GGUF model. Pass --model only to test a different GGUF. Run the opt-in legacy ONNX backend against the same manifest:

cargo run --release --features onnx-benchmark -- \
  benchmark-transcription perf/transcription-corpus/manifest.json \
  --backend onnx \
  > perf/transcription-corpus/transcribe-rs-onnx-int8.jsonl

Provisional M2 Max Comparison

On 2026-07-16, ten clips cut from existing local microphone recordings were used to exercise both backends. They contain 127 seconds of technical terms, ordinary prose, hesitations, and repeated-token failure cases. Each backend ran two full-corpus warmups followed by seven measured passes. The Metal experiment was repeated after another ONNX run.

Metric ONNX Int8 transcribe.cpp Metal Q8
Model load, cached Metal library 620 ms 269-277 ms
Model load, first launch after build not observed 9,921 ms
First clip 296 ms 142-150 ms
Median corpus time 3,703 ms 2,056-2,318 ms
Median clip time 368 ms 216-234 ms
p95 clip time 509 ms 303-356 ms
Maximum clip time 551 ms 356-567 ms
Peak resident memory 1.73 GB 1.05 GB
Provisional WER 8.07% 9.82%

With native diagnostics disabled for both backends, Metal Q8 cut median inference time by roughly 37-44%, p95 latency by roughly 30-40%, and peak resident memory by 40%. Its transcripts differed by five additional word edits over 285 reference words. Some differences were clear improvements and others were regressions.

The first launch of a freshly linked Metal binary spent about 9.9 seconds initializing its embedded Metal library. Immediate subsequent launches loaded the model in roughly 270 ms. A production spike must account for this installation-time cold cost and must not report the listener ready before the model and Metal pipelines are usable.

This corpus is useful for runtime comparison but not for the migration quality gate. Its references came from earlier ONNX meeting transcripts, the clips were cut at model-generated segment boundaries, and the speech is not representative hold-to-dictate input. Record independent human references before selecting the production backend.

The default is one full-corpus warmup excluded from summary statistics followed by seven measured passes. Warmup lines preserve cold inference costs, the model line reports cold load time, clip lines preserve first-pass transcripts and quality counts, measurement and run lines expose spread, and the final summary contains the comparison metrics.

Harness Validation

On 2026-07-16, the ONNX backend was run on an M2 Max against one existing 8.38-second microphone recording to verify the release-mode harness end to end:

  • model load: 706.8 ms
  • median warm inference: 275.5 ms
  • median RTF: 0.0329 (30.4x real time)
  • p95 inference: 333.0 ms
  • reference: 19 words, reproduced exactly

This is not migration evidence. The clip was meeting speech rather than a representative dictation corpus, and its reference came from an earlier run of the same ONNX backend. It proves only that decoding, preprocessing, repeated measurement, JSONL reporting, and quality accounting work together.

Measure peak resident memory around the command separately because allocator and backend-owned memory are not reliably visible from inside the process:

cargo build --release
/usr/bin/time -l target/release/hex benchmark-transcription \
  perf/transcription-corpus/manifest.json \
  > perf/transcription-corpus/transcribe-cpp-metal-v2-q8.jsonl

Decision Rule Used For The Migration

The production runtime has moved to transcribe.cpp; ONNX remains available only through the benchmark feature. At migration time, the rule was to retain ONNX until transcribe.cpp was measured against independently referenced hold-to-dictate speech, and to keep the migration only if quality had no material regression and latency and memory wins survived packaging, signing, and foreground-app smoke testing.

Record every backend revision, model and quantization, macOS version, power state, and result filename. Run comparisons plugged in, with no other inference process active, and alternate backend order when repeating the experiment.