mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* Rename voice-control to hex everywhere, without migrations The Cargo package and binary are now hex, every OS keeps its data in a hex folder, and VOICE_CONTROL_* variables are HEX_*. Hex is a fresh app: the macOS voice-control rename, its symlink, and the Windows and Linux beta settings imports are gone. The Windows installer gets a new AppId so it never takes over an old voice-control install's entry. * Format * Drop the Windows paste-last default only the beta import used
171 lines
7.4 KiB
Markdown
171 lines
7.4 KiB
Markdown
# Transcription Runtime Benchmark
|
|
|
|
**Status:** Historical benchmark method and provisional M2 Max results from
|
|
2026-07-16. These measurements are not a current model-selection decision.
|
|
|
|
The later [September 4 reconnaissance](performance-2026-09-04.md) measures
|
|
current-model inference stages and an exploratory thread-count sweep. It did not
|
|
change production performance defaults.
|
|
|
|
The commands below run the current harness, not a pinned historical runtime.
|
|
As of 2026-09-01, omitting `--model` selects the compiled default GGUF model,
|
|
Parakeet Unified English, rather than the app's saved selection or the Parakeet
|
|
v2 model measured here. Pass `--model` to select the intended GGUF explicitly;
|
|
the output filename does not select a model. Reproducing the archived comparison
|
|
also requires its original runtime revisions, model artifacts, corpus, and
|
|
hardware. See the [project README](../../README.md) for current model setup.
|
|
|
|
Run every command from the repository root.
|
|
|
|
## Goal
|
|
|
|
Compare Hex transcription runtimes on the same M2 Max inputs without changing
|
|
the production backend based on published results from other machines.
|
|
|
|
The primary performance metric is median warm real-time factor (`median_rtf`),
|
|
where lower is better. Secondary metrics are model load time, median and p95
|
|
per-clip latency, peak memory measured externally, and aggregate word error rate.
|
|
|
|
## Corpus
|
|
|
|
The [vocabulary corpus](../../tests/fixtures/vocabulary/README.md) adds a committed
|
|
synthetic term/distractor set, a public ordinary-speech guard fetcher, and private
|
|
manifest composition. Optional manifest `vocabulary`, per-clip `expected_terms`
|
|
and `group` fields enable recall and false-insertion scoring. `vocabulary` is a
|
|
scoring list only; it does not change recognition or imply Parakeet hint support.
|
|
|
|
The corpus is local and ignored by Git because Hex does not retain dictation
|
|
audio by default. Create `perf/transcription-corpus/manifest.json` and keep the
|
|
WAV files beside it:
|
|
|
|
```json
|
|
{
|
|
"clips": [
|
|
{
|
|
"id": "short-product-name",
|
|
"audio": "short-product-name.wav",
|
|
"expected": "OpenCode should keep this sentence concise."
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Record each clip once, transcribe the spoken ground truth manually, and then
|
|
keep both fixed across backend runs. Do not derive references from either model.
|
|
Use mono or multichannel PCM WAV files; the harness downmixes and resamples them
|
|
to the same 16 kHz mono input before timing inference.
|
|
|
|
Build a representative dictation corpus of at least 12 clips covering:
|
|
|
|
- one-phrase, one-sentence, and multi-sentence dictation
|
|
- clean speech and ordinary room or keyboard noise
|
|
- immediate speech onset and a natural pause before speech
|
|
- product names, technical prose, numbers, and punctuation requests
|
|
- corrections, hesitations, and repeated words
|
|
- clip durations from roughly 1 to 20 seconds
|
|
|
|
The production runtime pads clips shorter than 1.5 seconds. The harness applies
|
|
the same padding but uses the original spoken duration when calculating RTF.
|
|
|
|
## Command
|
|
|
|
Use a release build and save the production `transcribe.cpp` JSONL output:
|
|
|
|
```sh
|
|
cargo run --release -- benchmark-transcription \
|
|
perf/transcription-corpus/manifest.json \
|
|
> perf/transcription-corpus/transcribe-cpp-metal-v2-q8.jsonl
|
|
```
|
|
|
|
The command defaults to the installed GGUF model. Pass `--model` only to test a
|
|
different GGUF. Run the opt-in legacy ONNX backend against the same manifest:
|
|
|
|
```sh
|
|
cargo run --release --features onnx-benchmark -- \
|
|
benchmark-transcription perf/transcription-corpus/manifest.json \
|
|
--backend onnx \
|
|
> perf/transcription-corpus/transcribe-rs-onnx-int8.jsonl
|
|
```
|
|
|
|
## Provisional M2 Max Comparison
|
|
|
|
On 2026-07-16, ten clips cut from existing local microphone recordings were
|
|
used to exercise both backends. They contain 127 seconds of technical terms,
|
|
ordinary prose, hesitations, and repeated-token failure cases. Each backend ran
|
|
two full-corpus warmups followed by seven measured passes. The Metal experiment
|
|
was repeated after another ONNX run.
|
|
|
|
| Metric | ONNX Int8 | `transcribe.cpp` Metal Q8 |
|
|
| --- | ---: | ---: |
|
|
| Model load, cached Metal library | 620 ms | 269-277 ms |
|
|
| Model load, first launch after build | not observed | 9,921 ms |
|
|
| First clip | 296 ms | 142-150 ms |
|
|
| Median corpus time | 3,703 ms | 2,056-2,318 ms |
|
|
| Median clip time | 368 ms | 216-234 ms |
|
|
| p95 clip time | 509 ms | 303-356 ms |
|
|
| Maximum clip time | 551 ms | 356-567 ms |
|
|
| Peak resident memory | 1.73 GB | 1.05 GB |
|
|
| Provisional WER | 8.07% | 9.82% |
|
|
|
|
With native diagnostics disabled for both backends, Metal Q8 cut median
|
|
inference time by roughly 37-44%, p95 latency by roughly 30-40%, and peak
|
|
resident memory by 40%. Its transcripts differed by five additional word edits
|
|
over 285 reference words. Some differences were clear improvements and others
|
|
were regressions.
|
|
|
|
The first launch of a freshly linked Metal binary spent about 9.9 seconds
|
|
initializing its embedded Metal library. Immediate subsequent launches loaded
|
|
the model in roughly 270 ms. A production spike must account for this
|
|
installation-time cold cost and must not report the listener ready before the
|
|
model and Metal pipelines are usable.
|
|
|
|
This corpus is useful for runtime comparison but not for the migration quality
|
|
gate. Its references came from earlier ONNX meeting transcripts, the clips were
|
|
cut at model-generated segment boundaries, and the speech is not representative
|
|
hold-to-dictate input. Record independent human references before selecting the
|
|
production backend.
|
|
|
|
The default is one full-corpus warmup excluded from summary statistics followed
|
|
by seven measured passes. Warmup lines preserve cold inference costs, the model
|
|
line reports cold load time, clip lines preserve first-pass transcripts and
|
|
quality counts, measurement and run lines expose spread, and the final summary
|
|
contains the comparison metrics.
|
|
|
|
## Harness Validation
|
|
|
|
On 2026-07-16, the ONNX backend was run on an M2 Max against one existing
|
|
8.38-second microphone recording to verify the release-mode harness end to end:
|
|
|
|
- model load: 706.8 ms
|
|
- median warm inference: 275.5 ms
|
|
- median RTF: 0.0329 (30.4x real time)
|
|
- p95 inference: 333.0 ms
|
|
- reference: 19 words, reproduced exactly
|
|
|
|
This is not migration evidence. The clip was meeting speech rather than a
|
|
representative dictation corpus, and its reference came from an earlier run of
|
|
the same ONNX backend. It proves only that decoding, preprocessing, repeated
|
|
measurement, JSONL reporting, and quality accounting work together.
|
|
|
|
Measure peak resident memory around the command separately because allocator
|
|
and backend-owned memory are not reliably visible from inside the process:
|
|
|
|
```sh
|
|
cargo build --release
|
|
/usr/bin/time -l target/release/hex benchmark-transcription \
|
|
perf/transcription-corpus/manifest.json \
|
|
> perf/transcription-corpus/transcribe-cpp-metal-v2-q8.jsonl
|
|
```
|
|
|
|
## Decision Rule Used For The Migration
|
|
|
|
The production runtime has moved to `transcribe.cpp`; ONNX remains available
|
|
only through the benchmark feature. At migration time, the rule was to retain
|
|
ONNX until `transcribe.cpp` was measured against independently referenced
|
|
hold-to-dictate speech, and to keep the migration only if quality had no
|
|
material regression and latency and memory wins survived packaging, signing,
|
|
and foreground-app smoke testing.
|
|
|
|
Record every backend revision, model and quantization, macOS version, power
|
|
state, and result filename. Run comparisons plugged in, with no other inference
|
|
process active, and alternate backend order when repeating the experiment.
|