mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* add fluid speech pro to the speech model catalog FluidVoice's Fluid Speech Pro, a Parakeet TDT 0.6B v3 retrain that also declares Moondream's Parakeet Ultra as a base, for the same 25 European languages. TalkMate converted it to a Q8_0 transcribe.cpp GGUF at d59dc79 (741,819,456 bytes, sha256 c3848e98...); the file says parakeet/tdt-0.6b-fluid-speech-pro and announces its extra pre-Conformer block as stt.parakeet.encoder.pre_block_kernel, so it needs a transcribe.cpp that reads enc.pre_block. Like Parakeet v3 it takes no language hint, detects the language under Auto, does not stream, and is never recommended or badged. Its card publishes no LibriSpeech test-clean figure, so it carries no single word error rate and no benchmark record; its per-language FLEURS figures and the CC BY 4.0 credit are in its details. * bump(transcribe.cpp): 295c7e6, fluid speech pro's pre-conformer block Fluid Speech Pro's GGUF announces stt.parakeet.encoder.pre_block_kernel and carries eight enc.pre_block.* tensors: a residual FrameCleaner block between the subsampler and Conformer layer 0. The old pin's graph never read them, so the model loaded as plain Parakeet v3 and its encoder was wrong. Both transcribe-cpp crates move to 295c7e601 on hex/combined-fluid, a fast-forward of bcb639f that runs the block on pre_encode before xscaling when the KV is present, and fails the load rather than skipping tensors when it is only partly there. Cargo.lock changes only those two sources (`getrandom`'s tempfile pointer was reverted to what it was), and nix/package.nix carries the hash from a failed `nix build .#hex`, then a full one whose check phase passed. `hex model install fluid-speech-pro` followed by `hex model transcribe` on whisper.cpp's jfk.wav prints the expected sentence, as does Parakeet v3 on the same clip. * fluid speech pro: snake_case id, FLEURS-only figures, blurb, notices
402 lines
19 KiB
Markdown
402 lines
19 KiB
Markdown
# Hex Local Transcription Service and SDK
|
||
|
||
**Status:** The internal service and direct-child mode are implemented.
|
||
`@kitlangton/hex@0.3.0` publishes ready transcribers through Promise and Effect
|
||
entrypoints. Local Electron integration has been exercised, but the native helper
|
||
package remains private and is not included as a client dependency. An explicit
|
||
helper command is still required; signed consumer distribution remains unverified.
|
||
|
||
**Authority:** This document describes the embedded service protocol and labels
|
||
proposed distribution separately. The [SDK README](../../sdk/typescript/README.md)
|
||
is the current consumer guide. Delivery scope, evidence, and acceptance gates live in
|
||
[`../plans/typescript-sdk.md`](../plans/typescript-sdk.md).
|
||
|
||
## Product Boundary
|
||
|
||
HEX Service owns local transcription models and inference. Host applications
|
||
own recording.
|
||
|
||
```text
|
||
Host application HEX Service
|
||
---------------- -----------
|
||
microphone permission service discovery and authentication
|
||
device selection model download and verification
|
||
capture start/stop strict-Metal load and prewarm
|
||
levels and waveform PCM WAV resampling and inference
|
||
capture UX ----------> final raw transcript
|
||
settings and model choice
|
||
```
|
||
|
||
The host's signed app owns the microphone grant. A correctly configured packaged
|
||
host prompts under its own identity, not Hex; a development Electron runtime may
|
||
appear as Electron. The embedded helper never opens the microphone.
|
||
|
||
## Decisions
|
||
|
||
| Decision | Reason |
|
||
| --- | --- |
|
||
| Host owns microphone capture | Correct TCC identity, intuitive permission prompt, no shared permission broker. |
|
||
| Service owns models and inference | Hosts share verified model artifacts while each helper owns its warm runtime. |
|
||
| Completed audio goes over localhost | Simple final-only contract; no partial transcripts or bidirectional session protocol. |
|
||
| Model preparation is explicit | A transcription call never hides a large network download. |
|
||
| Installed-but-cold models may reload automatically | A request for a different model in the same helper replaces its warm runtime. |
|
||
| No product duration limit | Hotkey capture and service transcription end explicitly; operational resource exhaustion remains a typed failure. |
|
||
| Raw transcripts only | The host owns meaning, rewriting, and insertion. The running desktop app's authenticated API can apply Global mode's output processing when a request explicitly asks with `process=global` (opt-in); the embedded helper has no processor. |
|
||
| Trusted Node/Electron main process | Ordinary browser code and untrusted renderers must not receive native process authority or service credentials. |
|
||
| Host-spawned helper | The host owns the process, endpoint, and lifetime instead of discovering an independently elected service. |
|
||
| Shared model artifacts, private warm runtime | Hosts reuse verified downloads without coupling their model choice or process lifecycle. |
|
||
|
||
## Proposed Embedded Distribution (Not Yet Shipping)
|
||
|
||
The target SDK will declare signed, notarized helpers as optional
|
||
architecture-specific packages and select the current platform package
|
||
automatically. Applications will call `create()` without locating an executable.
|
||
On explicit host use, the SDK will spawn the included helper directly as a child
|
||
without an npm install script, administrator access, login-item approval, Dock icon,
|
||
or foreground window. The host must not launch the helper through Launch Services.
|
||
|
||
Because the embedded service does not capture audio, it does not need microphone
|
||
TCC identity. Full `Hex.app` already owns its global-hotkey capture and
|
||
transcription pipeline; desktop `connect()` can also use the app's microphone
|
||
through its separate capture API.
|
||
|
||
Verified model artifacts remain shared per user under Application Support.
|
||
Each embedded helper has its own warm model and inference queue. Multiple hosts
|
||
may reuse the same verified file, but they do not share Metal allocations or
|
||
runtime selection. Inside `Hex.app`, a clip of up to 30 seconds for the GGUF
|
||
model dictation has loaded runs on dictation's model between dictations, so
|
||
the app holds one copy; another model, or a longer clip, uses the service's own,
|
||
and a longer clip's copy is released when dictation holds the model. The current cache is filename-based; immutable revision-safe
|
||
storage for independently versioned helpers remains planned.
|
||
|
||
## Embedded Startup and Authentication
|
||
|
||
The host spawns `hex-service service --embedded` with piped stdin and stdout.
|
||
The helper binds `127.0.0.1:0` and writes exactly one JSON line to stdout after
|
||
the authenticated API is ready:
|
||
|
||
```json
|
||
{
|
||
"type": "ready",
|
||
"url": "http://127.0.0.1:49731",
|
||
"token": "hex_a1b2c3...",
|
||
"apiVersion": "2",
|
||
"pid": 12345
|
||
}
|
||
```
|
||
|
||
The host keeps the child's stdin open as its lifetime lease. EOF requests a
|
||
clean shutdown; the helper force-exits after a five-second grace period if
|
||
native work does not return. Embedded mode writes no shared discovery file and
|
||
acquires no global service lock, so two host applications can run independent
|
||
helpers.
|
||
|
||
The current executable starts lease monitoring after local API initialization,
|
||
not at process entry. Startup-failure and descendant-process cleanup remain
|
||
native-distribution acceptance gates in the delivery plan.
|
||
|
||
Every request requires `Authorization: Bearer <token>`. The token is delivered
|
||
only over the child's stdout pipe. The server deliberately sends no CORS
|
||
headers. A parsed request with a missing or incorrect bearer credential returns
|
||
an empty `401` response; malformed HTTP can be rejected before authentication.
|
||
The standalone Hex application may still publish owner-only discovery for its
|
||
own existing local API. Embedded `create()` uses the child handshake; desktop
|
||
`connect()` remains a separate discovery-based path.
|
||
|
||
`hex transcribe <file>` is the other client, and uses both paths: it talks to
|
||
the running app through its discovery file, and with no Hex running spawns
|
||
its own executable as `hex service --embedded` exactly as above, checking that
|
||
the ready line's `pid` is its child and its `url` is `http://127.0.0.1:<port>`.
|
||
It sends the file's decoded audio as 16 kHz mono WAV pieces of at most 30 s to
|
||
`POST /transcriptions`, prepares a missing model through
|
||
`POST /models/{id}/prepare` only after a terminal prompt, and closes the
|
||
child's stdin (killing it after three seconds) on every exit. A discovery file
|
||
for a live Hex with the wrong `apiVersion` or token is an error, not a reason
|
||
to spawn a second service.
|
||
|
||
## Wire Protocol (API 2)
|
||
|
||
```text
|
||
GET /health
|
||
GET /capabilities
|
||
GET /models
|
||
POST /models/{id}/prepare SSE preparation progress, then ok or error
|
||
POST /transcriptions audio/wav body, final transcript response
|
||
```
|
||
|
||
`POST /transcriptions` requires a `model` query parameter. Preparation and
|
||
transcription accept an optional `language` query parameter, defaulting to `en`.
|
||
`GET /models?language=...` uses that language for readiness metadata, not to filter
|
||
the returned catalog. Transcription requires `Content-Length`; chunked transfer
|
||
encoding is not supported.
|
||
|
||
`POST /transcriptions` accepts an optional `process=global` query parameter on
|
||
the running desktop app's authenticated API (Labs › Process API transcripts,
|
||
off by default; `true` and `1` are aliases). When present, the response keeps
|
||
`transcript` and `durationMs` and adds `processed`: the same text a paste with
|
||
no matching app or website mode would use, after global replacements, Global
|
||
mode corrections, the optional OpenCode rewrite, and Global's selected
|
||
transformations. Processing is best-effort and deadline-bounded like a paste,
|
||
so a failed stage keeps its input and the request still succeeds. It never
|
||
pastes, records History, updates the last dictation, or touches the clipboard.
|
||
Any other value, or a repeated parameter, answers `400 invalid-request`.
|
||
`hex service` and `--embedded` install no processor and answer `409
|
||
processing-unavailable`, even while the lab is off; with a processor installed
|
||
and the lab off the answer is `403 processing-disabled`. Processing runs one at
|
||
a time across the local API, so while another holds it the answer is `429
|
||
queue-full`; all three are decided before the audio body is read.
|
||
`GET /capabilities` always includes `outputProcessing`, true only when a
|
||
processor is installed and the lab is on.
|
||
|
||
`POST /transcriptions` also accepts an optional `timestamps` query parameter:
|
||
`none`, `segment`, or `word` (Labs › Timestamps in API transcripts, off by
|
||
default). The running desktop app reads the lab live, so its next request sees
|
||
a toggle. A `hex service` process without the app reads the saved
|
||
`labs.api_timestamps` once, at startup, and the embedded helper (the SDK's
|
||
direct child) loads no settings and reads the saved value once, on its first
|
||
request. Neither re-reads it, so changing the lab reaches a running helper only
|
||
after it restarts. Absent means `none`. Any other value, or a repeated
|
||
parameter, answers `400 invalid-request`. With the lab off, `segment` and
|
||
`word` answer `403 timestamps-disabled` before the audio body is read; `none`
|
||
and an absent parameter are unaffected. An unknown value is `400` whether or
|
||
not the lab is on.
|
||
|
||
```typescript
|
||
// The response with timestamps=segment, on top of today's fields.
|
||
{
|
||
transcript: string;
|
||
durationMs: number;
|
||
timestampGranularity: "segment";
|
||
segments: readonly { text: string; startMs: number; endMs: number }[];
|
||
}
|
||
|
||
// timestamps=word adds the model's own words.
|
||
{
|
||
transcript: string;
|
||
durationMs: number;
|
||
timestampGranularity: "word";
|
||
segments: readonly { text: string; startMs: number; endMs: number }[];
|
||
words: readonly { text: string; startMs: number; endMs: number }[];
|
||
}
|
||
```
|
||
|
||
Times are whole milliseconds from the start of the submitted audio, never
|
||
relative to the model's internal chunk: a long clip transcribed in chunks has
|
||
each chunk's offset added once, from its sample position in the clip, so
|
||
chunks that held no speech shift nothing. An end that padding (the minimum
|
||
clip duration, or a model's trailing silence) pushed past the decoded audio
|
||
is clamped to `durationMs`. A GGUF model answers a clip with no speech with
|
||
`200`, an empty `transcript`, and empty `segments` and `words`; Apple Speech
|
||
reports no speech instead, as `500 transcription-failed`, whether or not
|
||
timings were asked for.
|
||
|
||
The times are the model's own: nothing estimates them from text length or
|
||
chunk boundaries, and a word request is never answered with segments. When
|
||
the selected model or backend cannot produce the requested level the answer
|
||
is `422 unsupported-timestamps`, after the model is resolved and before
|
||
inference. A GGUF model's own `capabilities.max_timestamp_kind` metadata
|
||
decides, so a model that advertises only segments answers `segment` and
|
||
refuses `word`; Apple Speech gives segments only. The decision is the same on
|
||
both inference paths: the running app's shared dictation engine when it holds
|
||
the requested model, and the service's own copy otherwise. `none` adds no
|
||
field, and the response is byte-for-byte what it was before. With
|
||
`process=global`, the segments and words describe the raw transcript and
|
||
`processed` stays a separate field. `/streams` and `/dictations` are unchanged
|
||
and do not offer this parameter. `API_VERSION` stays `2`.
|
||
|
||
`POST /transcriptions` accepts completed PCM WAV audio. The service reads
|
||
request bodies incrementally into a bounded buffer, then validates WAV metadata
|
||
before allocating decoded samples and running inference. It downmixes and
|
||
resamples internally. API 2 accepts integer PCM and 32-bit float WAV at 8 kHz
|
||
through 192 kHz with one through eight channels. Uploads are capped at 64 MiB;
|
||
source frames and normalized 16 kHz mono Float32 samples are each capped at
|
||
16,777,216 (64 MiB of Float32 samples). The frame cap permits about 5.8 minutes at
|
||
48 kHz or 17.5 minutes at 16 kHz, before other limits. Headers have a two-second
|
||
deadline and uploads a thirty-second deadline. These limits are not a total
|
||
process-memory cap.
|
||
|
||
The HTTP transcription path does not save uploaded WAVs. The current executable
|
||
still initializes shared process logging and inherits native diagnostic flags
|
||
such as `TRANSCRIBE_DUMP_DIR`, which can write audio-derived tensors. Do not treat
|
||
this as an audited no-persistence boundary; embedded diagnostic isolation remains
|
||
part of the distribution plan.
|
||
|
||
There is no separate duration cutoff. One upload owns the audio-memory admission
|
||
slot until its queued or active inference releases the clip. Absolute header
|
||
and upload deadlines, byte and normalized-sample budgets, and typed resource
|
||
refusal bound admission from defective local clients. Detected client cancellation
|
||
or service shutdown suppresses results, but native inference may finish internally.
|
||
The ready SDK transcriber instead closes its owned helper on in-flight cancellation
|
||
and waits for cleanup before rejecting; low-level request abortion alone does not
|
||
prove native work stopped.
|
||
|
||
### Health
|
||
|
||
```typescript
|
||
export interface Health {
|
||
version: string;
|
||
apiVersion: "2";
|
||
}
|
||
|
||
export interface Capabilities {
|
||
audioFormats: readonly ["audio/wav"];
|
||
partialTranscripts: false;
|
||
serviceCapture: false;
|
||
outputProcessing: boolean;
|
||
developerControl: false;
|
||
}
|
||
```
|
||
|
||
These capability values describe the embedded helper. The desktop local API can
|
||
advertise capture and debug-only developer control. The SDK exposes
|
||
`serviceCapture` as a boolean and omits the native-only `developerControl` field.
|
||
`outputProcessing` is `false` from the embedded helper (it installs no
|
||
processor) and true from the running app only while Labs › Process API
|
||
transcripts is on.
|
||
|
||
### Models
|
||
|
||
```typescript
|
||
export type ModelId =
|
||
| "parakeet_unified_en"
|
||
| "parakeet_v2"
|
||
| "parakeet_v3"
|
||
| "orukeet"
|
||
| "fluid_speech_pro"
|
||
| "whisper_large_v3_turbo"
|
||
| "qwen3_asr06_b"
|
||
| "qwencleo_asr_1_7b"
|
||
| "ivrit_whisper_large_v3_turbo"
|
||
| "sense_voice_small"
|
||
| "cohere_transcribe"
|
||
| "underdog_bark_3_pro"
|
||
| "underdog_bark_3_clean"
|
||
| "parakeet_ultra"
|
||
| "nemotron_3_5_asr"
|
||
| "gigaam_v3"
|
||
| "nb_whisper_medium"
|
||
| "kb_whisper_medium"
|
||
| "granite_speech_5"
|
||
| "apple_speech"
|
||
// An imported or found model, listed while it is ready.
|
||
| `custom-${string}`;
|
||
|
||
export interface ModelInfo {
|
||
id: ModelId;
|
||
name: string;
|
||
installed: boolean;
|
||
verified: boolean;
|
||
managed: boolean;
|
||
downloadBytes: number | null;
|
||
languages: readonly string[];
|
||
supportsLanguageDetection: boolean;
|
||
// download | hex | imported | found: owner/name@abcdef1 (another app's
|
||
// Hugging Face download, used where it is). Older hosts omit it.
|
||
source?: string;
|
||
}
|
||
|
||
export type ModelProgress =
|
||
| { type: "downloading"; downloadedBytes: number; totalBytes: number }
|
||
| { type: "verifying" }
|
||
| { type: "loading" };
|
||
```
|
||
|
||
`ModelId` is the SDK's closed set of identifiers, not an availability guarantee.
|
||
The native catalog currently excludes `apple_speech`, and preparation and
|
||
transcription reject it. The retained Apple Speech implementation and
|
||
system-managed metadata do not make it a supported service model.
|
||
|
||
For the available GGUF models, preparation means download if needed, checksum
|
||
verification (or reuse of a matching verification receipt), strict-Metal load, and
|
||
prewarm. `managed` means system-managed, not downloaded by Hex, so these models
|
||
report `managed: false`. Preparation activates the helper's warm model but does
|
||
not change full `Hex.app` settings. The service serializes preparation, publishes
|
||
only verified replacement files, and preserves the previous warm runtime if
|
||
candidate loading fails. A verified download can remain installed after a load
|
||
failure; artifact publication and runtime activation are separate steps.
|
||
|
||
Preparing a model that is already loaded, including the desktop app's
|
||
dictation model, reruns the prewarm: an inference over 1.5 seconds of silence
|
||
(the minimum clip length) that, in the app, yields to dictation like meeting
|
||
chunks. ggml keeps the Metal weights resident for three minutes after a run;
|
||
then macOS may compress or swap them, and the next transcription pays to
|
||
restore them. A host that records its own audio can prepare when recording
|
||
starts, so the restore happens while it records rather than after. The
|
||
prewarm is skipped when the model ran in the last minute. When the app's
|
||
dictation model is the one requested, preparation uses it and never loads a
|
||
second copy; if dictation is recording or transcribing, the model is already
|
||
warm and preparation succeeds at once.
|
||
|
||
### Ready Transcriber
|
||
|
||
SDK `0.3.0` adds a model-bound operation to both entrypoints:
|
||
|
||
```ts
|
||
import { create } from "@kitlangton/hex"
|
||
|
||
const transcriber = await create({
|
||
command: ["/path/to/hex-service", "service", "--embedded"],
|
||
model: "parakeet_unified_en",
|
||
language: "en",
|
||
onProgress,
|
||
})
|
||
try {
|
||
const result = await transcriber.transcribe(wav, { signal })
|
||
input.insert(result.transcript)
|
||
} finally {
|
||
await transcriber.close()
|
||
}
|
||
```
|
||
|
||
Creation resolves after preparation. Reuse the transcriber across recordings;
|
||
close when the host no longer needs it. Cancelling an in-flight request closes the
|
||
helper, so subsequent work needs a new transcriber. An already-aborted request
|
||
does not close a healthy transcriber. Failed cleanup is reported, not hidden.
|
||
Effect provides the same ready shape with scope-owned cleanup instead of `close()`.
|
||
|
||
Progress callbacks describe model setup, not inference percentage. Transcription
|
||
returns only final text; partial transcripts and numerical transcription progress
|
||
are not exposed by this protocol.
|
||
|
||
### Low-Level TypeScript Surface
|
||
|
||
The [SDK README](../../sdk/typescript/README.md) documents host creation and
|
||
client usage. The [public types](../../sdk/typescript/src/types.ts) define the
|
||
complete client, audio, progress, and result contracts; this spec owns the wire
|
||
protocol and lifecycle guarantees rather than a second copy of those interfaces.
|
||
|
||
The common host flow is explicit about preparation:
|
||
|
||
```typescript
|
||
await client.models.prepare("parakeet_unified_en", { onProgress });
|
||
|
||
// The host records, meters, and encodes the clip.
|
||
const result = await client.transcribe({
|
||
audio: { data: wav, contentType: "audio/wav" },
|
||
model: "parakeet_unified_en",
|
||
signal,
|
||
});
|
||
|
||
input.insert(result.transcript);
|
||
```
|
||
|
||
The `@kitlangton/hex/effect` entrypoint provides scoped helper acquisition,
|
||
schema-backed typed errors, Effect operations, an Effect `Stream` for model
|
||
preparation progress, and a `Layer` for dependency injection. Closing the scope
|
||
closes the host lease and bounds exact-child termination.
|
||
|
||
`transcribe()` never downloads a model. It returns `model-not-ready` when the
|
||
artifact has not been prepared. It may transparently reload an installed but
|
||
cold model because model residency is transient.
|
||
|
||
## First Embedded Release Non-Goals
|
||
|
||
- No service-owned microphone capture, levels, hotkeys, paste, or clipboard.
|
||
- No partial transcripts.
|
||
- No compressed audio codecs.
|
||
- No OpenCode rewrite profiles.
|
||
- No non-localhost access.
|
||
- No ordinary browser bootstrap.
|
||
- No cross-host warm-runtime sharing or service election.
|
||
- No long-form segmentation, speaker turns, resumability, or search until a
|
||
real consumer requires them.
|