* add fluid speech pro to the speech model catalog FluidVoice's Fluid Speech Pro, a Parakeet TDT 0.6B v3 retrain that also declares Moondream's Parakeet Ultra as a base, for the same 25 European languages. TalkMate converted it to a Q8_0 transcribe.cpp GGUF at d59dc79 (741,819,456 bytes, sha256 c3848e98...); the file says parakeet/tdt-0.6b-fluid-speech-pro and announces its extra pre-Conformer block as stt.parakeet.encoder.pre_block_kernel, so it needs a transcribe.cpp that reads enc.pre_block. Like Parakeet v3 it takes no language hint, detects the language under Auto, does not stream, and is never recommended or badged. Its card publishes no LibriSpeech test-clean figure, so it carries no single word error rate and no benchmark record; its per-language FLEURS figures and the CC BY 4.0 credit are in its details. * bump(transcribe.cpp): 295c7e6, fluid speech pro's pre-conformer block Fluid Speech Pro's GGUF announces stt.parakeet.encoder.pre_block_kernel and carries eight enc.pre_block.* tensors: a residual FrameCleaner block between the subsampler and Conformer layer 0. The old pin's graph never read them, so the model loaded as plain Parakeet v3 and its encoder was wrong. Both transcribe-cpp crates move to 295c7e601 on hex/combined-fluid, a fast-forward of bcb639f that runs the block on pre_encode before xscaling when the KV is present, and fails the load rather than skipping tensors when it is only partly there. Cargo.lock changes only those two sources (`getrandom`'s tempfile pointer was reverted to what it was), and nix/package.nix carries the hash from a failed `nix build .#hex`, then a full one whose check phase passed. `hex model install fluid-speech-pro` followed by `hex model transcribe` on whisper.cpp's jfk.wav prints the expected sentence, as does Parakeet v3 on the same clip. * fluid speech pro: snake_case id, FLEURS-only figures, blurb, notices
19 KiB
Hex Local Transcription Service and SDK
Status: The internal service and direct-child mode are implemented.
@kitlangton/hex@0.3.0 publishes ready transcribers through Promise and Effect
entrypoints. Local Electron integration has been exercised, but the native helper
package remains private and is not included as a client dependency. An explicit
helper command is still required; signed consumer distribution remains unverified.
Authority: This document describes the embedded service protocol and labels
proposed distribution separately. The SDK README
is the current consumer guide. Delivery scope, evidence, and acceptance gates live in
../plans/typescript-sdk.md.
Product Boundary
HEX Service owns local transcription models and inference. Host applications own recording.
Host application HEX Service
---------------- -----------
microphone permission service discovery and authentication
device selection model download and verification
capture start/stop strict-Metal load and prewarm
levels and waveform PCM WAV resampling and inference
capture UX ----------> final raw transcript
settings and model choice
The host's signed app owns the microphone grant. A correctly configured packaged host prompts under its own identity, not Hex; a development Electron runtime may appear as Electron. The embedded helper never opens the microphone.
Decisions
| Decision | Reason |
|---|---|
| Host owns microphone capture | Correct TCC identity, intuitive permission prompt, no shared permission broker. |
| Service owns models and inference | Hosts share verified model artifacts while each helper owns its warm runtime. |
| Completed audio goes over localhost | Simple final-only contract; no partial transcripts or bidirectional session protocol. |
| Model preparation is explicit | A transcription call never hides a large network download. |
| Installed-but-cold models may reload automatically | A request for a different model in the same helper replaces its warm runtime. |
| No product duration limit | Hotkey capture and service transcription end explicitly; operational resource exhaustion remains a typed failure. |
| Raw transcripts only | The host owns meaning, rewriting, and insertion. The running desktop app's authenticated API can apply Global mode's output processing when a request explicitly asks with process=global (opt-in); the embedded helper has no processor. |
| Trusted Node/Electron main process | Ordinary browser code and untrusted renderers must not receive native process authority or service credentials. |
| Host-spawned helper | The host owns the process, endpoint, and lifetime instead of discovering an independently elected service. |
| Shared model artifacts, private warm runtime | Hosts reuse verified downloads without coupling their model choice or process lifecycle. |
Proposed Embedded Distribution (Not Yet Shipping)
The target SDK will declare signed, notarized helpers as optional
architecture-specific packages and select the current platform package
automatically. Applications will call create() without locating an executable.
On explicit host use, the SDK will spawn the included helper directly as a child
without an npm install script, administrator access, login-item approval, Dock icon,
or foreground window. The host must not launch the helper through Launch Services.
Because the embedded service does not capture audio, it does not need microphone
TCC identity. Full Hex.app already owns its global-hotkey capture and
transcription pipeline; desktop connect() can also use the app's microphone
through its separate capture API.
Verified model artifacts remain shared per user under Application Support.
Each embedded helper has its own warm model and inference queue. Multiple hosts
may reuse the same verified file, but they do not share Metal allocations or
runtime selection. Inside Hex.app, a clip of up to 30 seconds for the GGUF
model dictation has loaded runs on dictation's model between dictations, so
the app holds one copy; another model, or a longer clip, uses the service's own,
and a longer clip's copy is released when dictation holds the model. The current cache is filename-based; immutable revision-safe
storage for independently versioned helpers remains planned.
Embedded Startup and Authentication
The host spawns hex-service service --embedded with piped stdin and stdout.
The helper binds 127.0.0.1:0 and writes exactly one JSON line to stdout after
the authenticated API is ready:
{
"type": "ready",
"url": "http://127.0.0.1:49731",
"token": "hex_a1b2c3...",
"apiVersion": "2",
"pid": 12345
}
The host keeps the child's stdin open as its lifetime lease. EOF requests a clean shutdown; the helper force-exits after a five-second grace period if native work does not return. Embedded mode writes no shared discovery file and acquires no global service lock, so two host applications can run independent helpers.
The current executable starts lease monitoring after local API initialization, not at process entry. Startup-failure and descendant-process cleanup remain native-distribution acceptance gates in the delivery plan.
Every request requires Authorization: Bearer <token>. The token is delivered
only over the child's stdout pipe. The server deliberately sends no CORS
headers. A parsed request with a missing or incorrect bearer credential returns
an empty 401 response; malformed HTTP can be rejected before authentication.
The standalone Hex application may still publish owner-only discovery for its
own existing local API. Embedded create() uses the child handshake; desktop
connect() remains a separate discovery-based path.
hex transcribe <file> is the other client, and uses both paths: it talks to
the running app through its discovery file, and with no Hex running spawns
its own executable as hex service --embedded exactly as above, checking that
the ready line's pid is its child and its url is http://127.0.0.1:<port>.
It sends the file's decoded audio as 16 kHz mono WAV pieces of at most 30 s to
POST /transcriptions, prepares a missing model through
POST /models/{id}/prepare only after a terminal prompt, and closes the
child's stdin (killing it after three seconds) on every exit. A discovery file
for a live Hex with the wrong apiVersion or token is an error, not a reason
to spawn a second service.
Wire Protocol (API 2)
GET /health
GET /capabilities
GET /models
POST /models/{id}/prepare SSE preparation progress, then ok or error
POST /transcriptions audio/wav body, final transcript response
POST /transcriptions requires a model query parameter. Preparation and
transcription accept an optional language query parameter, defaulting to en.
GET /models?language=... uses that language for readiness metadata, not to filter
the returned catalog. Transcription requires Content-Length; chunked transfer
encoding is not supported.
POST /transcriptions accepts an optional process=global query parameter on
the running desktop app's authenticated API (Labs › Process API transcripts,
off by default; true and 1 are aliases). When present, the response keeps
transcript and durationMs and adds processed: the same text a paste with
no matching app or website mode would use, after global replacements, Global
mode corrections, the optional OpenCode rewrite, and Global's selected
transformations. Processing is best-effort and deadline-bounded like a paste,
so a failed stage keeps its input and the request still succeeds. It never
pastes, records History, updates the last dictation, or touches the clipboard.
Any other value, or a repeated parameter, answers 400 invalid-request.
hex service and --embedded install no processor and answer 409 processing-unavailable, even while the lab is off; with a processor installed
and the lab off the answer is 403 processing-disabled. Processing runs one at
a time across the local API, so while another holds it the answer is 429 queue-full; all three are decided before the audio body is read.
GET /capabilities always includes outputProcessing, true only when a
processor is installed and the lab is on.
POST /transcriptions also accepts an optional timestamps query parameter:
none, segment, or word (Labs › Timestamps in API transcripts, off by
default). The running desktop app reads the lab live, so its next request sees
a toggle. A hex service process without the app reads the saved
labs.api_timestamps once, at startup, and the embedded helper (the SDK's
direct child) loads no settings and reads the saved value once, on its first
request. Neither re-reads it, so changing the lab reaches a running helper only
after it restarts. Absent means none. Any other value, or a repeated
parameter, answers 400 invalid-request. With the lab off, segment and
word answer 403 timestamps-disabled before the audio body is read; none
and an absent parameter are unaffected. An unknown value is 400 whether or
not the lab is on.
// The response with timestamps=segment, on top of today's fields.
{
transcript: string;
durationMs: number;
timestampGranularity: "segment";
segments: readonly { text: string; startMs: number; endMs: number }[];
}
// timestamps=word adds the model's own words.
{
transcript: string;
durationMs: number;
timestampGranularity: "word";
segments: readonly { text: string; startMs: number; endMs: number }[];
words: readonly { text: string; startMs: number; endMs: number }[];
}
Times are whole milliseconds from the start of the submitted audio, never
relative to the model's internal chunk: a long clip transcribed in chunks has
each chunk's offset added once, from its sample position in the clip, so
chunks that held no speech shift nothing. An end that padding (the minimum
clip duration, or a model's trailing silence) pushed past the decoded audio
is clamped to durationMs. A GGUF model answers a clip with no speech with
200, an empty transcript, and empty segments and words; Apple Speech
reports no speech instead, as 500 transcription-failed, whether or not
timings were asked for.
The times are the model's own: nothing estimates them from text length or
chunk boundaries, and a word request is never answered with segments. When
the selected model or backend cannot produce the requested level the answer
is 422 unsupported-timestamps, after the model is resolved and before
inference. A GGUF model's own capabilities.max_timestamp_kind metadata
decides, so a model that advertises only segments answers segment and
refuses word; Apple Speech gives segments only. The decision is the same on
both inference paths: the running app's shared dictation engine when it holds
the requested model, and the service's own copy otherwise. none adds no
field, and the response is byte-for-byte what it was before. With
process=global, the segments and words describe the raw transcript and
processed stays a separate field. /streams and /dictations are unchanged
and do not offer this parameter. API_VERSION stays 2.
POST /transcriptions accepts completed PCM WAV audio. The service reads
request bodies incrementally into a bounded buffer, then validates WAV metadata
before allocating decoded samples and running inference. It downmixes and
resamples internally. API 2 accepts integer PCM and 32-bit float WAV at 8 kHz
through 192 kHz with one through eight channels. Uploads are capped at 64 MiB;
source frames and normalized 16 kHz mono Float32 samples are each capped at
16,777,216 (64 MiB of Float32 samples). The frame cap permits about 5.8 minutes at
48 kHz or 17.5 minutes at 16 kHz, before other limits. Headers have a two-second
deadline and uploads a thirty-second deadline. These limits are not a total
process-memory cap.
The HTTP transcription path does not save uploaded WAVs. The current executable
still initializes shared process logging and inherits native diagnostic flags
such as TRANSCRIBE_DUMP_DIR, which can write audio-derived tensors. Do not treat
this as an audited no-persistence boundary; embedded diagnostic isolation remains
part of the distribution plan.
There is no separate duration cutoff. One upload owns the audio-memory admission slot until its queued or active inference releases the clip. Absolute header and upload deadlines, byte and normalized-sample budgets, and typed resource refusal bound admission from defective local clients. Detected client cancellation or service shutdown suppresses results, but native inference may finish internally. The ready SDK transcriber instead closes its owned helper on in-flight cancellation and waits for cleanup before rejecting; low-level request abortion alone does not prove native work stopped.
Health
export interface Health {
version: string;
apiVersion: "2";
}
export interface Capabilities {
audioFormats: readonly ["audio/wav"];
partialTranscripts: false;
serviceCapture: false;
outputProcessing: boolean;
developerControl: false;
}
These capability values describe the embedded helper. The desktop local API can
advertise capture and debug-only developer control. The SDK exposes
serviceCapture as a boolean and omits the native-only developerControl field.
outputProcessing is false from the embedded helper (it installs no
processor) and true from the running app only while Labs › Process API
transcripts is on.
Models
export type ModelId =
| "parakeet_unified_en"
| "parakeet_v2"
| "parakeet_v3"
| "orukeet"
| "fluid_speech_pro"
| "whisper_large_v3_turbo"
| "qwen3_asr06_b"
| "qwencleo_asr_1_7b"
| "ivrit_whisper_large_v3_turbo"
| "sense_voice_small"
| "cohere_transcribe"
| "underdog_bark_3_pro"
| "underdog_bark_3_clean"
| "parakeet_ultra"
| "nemotron_3_5_asr"
| "gigaam_v3"
| "nb_whisper_medium"
| "kb_whisper_medium"
| "granite_speech_5"
| "apple_speech"
// An imported or found model, listed while it is ready.
| `custom-${string}`;
export interface ModelInfo {
id: ModelId;
name: string;
installed: boolean;
verified: boolean;
managed: boolean;
downloadBytes: number | null;
languages: readonly string[];
supportsLanguageDetection: boolean;
// download | hex | imported | found: owner/name@abcdef1 (another app's
// Hugging Face download, used where it is). Older hosts omit it.
source?: string;
}
export type ModelProgress =
| { type: "downloading"; downloadedBytes: number; totalBytes: number }
| { type: "verifying" }
| { type: "loading" };
ModelId is the SDK's closed set of identifiers, not an availability guarantee.
The native catalog currently excludes apple_speech, and preparation and
transcription reject it. The retained Apple Speech implementation and
system-managed metadata do not make it a supported service model.
For the available GGUF models, preparation means download if needed, checksum
verification (or reuse of a matching verification receipt), strict-Metal load, and
prewarm. managed means system-managed, not downloaded by Hex, so these models
report managed: false. Preparation activates the helper's warm model but does
not change full Hex.app settings. The service serializes preparation, publishes
only verified replacement files, and preserves the previous warm runtime if
candidate loading fails. A verified download can remain installed after a load
failure; artifact publication and runtime activation are separate steps.
Preparing a model that is already loaded, including the desktop app's dictation model, reruns the prewarm: an inference over 1.5 seconds of silence (the minimum clip length) that, in the app, yields to dictation like meeting chunks. ggml keeps the Metal weights resident for three minutes after a run; then macOS may compress or swap them, and the next transcription pays to restore them. A host that records its own audio can prepare when recording starts, so the restore happens while it records rather than after. The prewarm is skipped when the model ran in the last minute. When the app's dictation model is the one requested, preparation uses it and never loads a second copy; if dictation is recording or transcribing, the model is already warm and preparation succeeds at once.
Ready Transcriber
SDK 0.3.0 adds a model-bound operation to both entrypoints:
import { create } from "@kitlangton/hex"
const transcriber = await create({
command: ["/path/to/hex-service", "service", "--embedded"],
model: "parakeet_unified_en",
language: "en",
onProgress,
})
try {
const result = await transcriber.transcribe(wav, { signal })
input.insert(result.transcript)
} finally {
await transcriber.close()
}
Creation resolves after preparation. Reuse the transcriber across recordings;
close when the host no longer needs it. Cancelling an in-flight request closes the
helper, so subsequent work needs a new transcriber. An already-aborted request
does not close a healthy transcriber. Failed cleanup is reported, not hidden.
Effect provides the same ready shape with scope-owned cleanup instead of close().
Progress callbacks describe model setup, not inference percentage. Transcription returns only final text; partial transcripts and numerical transcription progress are not exposed by this protocol.
Low-Level TypeScript Surface
The SDK README documents host creation and client usage. The public types define the complete client, audio, progress, and result contracts; this spec owns the wire protocol and lifecycle guarantees rather than a second copy of those interfaces.
The common host flow is explicit about preparation:
await client.models.prepare("parakeet_unified_en", { onProgress });
// The host records, meters, and encodes the clip.
const result = await client.transcribe({
audio: { data: wav, contentType: "audio/wav" },
model: "parakeet_unified_en",
signal,
});
input.insert(result.transcript);
The @kitlangton/hex/effect entrypoint provides scoped helper acquisition,
schema-backed typed errors, Effect operations, an Effect Stream for model
preparation progress, and a Layer for dependency injection. Closing the scope
closes the host lease and bounds exact-child termination.
transcribe() never downloads a model. It returns model-not-ready when the
artifact has not been prepared. It may transparently reload an installed but
cold model because model residency is transient.
First Embedded Release Non-Goals
- No service-owned microphone capture, levels, hotkeys, paste, or clipboard.
- No partial transcripts.
- No compressed audio codecs.
- No OpenCode rewrite profiles.
- No non-localhost access.
- No ordinary browser bootstrap.
- No cross-host warm-runtime sharing or service election.
- No long-form segmentation, speaker turns, resumability, or search until a real consumer requires them.