24 KiB
Captures In Hex
Status: steps 2 and 3, and the input half of step 4 with the Scrawl pane,
are built (step 4's half on branch scrawl-in-hex) as an experiment behind a build
option (off in every public build) on Scrawl's real engine, the
scrawl-capture crate imported into crates/capture (see
Built so far); nothing ships, and Kit decides later whether
the fold happens. This is the Hex side of ROADMAP "Fold Scrawl Into Hex": how
Hex hosts Scrawl's capture engine without weakening dictation.
Built so far
Step 2: crates/capture is Scrawl's crates/capture at kitlangton/scrawl
e351ddb (branch engine/split), copied without history, package name
scrawl-capture kept, with private test values replaced by synthetic ones,
cargo fmt, and clippy fixes that change no behaviour. Hex depends on it by
path. It sits in [workspace] exclude, so cargo build, cargo test, and
--workspace never build it; it compiles only as the capture feature's
macOS dependency, sharing the root Cargo.lock and target/. On macOS
scripts/check and CI name it (-p "$(cargo pkgid)" -p scrawl-capture --features capture) to lint and test it. Scrawl's own changes now come over
by hand.
Step 3, capture_host on that crate:
hex capture start|stop|cancel|status // `--features capture`; POST /control `scrawl-capture`
└── listener thread -> capture_host::CaptureHost (exists only with the experiment on)
├── start -> owner-only <captures folder>/<unix ms>.scrawl -> reply at once
│ ├── DictationAudio::open_capture_sink(bounded 512 chunks, since = t0)
│ └── capture-audio thread -> Engine::start(Start { t0_boot_ns, ActiveDisplay,
│ AudioIn::External, bundle_dir }) (blocks: target lookup, ScreenCaptureKit)
├── capture-audio thread -> Timeline: 16 kHz, gaps padded, start offset kept
├── start while two stopped captures are still writing -> busy
├── stop -> close_capture_sink(through) (≤ 100 ms wait) -> reply with the bundle path
│ ├── capture-audio thread -> Capture::stop: screen, window log, and input end now
│ └── capture-finish thread, one at a time, ≤ 2 waiting ->
│ SealedCapture::finish(Some(RecordedAudio { start_offset_ms, gaps }))
│ ├── HexTranscriber -> parakeet::transcribe_with_words, 30 s chunks,
│ │ each a foreground job on the dictation engine's loaded model
│ ├── Scrawl writes recording.mp4, audio.wav, frames/, scrawl.json, summary.md
│ │ (its own files 0600) -> Finished { words, gaps, missing_transcript }
│ └── Hex makes every file 0600 and folder 0700 (recording.mp4, folders)
└── cancel -> Capture::cancel, remove the bundle
- Hex's seam is three small traits in
src/capture_host.rs,CaptureEngine,Recording, andSealed, implemented forscrawl_capture::capture::{Engine, Capture, SealedCapture}so the host's tests run without a screen. Hex implements the crate'sHost(boundedtry_sendevents, logs and metrics to tracing),ConfigSource, andTranscriber.hex capture statusshows the last capture's words, audio gaps, and why it has no transcript. ConfigSourceis the engine's only settings source; nothing on Hex's path reads~/.config/scrawl/config.json(the crate reads Scrawl's config and logs only afterenable_scrawl_host_defaultsorenable_file_log, which Hex never calls). Hex has no capture settings yet, soHexConfigsets Scrawl's defaults deliberately: auto-crop on (display captures) and theagentContextsummary format off.- The engine is
Engine::new(..).with_resources(SharedResources::none()); Hex never callsinstall_shared_resources, so no ingest server (47777), delivery retries, or global shortcuts.Engine::newroutes the crate's process-wide log toHost::logwhen no sink is set, so each line reaches Hex's tracing once and nothing goes to~/Library/Logs/Scrawl.Engine::newandEngine::startcall no TIS/TSM (layout-aware shortcuts are resolved only ininstall_shared_resources). - One capture records at a time.
stopseals it and the capture-audio thread callsCapture::stopat once, ending ScreenCaptureKit, the window log, and input; theSealedCapturethen waits for the one writer, at most two at a time. WithSharedResources::none()each capture has its own input telemetry, so a capture recording while an earlier one is written shares nothing with it. - Bundles go to
<captures folder>/<unix ms>.scrawl, Scrawl's own folder unlesscapture.foldernames another (step 3 used<support>/captures): Scrawl's ownscrawl.jsonformat, words in ms on the capture's clock. hex dev capture-replay <wav>…runs the sameCaptureHostheadlessly on a fresh transcription engine, feeding WAV files instead of the microphone; it records the screen too (debug builds only, like everyhex devcommand).- Views read one
CaptureSnapshot(capture_host::snapshot, woken bycapture_host::watch): the recording capture (bundle, start, whether the screen started), stopped captures with the engine's latest stage, and the last settled one (done, failed, cancelled, with words and gaps). The host publishes it on every change; engine progress moves a capture on but never settles it. The capsule, the router, andhex capture listsubscribe here rather than to the host. - Routing and delivery (step 5), a cabled iPhone, the capsule, the on-screen
picker, and the ink and laser overlays are built; see those sections and
docs/features/capture.md. Not built yet: iPhone ingest (ScrawlKit pairing), moving the ingest token, and a listener key path for the open list (⌥⇧V, typing, arrows, Return).
Step 4, first half (the shortcut, input, Escape) with the pane, import, and coexistence from step 5:
settings.json capture.* (CaptureSettings, cfg hex_capture; other builds keep the raw key)
├── enabled (off) · hotkey (⌥⇧S) · transcriber (parakeet|apple) · folder (None = Scrawl's)
│ · media_days / bundle_days (Scrawl's Week preset: 7 / 30)
├── every key in `hex settings` (control/settings.rs CAPTURE_KEYS)
└── load -> import once (migration `scrawl-capture-settings`, not in MIGRATIONS):
~/.config/scrawl/config.json + Scrawl's shortcut file, read-only
recognition loop, every iteration
├── CaptureHost::reconcile -> start/stop the host as capture_on() changes
│ (experiment && capture.enabled); watch_scrawl polls Scrawl.app every 2 s
├── runtime_hotkeys() drops capture while Scrawl.app runs
├── fresh ⌥⇧S press (CaptureShortcut) -> CaptureHost::toggle: start | stop
│ (a press within SHORTCUT_DEBOUNCE, 400 ms, of the start is ignored)
└── Esc, nothing else recording, a capture recording -> cancel it first
suppression event tap (the one session tap)
├── observation mask + mouse down/up/drag/scroll when capture_available()
├── wants_input() (a capture records) -> report_capture_input -> TAP_INPUT
│ (bounded 1 024, drops counted) -> LiveInput::forward -> HostInput
│ (Moved / ModifiersChanged stand in for Scrawl's 8 ms poller)
└── capture-only events end there; keys, flags, and downs continue to the listener
crates/capture hosted_input.rs: Scrawl's ink and click rules as pure functions
(is_pin_click, drew_before, is_real_ink, on_own_window); Scrawl's own key tap
calls the same key_input / key_name. after_shortcut(): Shift must come up once.
Settings › Experiments › Scrawl (rune_ui/scrawl.rs; PlatformState.scrawl is
Some while capture_available(), so the pane exists only then)
├── Capture with Hex · Shortcut · Screen Recording · Transcriber
├── Captures folder (Reveal | choose) · Keep (retention presets)
├── "Scrawl is running" + Use Hex for captures -> quit_scrawl (NSRunningApplication)
└── "Hex copied Scrawl's settings once…" when the import read a file
capture-retention thread -> retention::run_pass(capture.* policy): 90 s, then hourly;
skipped while a capture is unfinished or Scrawl.app runs
- Deviations from Scrawl: the shortcut acts on the press (Scrawl's Carbon hotkey also starts on press, but Hex has no "start on key-up" pending state), a press within 400 ms of the start is ignored rather than stopping a capture that has barely begun, and pointer position comes from tap events rather than an 8 ms poller.
- The pane exists only in
cfg(hex_capture)builds: a public build never compiles the module, its strings, or its scenes, and a feature build shows it only while the experiment is on. Its scenes (mac-scrawl*) render only from a feature build, so they have no goldens (goldens run on Linux). - iPhone pairing is omitted, not shown disabled: it needs the ingest server, which Hex doesn't run yet. A cabled iPhone needs no row.
Features and gating
- Hex's Cargo feature
capture(off by default) turns on the optional macOS-only dependenciesscrawl-capture(default-features = false, features = ["screen", "helper"]) andanyhow(the crate's traits returnanyhow::Result). build.rssetscfg(hex_capture)for a macOS build, debug or release, with the feature on, and every line of capture code is behind it. Builds without it (every public release, Linux, Nix, Windows) never compile the crate or its Swift helper and refusescrawl-capturewithunsupported.- The Release, test-build, and Nix builds never pass the feature.
scripts/release-app.shrefusesHEX_CAPTURE, andscripts/validate-app.sh(run on every macOS release artifact) refuses a bundle withContents/MacOS/scrawl-helperor ahexcontaining the capture host's start-up log line. scripts/checkand macOS CI lint and test Hex and the crate with--features capture(the crate's ~260 tests run in seconds). Linux, Windows, and Nix builds never select it: the dependency is macOS-only and the crate is outside the workspace.- Off:
phone(ingest server, pairing),mic(the engine's own cpal stream),input(rdev taps),hex-client,parakeet,cli.helperstays on forscrawl-helper, which decodesrecording.mp4into full-quality keyframes at finish (without it the bundle keeps the 480 px sampled frames).
Helper and permissions
scrawl-helperis built by the crate'sbuild.rsbeside the binary (target/<profile>, the parent of Cargo'sbuilddirectory), and the engine looks for it beside its own executable (helper::path);$SCRAWL_HELPERoverrides. Missing, finish keeps the 480 px candidate frames as keyframes. Hex's capture path uses it for movie decoding, which needs no TCC, fordevices(the USB registry, no TCC), and formirror --no-microphone, a cabled iPhone's capture, which needs Camera.- Camera belongs to Hex, the helper's responsible process, so only
HEX_CAPTURE=1builds addNSCameraUsageDescriptionto Hex.app's Info.plist (and the helper's embedded one);scripts/validate-app.shrefuses a release bundle that has it. HEX_CAPTURE=1 scripts/build-app.shbuilds with--features captureandSCRAWL_HELPER_PLIST=packaging/macos/scrawl-helper-Info.plist(embedded bundle IDanomaly.hex.scrawl-helper, name Hex), copies the helper toHex.app/Contents/MacOS/scrawl-helper, and signs it before the app, with the app's identity, hardened runtime, a timestamp, identifier<bundle id>.scrawl-helper, and no entitlements.- Screen Recording belongs to the process that runs ScreenCaptureKit: Hex itself (or the terminal running a dev build). The helper is Hex's child, so anything it touches is attributed to Hex, its responsible process.
Shape
Hex.app (one process, macOS only for captures)
├─ hex (root package of a new workspace)
│ ├─ capture_host ← new: the experiment's one owner
│ │ ├─ Host impl logs → tracing, metrics, own windows, clipboard
│ │ ├─ Transcriber impl Parakeet words on the transcription engine
│ │ ├─ ConfigSource impl settings.json `capture.*` + one-time import
│ │ └─ capsule + ink overlay (Rune overlays, storybook scenes)
│ ├─ suppression + capture shortcut, + pointer events while capturing
│ ├─ dictation_audio + bounded capture sink on the same timeline
│ └─ transcription_engine + foreground capture clip (after queued dictation)
└─ crates/capture (package `scrawl-capture`, imported without history)
├─ features on in Hex: screen, helper (phone later, for iPhone ingest)
└─ features off in Hex: input, mic, hex-client, parakeet, cli
The engine is Scrawl's: screen sampling, keyframes, sketch and summary, rendering, autocrop, bundles, delivery, routing (with Jev), retention, recovery, and iPhone ingest. Hex supplies the microphone, the input, the transcription, the interface, and the settings. The engine never opens a microphone, installs an event tap, asks for Input Monitoring, or pastes on its own inside Hex.
Workspace and builds
- The root
Cargo.tomllistscrates/capturein[workspace] exclude(built). The root is thehexpackage. - Hex depends on
scrawl-captureonly under[target.'cfg(target_os = "macos")'.dependencies], withdefault-features = false, features = ["screen", "helper"]today (phonejoins with iPhone ingest). Linux and Windows builds never compile it, so their CI and packages don't change. crates/capture/Cargo.tomllists explicit versions (noworkspace = true) so the Scrawl app can depend on it by Git revision until it retires.- macOS CI lints and tests
-p scrawl-capturewith--features capture(built). crates/capture/build.rscompiles the engine's Swift helper. Hex builds it with a Hex identity (SCRAWL_HELPER_PLIST), andscripts/build-app.shcopies and signs it intoContents/MacOS/(built forHEX_CAPTURE=1).
Experiment gate
Experiment capture. With it off, nothing below starts: no shortcut is
registered, no capture sink exists, no ingest port is bound, no retention or
delivery-retry worker runs, and the helper is never launched. The gate sits
where the work starts (the capture_host worker), not only in the window.
Input
- The capture shortcut (default
⌥⇧S) is a third target insuppression, beside dictation and Voice Action, with the same capture and conflict rules as the other bindings. It refuses bindings that overlap dictation. - While a capture records, and only then, the session tap also reports mouse
down, up, drag and scroll, and the Shift state.
capture_hostfeeds them toCapture::input(). Scrawl's ink and click rules (Shift ink, pin and ink thresholds, ignoring clicks on Hex's own windows) move into the engine as pure functions over these events, so Hex runs no second tap. - Escape cancels an active capture before it cancels dictation work. A capture never uses the hold threshold or the post-release tail.
- Built: see step 4 in Built so far.
Microphone
dictation_audiohas a bounded capture sink (open_capture_sink,close_capture_sink): the capture receives the same timestamped samples dictation sees. There is never a secondcpalstream.- Dictation keeps working during a capture. Both are projections of one timeline, so holding the dictation shortcut mid-capture still pastes, and the capture still records that speech as narration.
- With "Release microphone while idle", a capture holds the stream open for its duration, like a recording.
- Sink pressure drops capture audio and marks a gap; it never stalls the timeline or dictation.
Transcription
capture_hostimplements the engine'sTranscriberwith the transcription engine's loaded model and word timestamps (parakeet::transcribe_with_words; dictation still discards its words).- A finished capture's clip is foreground work: it runs after dictation that was already queued and before meeting chunks. It never loads a second model. It goes to the engine in 30 s chunks, one job each, so a dictation sent meanwhile waits for one chunk at most. A chunk waits while dictation captures or transcribes, and the engine holds one that arrives during a live stream until the stream ends, so it never interrupts one.
- Scrawl's spoken cues need measured word times. Compared with Apple's
SpeechAnalyzer on 36 real captures (Oct 25), Parakeet Unified English's word
times run late: starts by a median 280 ms (about 520 ms after a pause), ends
by 180 ms, with no drift and no cost at the 30 s seams; raw, 53% of cue words
are over 300 ms late.
HexTranscribersubtracts the lag fitted on half the captures (WORD_LAGSincapture_host: starts 280 ms, or 520 ms when the gap since the previous word's end is at least 250 ms; ends 195 ms), which puts 96% of the other half's cue words within 300 ms of Apple. The lag is keyed to the pinned model file: another model, or a new pin, keeps raw times until it is measured again. The Apple helper path stays selectable (hex settings set capture.transcriber apple).
Interface
- The capsule (recording, processing, done, route toast, picker) and the ink
and laser overlay are click-through Rune overlays, like the meeting capsules.
Their window ids go to the engine through
Host::own_windowsso screen capture leaves them out and clicks pass through. - Each state has a
--snapscene and a storybook entry. - Scrawl's switch chips (
docs/capture.md): a shortcut-started recording opens onto Area · Window · Screen for 3.5 s (never folding while hovered) or until the first click, scroll, or stroke. A chip restarts rather than retargets: Window and Area discard the recording and turn the capsule into the picker. The host side is built:CaptureTarget,CaptureCommand::{Start, Discard, Switch}throughRecognitionControl::CaptureCommand,RecordingCapture::{target, worked},SettledCapture::error, andbegin_pickingfor ⌥⇧S and Esc while the picker is open. The shortcut starts on its modifiers' release, as Scrawl's does. The iPhone chip's host side is built too:CaptureTarget::Phone,TargetKind::Phone,CaptureSnapshot.phone(the cabled phone's name from acapture-phonelook at each start and onCaptureCommand::FindPhone). The mirror runs with--no-microphone; the narration is Hex's timeline audio moved onto the movie's clock (RecordedAudio::since) and transcribed by Hex's transcriber. A mirror that ends on its own (Event::PhoneEnded) finishes the capture at the listener's next reap. - The done capsule's route chips read
SettledCapture.route(its answer, then only the deliveries the capsule can't know itself) and sendCaptureCommand::{Send, Undo, Redirect}. Built;offerisn't drawn. - A Scrawl pane under Experiments, feature-owned (D-5): opt-in, shortcut, Screen Recording, transcriber, captures folder, retention, and "Use Hex for captures" while Scrawl runs (built; iPhone joins with ingest).
- Screen Recording is asked for at the first capture or from the pane, not in core onboarding.
Routing and delivery
- Built (
capture_host/routing.rs): Scrawl's router (crates/router) runs in Hex's process withcapture.routing(imported once from Scrawl; never a Jev key) and Scrawl's own router folders, soscrawl router undo, rule offers, and evals see Hex's sends as Scrawl's. Acapture-lookthread reads the tab, terminal, and sessions at the stop; the onecapture-routethread (queue 16) decides once the bundle is written, waits out a rule's 3 s undo, sends, and carries out the capsule's Send, Undo, and Redirect on the newest capture in order. Each change lands inlast.route. - Delivery uses the engine's delivery path. Queued sends retry every 15 s, except in tests, while Scrawl.app runs, or while anything serves the ingest port (that process retries). Hex never stores the OpenCode server password itself; the engine reads it as Scrawl does.
- Not built:
live.ndjsonevents for routing (ids, bands, scores, model, stages, and outcomes, never transcript, summary, or window text).
Coexistence with Scrawl
- One owner at a time. While Scrawl (
dev.kitlangton.scrawl) runs, Hex does not register the capture shortcut, run retention, bind the ingest port (47777), or run delivery retries. Settings shows "Use Hex for captures", which quits Scrawl only on that click (built, without ingest). - Configuration: a one-time, read-only import of Scrawl's settings into
settings.jsoncapture.*keys, each with ahex settingskey (built). The ingest token moves to an owner-only file in Hex's support folder so paired phones keep working (not built). - Captures stay in Scrawl's captures folder; Hex writes and cleans them there (built).
Invariants to add
- Capture never opens a microphone, installs an event tap, or loads a model of its own, and never blocks or drops dictation audio.
- Capture queues are bounded: the capture sink, the engine's accessibility and retention queues (unbounded in Scrawl today; bound them before import), and the frame candidate buffer (lower than Scrawl's 384 MB).
- One engine per process (the engine keeps process-global state).
- The experiment off means no capture work at all.
CLI
Through POST /control, like everything else, behind the build option
(built; docs/cli.md): hex capture start|stop|cancel|status,
hex capture list [--limit N]|show <capture>, hex capture send <capture> [--to <session>], and hex capture route <capture> [--ask-jev]. Not built:
engine tools (autocrop eval, capture simulation, reprocess) under
hex dev capture in debug builds.
Order on the branch
- Scrawl prepares the engine on its own exploration branch (its inventory's
refactors:
lib.rs, pipeline out ofmain.rs, event, config, permission, clipboard and transcriber seams, oneCapturetype, Cargo features). - Import
crates/capturehere, with the macOS-only dependency and CI. Built: copied without history frome351ddb. capture_hostwith the experiment gate, the capture sink, Parakeet words, and a headlesshex capture start|stopproducing a bundle. Built.- Input and Escape, then the Rune capsule, picker, and ink overlay with scenes and motion traces. Built.
- Router in-process, the Scrawl pane, import, and coexistence. Built.
- Compare Parakeet and SpeechAnalyzer word times on recorded captures. Done
(Oct 25): Parakeet's lag is corrected in
capture_host.
The Oct 18 experiment/scrawl branch was a copy that drifted; feat/capture
supersedes its capture sink and word transcriber, so it can be deleted.
Asks for Scrawl's API
Answered by engine/split: Start::t0_boot_ns, RecordedAudio with
start_offset_ms and gaps, words relative to the first sample, and
SharedResources::none() by default.
Then answered at 86d7d6c: Capture::stop returning a SealedCapture
(the screen stops at seal, finish only writes), per-capture telemetry
without global shortcuts, no TIS/TSM in Engine::new, and autocrop and
agentContext from ConfigSource.
Then answered at e351ddb (the imported commit): Engine::new bridges the
crate's log to Host::log when no sink is set, and a library writes Scrawl's
log file or reads its config only when asked; Engine::start's docs say it
blocks; bundle files Scrawl writes are 0600; and Finished reports gaps
and missing_transcript.
Left from hosting the real engine:
- No audio-only target: every capture records the screen and needs Screen Recording. Hex's narration-only tests use a fake engine.
Engine::startblocks on target discovery and ScreenCaptureKit; Hex runs it off the listener thread.finishlaunchesscrawl-helperto decode the movie, so a library host ships and signs a second executable (Hex does, withHEX_CAPTURE=1) or loses full-quality keyframes. Decoding in-process would drop it.recording.mp4, the window log, and folders still get the process umask; Hex tightens them afterfinish.phone,routing,delivery, and the ingest HTTP code compile regardless of features, and the crate pulls a secondwindows(0.61) intoCargo.lockfor its Windows target.anyhowin the public traits makes acolor_eyrehost convert every error.