mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* annotate: install the agent skill and cli from the annotate pane (labs) * rune_ui: drop the unused mut from the agents fixture * annotate: use the shared agents skill folder and harden the agent install (labs) * annotate: harden the hex link and the managed skill rows (labs)
1497 lines
103 KiB
Markdown
1497 lines
103 KiB
Markdown
# Annotate (macOS, Linux, and Windows)
|
||
|
||
Annotate is Scrawl's capture engine hosted inside Hex (code and settings keep
|
||
the name `capture`), steps 3 to 5 of
|
||
[the plan](../plans/capture-in-hex.md). It runs the real
|
||
`scrawl-capture` crate, imported into `crates/capture`: it records the active
|
||
display with ScreenCaptureKit, on Linux the screen-cast portal and
|
||
PipeWire (`capture.linux`), on Windows Windows.Graphics.Capture
|
||
(`capture.windows`), (or a cabled iPhone through `scrawl-helper`'s
|
||
mirror) and narration from Hex's microphone, takes ink and clicks (Shift on
|
||
macOS and Windows, or the Annotate key held or the pen), transcribes the
|
||
narration with word timestamps on Hex's loaded model (a model that gives
|
||
none, GigaAM v3 say, has its text spread over the recording as estimated
|
||
times), writes a Scrawl bundle into Scrawl's captures folder, copies it to
|
||
the clipboard, and sends it to an OpenCode session with Scrawl's router
|
||
(`crates/router`), run in Hex's process. Settings › Annotate (under Labs in
|
||
the sidebar) holds its settings. Scrawl's capsule (switch chips, the
|
||
on-screen picker, route chips) and its ink and laser overlay are Rune
|
||
overlays. A capture's `summary.md` and the text pasted into an agent start
|
||
with `[hex-annotate <clock> · <target> · <folder>]`; older captures say
|
||
`[scrawl …]`.
|
||
|
||
```ts
|
||
Turn it on // capture.gate
|
||
├── On by default in every macOS, Linux, and Windows build, stable included
|
||
│ (capture.enabled); the `annotate-on-by-default` migration turns it on
|
||
│ once for older settings
|
||
│ └── macOS bundles sign scrawl-helper beside `hex`; validate-app.sh requires it
|
||
├── Nothing is asked at launch: Screen Recording (macOS) at the first capture,
|
||
│ the portal's dialog (Linux) at the first Screen capture; Windows asks
|
||
│ nothing
|
||
├── A start without a permission it needs (macOS: Screen Recording unless
|
||
│ iPhone, then Microphone, then Accessibility; capture_host::permissions)
|
||
│ -> refused with code `permission`, no error tone, and the capsule's
|
||
│ permission helper opens (capture.permission)
|
||
└── Off -> no capture host, engine, sink, thread, or folder; `hex annotate` -> feature-off
|
||
|
||
Draw // capture.draw (capture_host::drawing)
|
||
├── One key: tap (under DRAW_HOLD, 250 ms) to start and again to stop; held
|
||
│ while it records, the pointer draws with no click until it comes up
|
||
│ (macOS: CaptureShortcut tells the tap from the hold at release; Linux:
|
||
│ the portal's Activated/Deactivated or the X11 grab's press and release,
|
||
│ linux/input Capture/CaptureUp; Windows: the keyboard hook's
|
||
│ HotkeyAction::Capture/CaptureUp -> CaptureHost::annotate_down/up)
|
||
├── The capsule's pen -> on until clicked again, or the key is tapped, or
|
||
│ held and let go (Action::Pen -> capture_host::toggle_pen). Like a real
|
||
│ pen it inks only between mouse-down and mouse-up (capture_host::
|
||
│ pen_button; Linux: the overlay's drag, macOS: the tap's buttons); moving
|
||
│ with the button up draws nothing, and clicks still never reach the app
|
||
│ └── Releasing the button ends the stroke on the `capture-input` thread;
|
||
│ the pointer it reads (capture_host::ink_source) has its own lock, so
|
||
│ it never re-takes the input target's lock that thread holds (#572:
|
||
│ Windows hung on the first stroke). Check: capture_host
|
||
│ ending_a_stroke_while_the_input_target_is_locked_does_not_deadlock;
|
||
│ not observed on a Windows PC
|
||
├── Esc while drawing -> stops drawing, never the recording (recognition.rs
|
||
│ on macOS; the Escape grab on X11 (Wayland binds no Escape); the keyboard
|
||
│ hook on Windows, when no dictation records; and the overlay's own key
|
||
│ handler where it gets the keyboard)
|
||
├── macOS and Windows -> Shift held alone still draws, as in Scrawl
|
||
├── While drawing -> the ink overlay takes the pointer with a pen cursor, so
|
||
│ clicks never reach the app; the capsule stays clickable above it (macOS:
|
||
│ its level; Linux: the overlay leaves a hole over the capsule's pill).
|
||
│ Every move draws ink; there are no pins and no undo
|
||
├── `hex annotate draw` (Linux) -> the pen on or off, for desktops without
|
||
│ the portal (it starts a recording drawing)
|
||
├── Default key: ⌥⇧S on macOS; F9 on Linux and Windows, a single key one hand holds
|
||
│ while the other moves the mouse (F10 is GTK's menu key, F11 full screen,
|
||
│ F12 often a dropdown terminal). Saved bindings stay.
|
||
│ Check: capture_host the_shortcut_records_once_its_keys_come_up; none
|
||
│ physical for the hold on a portal desktop ("Annotate pen" is logged)
|
||
├── Ink holds 650 ms, then fades over 650 ms on screen; every stroke stays in
|
||
│ the bundle (keyframes, contact sheet, scrawl.json)
|
||
├── Linux: strokes are placed on the recorded screen in the ink overlay's
|
||
│ own points. A native Wayland layer surface lays out in the desktop's
|
||
│ logical points: a monitor pick whose portal names the output
|
||
│ (`mapping_id`, as xdg-desktop-portal-hyprland 1.4.0+ does, e.g. `DP-3`)
|
||
│ opens the pen on that output with its logical size from xdg-output
|
||
│ (linux::output::find), so strokes land on the recorded monitor even when
|
||
│ the overlay had opened on another one; otherwise it uses the ScreenCast
|
||
│ portal's stream size when it reports one smaller than the frames, so a
|
||
│ fractional scale (1.5, 1.33), which reaches the overlay rounded up to 2,
|
||
│ no longer pins strokes to a top-left part of the screen; without that
|
||
│ size it divides the pixels by the overlay's scale. The XWayland window
|
||
│ GNOME gets instead lays out in its own points and never uses either
|
||
│ logical size: it takes the size its view laid it out in
|
||
│ (capture_host::InkSpace), so ink reaches the whole screen whether the
|
||
│ compositor scales XWayland natively (1920x1200 X11 pixels at Xft's 1 on
|
||
│ a 125% screen, where the portal says 1536x960) or traditionally (the
|
||
│ logical 1536x960, where the stream is 1920x1200 pixels); before that size
|
||
│ is known it divides the pixels by the overlay's scale. Window picks still
|
||
│ report no position and no output. Check: scrawl-capture
|
||
│ capture::platform::tests a_fractionally_scaled_screen_keeps_its_logical_size,
|
||
│ linux::output::tests the_named_output_carries_its_global_and_logical_size,
|
||
│ and capture_host the_ink_target_is_the_overlay_s_own_space; neither the
|
||
│ follow nor a fractional scale is observed on real hardware here. Known
|
||
│ gaps: the overlay starts below a desktop bar's reserved area, so saved ink
|
||
│ sits that bar's height off the live ink; on XWayland the ink is one
|
||
│ full-screen primary-monitor window, so it cannot follow a named output,
|
||
│ and a recording of another monitor maps its strokes to the wrong screen
|
||
└── Linux recordings show the overlay itself (Wayland can't leave it out)
|
||
|
||
On Linux // capture.linux
|
||
├── X11 session (DISPLAY set, WAYLAND_DISPLAY empty; never XWayland) ->
|
||
│ Screen and Area are read from the X server itself (scrawl_capture::
|
||
│ linux::x11): MIT-SHM GetImage each frame borrowed in place (plain GetImage
|
||
│ without SHM), the pointer cached from XFixes on XFixesCursorNotify and
|
||
│ placed each frame from QueryPointer, 60 fps to x264 (`superfast` up to
|
||
│ 1440p, `ultrafast` above, wall-clock stamped into `fps=60`) through
|
||
│ ffmpeg. No
|
||
│ portal and no dialog, so i3 and other desktops without a ScreenCast
|
||
│ backend record (#690, @nefthy). Screen is the RandR monitor under the
|
||
│ pointer; Area is `hex annotate start --area X,Y,W,H` (pixels), clipped to
|
||
│ the screen. Hex's own overlays show in it. A Window still goes through
|
||
│ the portal's picker and, without one, fails saying to record the screen
|
||
│ or an area. Can't open the display, an unusual pixel depth, or no
|
||
│ ffmpeg/x264 -> the capture fails at once with why in the capsule, never
|
||
│ a capture without a picture. Check: x11 areas_are_clipped…,
|
||
│ cursor_blends_premultiplied_argb_in_place; on hex-linux-nvidia (T4, Ubuntu
|
||
│ 24.04) native Xorg + i3 recorded 1920x1080 at 60/1 fps (598 frames in
|
||
│ 9.97 s, 597 grabs) and a 1280x720 Area (298 frames in 4.97 s, 297 grabs),
|
||
│ and rootful Xwayland at 3840x2160 recorded 2560x1440 at 60/1 fps (598
|
||
│ frames in 9.97 s from 282 raw 4K grabs; raw 4K pipe readback is CPU-bound)
|
||
├── Wayland -> the ScreenCast portal (zbus): Screen asks the desktop once and
|
||
│ keeps its restore token (<support>/screencast-token), so later captures
|
||
│ start without a dialog and record the remembered monitor; a monitor pick
|
||
│ the portal names (`mapping_id`: xdg-desktop-portal-hyprland 1.4.0+) moves
|
||
│ the pen to that output (see the pen bullet above). Window shows the
|
||
│ desktop's picker every time (the capsule's Window chip goes straight to
|
||
│ it) and names no output, so the pen keeps its old placement; the cursor
|
||
│ is embedded
|
||
├── Labs › Record the screen under the pointer (`labs.annotate_pointer_screen`,
|
||
│ off) -> each Screen capture first asks the compositor for the output the
|
||
│ pointer is on, with no surface and no frame: one
|
||
│ `ext_image_copy_capture_manager_v1.create_pointer_cursor_session` per
|
||
│ output source, and the session that reports `enter` names its `wl_output`
|
||
│ (`src/linux/pointer_output.rs`; two round trips, everything destroyed
|
||
│ after). Its screen then has its own token file,
|
||
│ `<support>/screencast-token-<output>` (output sanitized to [A-Za-z0-9_-]),
|
||
│ so each monitor is asked once; a monitor without one shows the picker
|
||
│ once. After Start the token is saved for the output the desktop actually
|
||
│ recorded: `mapping_id`, else the output whose xdg-output logical geometry
|
||
│ holds the stream's position (`src/linux/output.rs`), else the pointer's.
|
||
│ The refused-token retry removes and forgets whichever file it read. Off,
|
||
│ or when the lookup fails (no seat, no staging protocols, no session
|
||
│ entering, an output below wl_output v4), the single file above is used
|
||
│ exactly as before. Needs the compositor to export ext-image-copy-capture
|
||
│ to ordinary clients (COSMIC does only for non-sandboxed ones) and, on
|
||
│ wlroots, a hardware cursor (a software cursor reports nothing, read from
|
||
│ wlroots' source rather than observed); not observed on real COSMIC
|
||
│ hardware
|
||
├── Frames -> the portal's PipeWire stream, libpipewire-0.3 loaded at run
|
||
│ time (`maxFramerate` 60/1); 32-bit RGB as a linear DMA-BUF (niri, GNOME,
|
||
│ KDE) or shared memory. A stream that ends or fails before its first frame
|
||
│ ends the capture with that reason, keeping no voice; a stream with no frame
|
||
│ yet when the person stops is saved with its voice and marks, the summary
|
||
│ saying the stream sent no frames
|
||
├── Encoding -> VA-API (libva loaded at run time): each DMA-BUF imported, scaled
|
||
│ to a 2560 px long edge (1920 for a window, exact even sides padded to 16 px
|
||
│ macroblocks and cropped in SPS) and converted on the video engine, H.264
|
||
│ High at a constant QP and computed level (4.2 at 1080p60, 5.1 at 1440p60,
|
||
│ 5.2 at 4K60) with Hex-written headers, 60 fps; `ffmpeg -c copy` muxes
|
||
│ recording.mp4. Every /dev/dri render node is
|
||
│ tried in order, so a hybrid laptop whose first node is a GPU the VA
|
||
│ driver can't use encodes on the next (#853). No VA-API -> frames copied
|
||
│ and encoded at 60 fps with x264 (`superfast` up to 1440p, `ultrafast`
|
||
│ above) through ffmpeg (slow on DMA-BUF-only compositors; on NVIDIA GBM +
|
||
│ xdg-desktop-portal-wlr without DONT_FIXATE, linear DMA-BUF allocation is
|
||
│ refused by the compositor and delivers no frames);
|
||
│ VA-API refusing the first frame's buffer (#1179: Intel Arc on the xe
|
||
│ driver) -> the same CPU path instead of failing. The negotiated DRM
|
||
│ modifier goes to the import and the log; the CPU never reads a tiled
|
||
│ buffer as rows. A first frame neither path can read (a DMA-BUF the CPU
|
||
│ can't map, or tiled) fails the capture with why, never a capture without
|
||
│ a picture or a crash. Buffers PipeWire removes mid-stream (a
|
||
│ renegotiation) are never queued again, and Hex maps buffers itself and
|
||
│ unmaps a removed one only after the recorder's copy, so a renegotiation
|
||
│ can't crash Hex. Check: vaapi render_nodes_in_order,
|
||
│ level_and_sps_crop_match_resolution_and_60fps, pipewire
|
||
│ offered_formats_parse_back, a_negotiated_tiled_modifier_is_read,
|
||
│ capture::platform output_sizes_stay_exact_even_dimensions… and
|
||
│ cpu_x264_uses_superfast…; PipeWire 1920x1080 SHM -> x264 recorded 60/1 fps
|
||
│ (598 frames in 9.97 s, 594 delivered) on hex-linux-nvidia; VA-API 60 fps
|
||
│ encoding not observed on Intel/AMD hardware (T4 has no VA-API encoder), nor
|
||
│ on a hybrid laptop or an Intel Arc laptop
|
||
├── Candidates -> a small copy from the video engine on demand every
|
||
│ FRAME_SAMPLE_INTERVAL (at most 4× smaller per pass: radeonsi crashes past
|
||
│ it)
|
||
├── Reading movies back (keyframes, probe, audio) -> `ffmpeg`/`ffprobe`
|
||
├── Missing PipeWire, ffmpeg, or a picker answer -> that capture fails with why
|
||
│ └── The screen-share picker cancelled (or the stream never starting) after
|
||
│ the capture started -> its collector notices within 100 ms, cancels
|
||
│ it, and settles `failed`; the listener's next reap ends it, so the
|
||
│ shortcut and dictation work again (#681). Check: capture_host
|
||
│ a_screen_source_that_gives_up_ends_the_capture; not observed on GNOME
|
||
├── Narration -> the listener's microphone through ListenerTimeline (each
|
||
│ stream anchored to the monotonic clock); released-while-idle microphones
|
||
│ open for the capture
|
||
├── Words -> one foreground job on the listener's engine (its loaded model)
|
||
├── Copy -> the summary text through Hex's Linux clipboard (wl-copy or the
|
||
│ X11 clipboard), as the Mac copies it; Labs › Copy a capture as one file puts
|
||
│ `<stem>.scrawl.slim.zip` there instead, as a `text/uri-list` file
|
||
├── Paste -> as the Mac: the contact sheet (image/png), then the summary,
|
||
│ each with Hex's own Ctrl+V (linux/paste). An app that takes no image (a
|
||
│ terminal) gets only the summary, which names the contact sheet; the Mac
|
||
│ pastes a terminal the sheet's path instead
|
||
├── Finished capsule -> "Annotation" with Paste and, if there are destinations, Send to…;
|
||
│ unlike the Mac, a guess doesn't open its suggestions by itself
|
||
│ (Flow::suggesting_on_route(false)): Send to… is a click away
|
||
├── Shortcuts -> F9 (tap to start and stop, hold to draw) through the
|
||
│ GlobalShortcuts portal ("Annotate") or X11 grabs. Without the portal,
|
||
│ General's commands list and this pane's Shortcut row give
|
||
│ `hex annotate toggle` to bind in the desktop instead. There is no
|
||
│ separate draw shortcut and no send-to list shortcut on any OS
|
||
├── GNOME Wayland -> no layer-shell, so Rune can't show the capsule or ink
|
||
│ until Labs › Annotate on GNOME (`labs.gnome_annotate`, off by default)
|
||
│ opens them on XWayland (`OverlayOptions::xwayland_input`): override-redirect
|
||
│ windows that take the pointer in their input shape and never keyboard
|
||
│ focus, only while this user's Xwayland already runs (never started by Hex;
|
||
│ the meeting capsules' decision in linux/meeting_capsules.rs). The ink is
|
||
│ made with pointer regions and `KeyPolicy::None`, since Rune refuses an
|
||
│ OnDemand overlay on XWayland, and keeps regions with no hit area while not
|
||
│ drawing because a click-through XWayland window can't take the pointer
|
||
│ later. Esc can't stop drawing there: the capsule's pen button does, or
|
||
│ letting go of a held Annotate key (a tap of it stops the recording). Its
|
||
│ pen draws in that window's own points, which at a fractional scale are
|
||
│ neither the desktop's logical points nor the stream's pixels (see the
|
||
│ pen bullet's known gaps). The in-app picker shows on Linux only in the
|
||
│ edge states that return to Selecting (Window starts the portal's picker,
|
||
│ Area isn't offered), and takes the same opt-in whenever it does. Check:
|
||
│ labs gnome_annotate_is_its_own_bit_*; not run on a GNOME session
|
||
├── While it records, dictation starts nothing (the portal
|
||
│ or grab already took their keys, so nothing reaches the app)
|
||
├── Pane -> no Screen Recording row, no transcriber choice, no "Show in screen
|
||
│ recordings" (the portal asks; Wayland can't keep overlays out)
|
||
├── Sending -> OpenCode's sessions and prompts over HTTP to the running
|
||
│ service its registration names (service.json); nothing is started, and
|
||
│ with no service the send says "OpenCode isn't running". The capsule's
|
||
│ rows are clicked (no keyboard).
|
||
│ See capture.send: everything goes inline, so a remote server works
|
||
└── Gaps: no Area on Wayland (it gives Hex no pointer to aim it; X11 takes
|
||
`--area`), no area picker on screen, no iPhone, no
|
||
laser or click log outside draw mode, no window log, `hex capture status`
|
||
unsupported, choosing another folder only through settings.json
|
||
|
||
On Windows // capture.windows
|
||
├── Source -> the engine's Windows.Graphics.Capture recorder (win/media.rs):
|
||
│ Screen, Window (the picker lists windows), and Area; no permission asked
|
||
├── Units -> the engine's targets, windows, and pointer are physical pixels;
|
||
│ the picker and ink overlays sit on the primary display and lay out in its
|
||
│ points, converted with capture_host::pixels_per_point (mixed-DPI displays:
|
||
│ the primary's scale applies everywhere)
|
||
├── Narration -> the listener's microphone through ListenerTimeline, fed by
|
||
│ windows/dictation; released-while-idle microphones open for the capture
|
||
├── Words -> one foreground job on the listener's engine (its loaded model)
|
||
├── Input -> windows/input's low-level hooks: the Annotate key (F9; tap to
|
||
│ start and stop, hold to draw), Escape while drawing, and while a capture
|
||
│ records every pointer move, click, wheel turn, and modifier change
|
||
│ (capture_host::report_input, as the macOS tap): Shift alone inks, the
|
||
│ laser follows the pointer, clicks are logged. Injected events are skipped
|
||
├── While it records, dictation starts nothing
|
||
├── While dictating -> General's setting through windows/audio_control, one
|
||
│ suppressor per capture, restored when it ends
|
||
├── Copy and paste -> the engine's own clipboard and Ctrl+V (handoff): the
|
||
│ contact sheet, then the summary; with Labs › Copy a capture as one file, the
|
||
│ copy is the light zip as CF_HDROP instead
|
||
├── Pane -> no Screen Recording row, no transcriber choice
|
||
├── Overlays -> Rune waits for a swap-chain slot before each frame (Rune
|
||
│ #985), so the picker, capsule, and ink's frames in flight stay bounded
|
||
│ through a long annotation (#1833; not measured on Windows)
|
||
├── The finished capsule -> windows/input's hooks stand in for the macOS tap
|
||
│ (#2113): the search field takes its own clicks (its `.hit()` box is
|
||
│ Windows-only, so the window stops being click-through over it and the
|
||
│ click no longer lands on the app in front), a mouse button going down
|
||
│ away from the capsule folds the list or the search (`CapsuleKey::Press`;
|
||
│ a press on the capsule's own boxes is its click), and the keyboard hook
|
||
│ hands the open list or the suggestions the keys `capture_host::list_key`
|
||
│ names: text, ⌫, ↑↓, ⏎/⇧⏎ and ⇥, Escape and Shift. Text is read from
|
||
│ `ToUnicodeEx` in the foreground window's layout with "change no state",
|
||
│ so a key the app gets is never disturbed and a dead key (`^`, `´`, `¨`)
|
||
│ types nothing in the search rather than half an accent landing nowhere.
|
||
│ The search also keeps the caret's own keys from the app — ⌦, ←→, Home and
|
||
│ End, Page Up and Page Down, Insert — though it draws no caret movement for
|
||
│ them; over the suggestions they pass. A key that edits nothing (Caps Lock,
|
||
│ Num Lock, Scroll Lock, the media and browser keys, Print Screen, Sleep,
|
||
│ Apps, a function key) goes on to the app even in the search, as does a key
|
||
│ the app already had down with its repeats and release, wherever the capsule
|
||
│ stood when it went down. Ctrl, Alt and the Windows key make a chord and stay
|
||
│ the app's, so AltGr types its character in the app, not the search; Hex's
|
||
│ own shortcuts do too, and Escape over the suggestions is the app's. While
|
||
│ a dictation records, Escape cancels that first, as it does everywhere on
|
||
│ Windows, so it never reaches the search. Check:
|
||
│ `windows::input::capsule`'s mapping tests; not observed on a Windows PC
|
||
└── Gaps: no `hex annotate` (Windows has no control transport yet), no
|
||
iPhone, keys pressed in the app are not logged. Checked only by
|
||
cross-compiling (`cargo xwin clippy`, x86_64 and aarch64); never run on
|
||
a Windows machine
|
||
|
||
Settings › Annotate // capture.pane
|
||
├── Capture with Hex (capture.enabled, default on) -> off: no host, shortcut,
|
||
│ input, retention, or folder
|
||
│ └── On while the shortcut collides -> refused, the shortcut row says why
|
||
├── Shortcut (capture.hotkey, default ⌥⇧S, F9 on Linux and Windows) -> same keycap and recorder as dictation;
|
||
│ a key is required; refused when it overlaps dictation or paste. On a Linux desktop with no
|
||
│ GlobalShortcuts portal (the `commands` fallback) the row is the `hex annotate toggle` command
|
||
│ to bind in the desktop instead, as General's commands list has it
|
||
├── How it works (on) -> start (tap), draw (hold, no click; Shift too on
|
||
│ macOS), the capsule's pen with Esc, stop (tap again), and send, each
|
||
│ with its keys. Check: rune_ui::scrawl tests
|
||
├── Screen Recording -> read once a second and again when the pane opens or
|
||
│ the window becomes key, never with a prompt (capture_host::screen_access):
|
||
│ the preflight, which keeps its first answer for the life of the process,
|
||
│ then, while it says no, a fresh `scrawl-helper permissions` on its own
|
||
│ thread: at most every 60 s while nothing shows it, and every 2 s while
|
||
│ its row or the capsule's permission helper is on screen, plus a one-frame
|
||
│ probe once that says granted
|
||
│ ├── granted -> Allowed, no button
|
||
│ ├── granted in System Settings but unusable until Hex relaunches ->
|
||
│ │ "Turns on when Hex restarts" and Restart Hex, never Allow over a switch
|
||
│ │ already on
|
||
│ ├── Returned from System Settings still untrusted -> "Still grey? Fix it"
|
||
│ │ (`ScreenCapture` reset on running bundle + retired IDs, re-prompt + guide)
|
||
│ └── Allow -> clears retired Hex IDs, rechecks, then opens the permission
|
||
│ guide over System Settings' list (PermissionKind::ScreenRecording); flips
|
||
│ on the next read. The guide's own preflight caches the launch answer,
|
||
│ so it reads the grant from TCC and the host closes it as soon as the
|
||
│ watched read says granted or needs a restart; it hides while another
|
||
│ app is in front of System Settings and returns when System Settings
|
||
│ does. Never part of onboarding (macOS also asks at the first
|
||
│ capture). Check: rune_ui::scrawl tests
|
||
├── Show in screen recordings (capture.show_overlays_in_recordings, default
|
||
│ off) -> see capture.own-windows; applies at once, to open windows and a recording capture
|
||
├── Transcriber (capture.transcriber) -> Parakeet (default) | Apple
|
||
├── Captures folder (capture.folder, default Scrawl's) -> Reveal | choose another
|
||
├── Keep (capture.media_days / bundle_days) -> Scrawl's presets, or imported days
|
||
├── Scrawl.app running -> "Scrawl is running" card + Use Hex for captures
|
||
│ └── Click -> asks Scrawl to quit (nothing else quits it); Hex takes the
|
||
│ shortcut within 2 s of Scrawl leaving
|
||
└── No iPhone pairing row: ScrawlKit pairing needs the ingest server, which Hex
|
||
doesn't run. A cabled iPhone needs no row (capture.phone)
|
||
|
||
Agents (on; Labs › Install agent skill from Annotate) // capture.agents; labs.annotate_agent_install
|
||
├── Off -> the pane is exactly as before, and nothing reads a coding agent's folder
|
||
├── One row per coding agent: whether its folder exists, and whether it already has this Hex's skill
|
||
│ ├── Claude Code -> $CLAUDE_CONFIG_DIR or ~/.claude, then skills/hex-annotate/SKILL.md
|
||
│ │ (found with that folder)
|
||
│ ├── Codex -> ~/.agents/skills/hex-annotate/SKILL.md (found with ~/.codex or
|
||
│ │ $CODEX_HOME); $CODEX_HOME/skills is only the legacy place
|
||
│ ├── OpenCode -> $XDG_CONFIG_HOME or ~/.config, then opencode/skills/hex-annotate/SKILL.md
|
||
│ │ (found with the opencode folder)
|
||
│ └── Pi -> ~/.pi/agent/skills/hex-annotate/SKILL.md ($PI_CODING_AGENT_DIR or ~/.pi;
|
||
│ found with that folder), or the shared ~/.agents copy when one exists
|
||
├── npx skills writes Codex and Pi skills to ~/.agents/skills, so that copy counts for
|
||
│ both: read by content, never checked for a second copy, and an outdated shared copy
|
||
│ is the one installing updates
|
||
├── A found agent without the skill reads Not installed and starts checked; the file
|
||
│ differing from this Hex's reads Update available, and installing replaces it
|
||
├── Install -> agent_install::install writes the same embedded
|
||
│ skills/hex-annotate/SKILL.md that hex annotate skill prints into each checked agent's
|
||
│ folder: directories created, a staging file renamed into place in the same folder,
|
||
│ and no other file touched
|
||
├── A symlinked SKILL.md, or a symlinked hex-annotate folder, is managed elsewhere: the
|
||
│ row says Managed elsewhere, it has no switch, and nothing is written through the link
|
||
├── An agent this computer doesn't have reads Not found, has no switch, is never checked,
|
||
│ and gets no folder created for it; Install is disabled while nothing is checked
|
||
├── Install with the skills CLI -> npx skills add anomalyco/hex --skill hex-annotate,
|
||
│ beside Copy for people who manage skills themselves
|
||
├── Command-line tool (macOS) -> a hex symlink to this app's executable (current_exe) in
|
||
│ the first of /opt/homebrew/bin and /usr/local/bin that exists and is writable, else
|
||
│ ~/.local/bin (created)
|
||
│ ├── An existing hex that is a symlink to this Hex or any
|
||
│ │ Hex.app/Contents/MacOS/hex is kept and reads Installed at <path>; a link whose
|
||
│ │ target is gone counts only when it names a Hex app, and is then replaced
|
||
│ ├── A regular file, or a link to anything else (a dangling one included), is never
|
||
│ │ replaced: the row reads Owned by another program
|
||
│ ├── A hex running from App Translocation refuses to link and says to move Hex to
|
||
│ │ Applications first
|
||
│ └── A ~/.local/bin link that $PATH doesn't name adds "<folder> isn't on your PATH"
|
||
├── While Labs › Agent tools for captures is off, a note says the skill's zoom, inspect,
|
||
│ and frames commands need it
|
||
└── Checks: agent_install (finding each agent, installing, updating an outdated file,
|
||
leaving other files alone, the shared ~/.agents copy for Codex and Pi, a symlinked
|
||
skill or folder, choosing the link folder, never replacing a foreign hex (a live
|
||
foreign link or a regular file is never removed even by link() itself), a dangling
|
||
Hex-app link, App Translocation), and rune_ui::scrawl's
|
||
the_agents_section_needs_its_lab, installing_writes_each_checked_agents_skill,
|
||
an_agents_row_checks_only_that_agent, a_linked_skill_has_no_switch.
|
||
Proof limits: the macOS symlink path runs the same POSIX rules its unit tests
|
||
cover on Linux; nothing was run on a real Mac, and no real Claude Code, Codex,
|
||
OpenCode, or Pi read a capture after this installed the skill
|
||
|
||
First launch with Annotate on // capture.import
|
||
├── ~/.config/scrawl/config.json -> transcriber (apple; else Parakeet), capturesDir,
|
||
│ retention, read once (migration `scrawl-capture-settings`); token not used
|
||
├── Scrawl's router settings -> capture.routing, read once (migration
|
||
│ `scrawl-routing-settings`): rules, autoSend, smartRouting and jevRouting;
|
||
│ never the Jev key (Jev uses $OPENCODE_API_KEY or OpenCode's credential)
|
||
├── Scrawl's Application Support config.json -> its shortcut, unless it collides
|
||
└── Scrawl's files are never written
|
||
|
||
⌥⇧S (capture.shortcut; captures on, Scrawl not running)
|
||
├── Press, held as long as you like -> armed; nothing records while ⌥⇧ is down
|
||
│ └── One of its modifiers comes up -> start on the display under the pointer,
|
||
│ as `hex capture start` // a fresh press only, no repeats
|
||
├── Esc before the modifiers come up -> nothing records
|
||
├── Press while recording -> stop, as `hex capture stop` // ignored within 400 ms
|
||
├── Press while the capsule's picker is open -> the picker records what it is on
|
||
├── Esc while a capture records -> nothing: it reaches the app being recorded,
|
||
│ and Hex's tap lets it through; a dictation under it goes on too (#786)
|
||
├── The binding is swallowed, never chord evidence for dictation
|
||
├── Hold the dictation shortcut mid-capture -> dictation pastes as usual
|
||
└── Another app holds Secure Event Input -> ⌥⇧S still starts and stops (a
|
||
Carbon hotkey while it lasts; Esc doesn't arrive). See dictation.md's
|
||
Shortcuts Under Secure Input
|
||
|
||
While a capture records // capture.quiet
|
||
├── Start -> a recording-environment session with General's While dictating
|
||
│ (read at each start; captures have no setting of their own): mutes output
|
||
│ or pauses Music, Spotify, VLC; Do nothing changes nothing. No idle-sleep
|
||
│ assertion, and it never counts as an intentional dictation
|
||
├── Stop, cancel, discard, failed start, captures off, quit -> released; a
|
||
│ switch carries it to the next capture without restoring in between
|
||
├── Dictation shortcut -> starts nothing while it records
|
||
│ (macOS: recognition.rs swallows it in the tap; Linux: linux/dictation
|
||
│ ignores Start). Nothing leaks to the app in front
|
||
└── Dictation that began before the capture -> each effect is held from the first session that
|
||
wants it to the last, so neither unmutes or resumes what the other still
|
||
holds; a While dictating change mid-capture composes too (the capture keeps
|
||
media paused, the dictation mutes). Only what Hex
|
||
changed is restored. The dictation stop tone plays at once while the capture
|
||
keeps output muted. Check: recording_environment tests (overlap both ways),
|
||
capture_host a_capture_stays_quiet_until_it_stops_is_cancelled_or_hex_quits
|
||
|
||
Switching target // capture.switch; the capsule's chips
|
||
├── The recording capsule shows no target chips: a recording is the screen.
|
||
│ Switching (below) remains for the CLI, the picker, and the playground
|
||
├── CaptureCommand::Switch(target) -> discard and start in one step: no snapshot
|
||
│ shows nothing recording in between (Scrawl's switch_recording)
|
||
│ ├── No room to start (two writing, Hex stopping) -> refused `busy` before
|
||
│ │ the discard; the recording goes on
|
||
│ └── The start fails after the discard -> the discarded capture settles
|
||
│ `Failed` with the start's error
|
||
├── Window or Area -> CaptureCommand::Discard: bundle removed, no outcome, gone
|
||
│ from the snapshot at once (and from MAX_FINISHING), so the capsule becomes
|
||
│ the picker
|
||
├── Picker open (take_keys: Picker) -> Hex's tap hands it ⌥⇧S and Esc; it never
|
||
│ takes focus
|
||
├── A pick -> CaptureCommand::Start(Window | Area | Display); nothing while one
|
||
│ records
|
||
├── The picked window closed, or Screen Recording refused -> last outcome
|
||
│ `failed` with its reason (`error`)
|
||
└── iPhone -> CaptureCommand::Switch(Phone) or Start(Phone), while the snapshot
|
||
names a cabled phone (capture.phone)
|
||
|
||
A cabled iPhone // capture.phone
|
||
├── Each capture start, and CaptureCommand::FindPhone (the picker opening) ->
|
||
│ a capture-phone thread runs `scrawl-helper devices` (the USB registry, no
|
||
│ permission) -> CaptureSnapshot.phone = its name, or None
|
||
├── Start(Phone) -> scrawl-helper's mirror window shows the phone and records
|
||
│ its native video into recording.mov; `--no-microphone`: the mirror never
|
||
│ asks for or opens a microphone
|
||
│ ├── Camera access -> macOS asks on Hex's behalf (every bundle carries
|
||
│ │ NSCameraUsageDescription; validate-app.sh requires it)
|
||
│ └── Pointer and pen input -> normalized to the mirror's picture
|
||
├── Narration -> Hex's timeline, as for the screen, moved onto the movie's clock
|
||
│ (its first frame, measured from the capture's start) and transcribed by Hex's
|
||
│ transcriber (capture.transcriber), not Scrawl's
|
||
├── Stop -> as any capture; keyframes decoded from the phone's own video
|
||
└── The mirror ends on its own (unplugged, no phone in 12 s, camera denied) ->
|
||
the listener finishes the capture: written if it recorded, else `failed`
|
||
with the helper's reason
|
||
|
||
The capsule // capture.capsule (rune_ui/capture_capsule)
|
||
├── A Rune overlay at the top of the pointer's display that never takes focus
|
||
│ and lets clicks through except on its live chips, buttons, and list; a
|
||
│ new recording brings it to the pointer's display even while it's up on
|
||
│ another;
|
||
│ its window id goes to the engine (`Host::own_windows`, one id per role, so a
|
||
│ recreated window replaces the old one), so recordings leave it out
|
||
├── Follows `capture_host::watch()`: Recording from the instant `recording` is
|
||
│ Some (the bundle's t0; timer, meter, ✓ finish, ✕ cancel) -> Processing ->
|
||
│ done (the lead alone, no word), Failed, or Cancelled, then leaves on its
|
||
│ own; a written capture whose transcript has no words at all (none timed,
|
||
│ none in its text) reads "no speech"
|
||
├── Tones (`capture_capsule::tone`, at the dictation volume; zero is
|
||
│ silent), from the chosen sound set: start and stop are the set's start
|
||
│ and stop, a send going its done (Annotate's own sent sound when the set
|
||
│ has none), a discarded recording its cancel, a failure its error (Basso
|
||
│ for Classic on macOS); a taken-back send keeps Annotate's own undo sound
|
||
├── Failed -> "Couldn't start" until the engine reports it
|
||
│ recording (`screen`), then "Couldn't save"; a failed send ->
|
||
│ "Couldn't send to X". Each keeps its `error`; a start or save that fails
|
||
│ while Screen Recording is off (a fresh read) opens the permission helper
|
||
│ instead; a start that fails before any snapshot showed it
|
||
│ recording still shows, but never over a newer recording, and a capture
|
||
│ already under way that fails in the background never takes it over
|
||
├── Permission helper (capture.permission) -> "Hex needs Screen Recording ·
|
||
│ to see your screen · Allow ✕" (Microphone, Accessibility alike)
|
||
│ ├── Allow -> macOS's microphone prompt, else System Settings with the
|
||
│ │ floating guide beside it (permission_guide), never at launch
|
||
│ ├── Followed once a second (feed::watch_grant, screen_access_watched) ->
|
||
│ │ "… is on · Restart Hex" when macOS needs one, then "… is on ·
|
||
│ │ Capture ⌥⇧S"; Capture or the shortcut records the pointer's display
|
||
│ ├── ✕ closes it; unanswered it folds after 2 min; the shortcut brings it
|
||
│ │ back as it stood
|
||
│ ├── A capture written with no frames -> sent with only the voice and
|
||
│ │ marks, its summary naming the actual reason the picture is missing
|
||
│ │ (Screen Recording is off for Hex, or the screen source failed or sent
|
||
│ │ no frames), no contact sheet; a permission miss opens the helper once
|
||
│ │ its capsule folds
|
||
│ └── Story: hex://story/prototypes/capture-capsule/permission; goldens
|
||
│ capture-capsule-permission{,-restart,-ready}
|
||
├── A shown recording that leaves with no outcome (a discard, from a chip or
|
||
│ `hex capture discard`) -> the capsule leaves; the picker a chip opened
|
||
│ stays. A switch that can't start settles the recording it discarded as
|
||
│ failed, which the capsule shows as "Couldn't start"
|
||
├── Recording -> `● 0:07 │ ✎ │ ■ ✕`: the timer, the toolbar (the pen, its
|
||
│ only tool, lit with the accent fill while drawing), a divider, stop
|
||
│ (Finish) and ✕ (asks "Discard?" first), all equal square icon buttons
|
||
│ with tooltips "Draw", "Stop", "Discard". Goldens: recording,
|
||
│ recording-drawing (rendered on apollo)
|
||
├── Target chips (playground stories only; feed sends no Input::Chips) ->
|
||
│ Area · Window · Screen, plus iPhone while a phone is cabled, for 3.5 s
|
||
│ ├── `worked` -> they fold once seen for 0.9 s; never while the pointer is on
|
||
│ │ the capsule, and 1.2 s after it leaves
|
||
│ ├── A chip restarts rather than retargets (Scrawl's `switch_recording`)
|
||
│ ├── Screen -> it's what's recording: the chips fold, no command
|
||
│ ├── iPhone -> CaptureCommand::Switch(Phone): the recording restarts on the
|
||
│ │ phone
|
||
│ └── Window or Area -> Discard; the capsule becomes the picker, and
|
||
│ FindPhone looks for a phone cabled since the recording began
|
||
├── Picker: ⌥⇧S records what the on-screen picker shows; Esc closes it
|
||
├── ✓ -> `hex capture stop`
|
||
├── ✕ while recording -> asks first: a destructive "Discard" and "Keep
|
||
│ recording", the recording running on; Discard -> `hex capture cancel`.
|
||
│ Unanswered, it goes back to recording after 4 s (never while hovered, and
|
||
│ 1.5 s after the pointer leaves). Esc never discards
|
||
├── Routing (Scrawl's routing.md) -> `last.route` once its answer arrives; the
|
||
│ capsule follows refinements and delivery separately, including Queued
|
||
│ (then Sent), how an Undo ended, and a failed send
|
||
│ ├── A saved rule -> "Sending to X" and Undo, whose ring drains through the
|
||
│ │ Undo window (5 s) and is gone when Undo is; no words count it down.
|
||
│ │ A repository awaiting its session shows the repository name. Undo
|
||
│ │ cancels before posting. The capsule remains visible until delivery;
|
||
│ │ then it reads "Sent to X" and plays the Sent tone. A picked send still
|
||
│ │ goes immediately, with the same ring on its Undo. No runner-up chip for a rule;
|
||
│ │ after delivery and the Undo window, the capsule leaves by
|
||
│ │ itself (with the list open it stays; the pointer on it holds it, and a
|
||
│ │ folded list gives it a brief hold). Picking the rule's own session there (without
|
||
│ │ "Always") only closes the list; nothing is sent again
|
||
│ ├── The Undo ring -> one per send (rule or pick), tied to the window's end
|
||
│ │ (`Done.countdown`) and drawn by `capture_capsule::undo_ring` as render-time
|
||
│ │ opacity tweens, so nothing ticks: the capsule wakes only when the window
|
||
│ │ ends. Reduce Motion steps it a second at a time. It is the Undo chip's
|
||
│ │ own outline, draining clockwise from the top
|
||
│ ├── The pointer on the capsule -> holds the countdown, one that starts
|
||
│ │ under it too: the ring stands still (Reduce Motion too), Undo never
|
||
│ │ runs out, and a rule's send waits (`CaptureCommand::Hold`). Leaving
|
||
│ │ goes on with the time it had left, at least 1 s. A pick already went,
|
||
│ │ so only its Undo waits (`Flow::hold_countdown`)
|
||
│ ├── A capture of Ghostty -> the focused pane's folder and title
|
||
│ │ (Ghostty's AppleScript), and what runs in front in it, read from the
|
||
│ │ process tree under Ghostty (libproc: the shell's tty and foreground
|
||
│ │ group, that process's name, arguments and folder; no new permission).
|
||
│ │ Several panes are told apart by title and folder; still unclear, the
|
||
│ │ process is left out (`look::terminal`)
|
||
│ │ ├── OpenCode's TUI in front -> the session it shows is the top
|
||
│ │ │ suggestion, "open in this terminal": the session titled as the
|
||
│ │ │ terminal's `OC | <title>` (a cut title matches the long one,
|
||
│ │ │ one in the pane's folder first), else the `-s ses_…` it was
|
||
│ │ │ started with. A suggestion like any other, never sent by itself
|
||
│ │ ├── OpenCode's home screen (`OpenCode`) or a plain shell -> the
|
||
│ │ │ folder and the title's words are clues, as before
|
||
│ │ ├── ⇧ (a folder that isn't home or `/`) -> the rule is
|
||
│ │ │ `routing.terminals` keyed on the folder's project root ("hex
|
||
│ │ │ terminal" in the header and the footer's app badge), naming the
|
||
│ │ │ session picked; it covers that folder and everything under it (the
|
||
│ │ │ longest key wins) and sends like a site rule. Ghostty itself stays
|
||
│ │ │ a general app, so no rule ever sends all of Ghostty
|
||
│ │ └── Checks -> `look::tests` (TUI session first, command-line
|
||
│ │ fallback, home screen, home folder, terminal rule),
|
||
│ │ `look::terminal::tests`, `policy` and `config` rule keys, and
|
||
│ │ `capture_host::routing` `an_always_rule_from_a_terminal_keys_on_its_folder`.
|
||
│ │ The process walk was measured once against a real Ghostty; no
|
||
│ │ end-to-end capture of a TUI was run
|
||
├── A finished capture reads "Annotation". While routing is pending it reads
|
||
│ "Looking for sessions…" instead. Copy failure has its own warning
|
||
│ ├── Anything else with somewhere to go (a guess, nothing fitting, no
|
||
│ │ answer) -> macOS: "Annotation", Paste, a magnifier and ✕. The
|
||
│ │ magnifier is named Search with the current binding. Suggestions open under
|
||
│ │ it; nothing sends until Kit picks. Linux: closed,
|
||
│ │ with Paste and Send to…, which opens them on a click
|
||
│ │ ├── "Send to" over up to five rows, suggested then the newest
|
||
│ │ │ sessions, with no numbers and no footer; "⇧-click to always send
|
||
│ │ │ [site] here" appears on hover when a pick can save a rule.
|
||
│ │ │ The first carries its reason in the accent. A reason is a real one
|
||
│ │ │ in plain words ("you were just in it", "same page", "mentioned",
|
||
│ │ │ "recent") or nothing; never "guess". No row moves once shown: a
|
||
│ │ │ later answer gives a row its new reason in place and adds new
|
||
│ │ │ sessions last (`routes::keep_rows`, as the router does)
|
||
│ │ ├── Shift down (a guess or rule with a site, terminal folder or app) -> the row under
|
||
│ │ │ the pointer reads "always" in the accent and the header reads
|
||
│ │ │ "Always send [site] to [row]" (with the pointer off the rows,
|
||
│ │ │ "Always send [site] to…" and no row is marked). In the search the
|
||
│ │ │ target is the row under the pointer, else the highlighted one: the
|
||
│ │ │ highlight follows the pointer as in a menu and the arrows go on
|
||
│ │ │ from it. The footer reads "Always send [site] to [row]", the site
|
||
│ │ │ and the row's name each in a rounded-rect badge with its kind's
|
||
│ │ │ icon (a globe for a site, a window for an app, a bubble for the
|
||
│ │ │ session). Shift+click (or ⇧⏎ in the search)
|
||
│ │ │ sends and saves the rule: "Sent to X · always", and its Undo
|
||
│ │ │ takes back the send and the rule (what the key held before, or
|
||
│ │ │ nothing, comes back). The rule names that session (`ses_…`),
|
||
│ │ │ never its folder: a folder rule goes to the folder's newest
|
||
│ │ │ session, so with SEER and RUNE DESKTOP in one checkout, a rule
|
||
│ │ │ saved for RUNE DESKTOP went to SEER
|
||
│ │ ├── A click -> sends at once: "Sent to X", the Sent tone, and a
|
||
│ │ │ Undo with its ring for the same 5 s; ✕ closes;
|
||
│ │ │ the capsule then leaves by itself with its exit
|
||
│ │ ├── Search or the capture shortcut (⌥⇧S) -> the search: "All
|
||
│ │ │ sessions" over every session the router's look read (its two
|
||
│ │ │ newest pages, every folder's, and any it named), filtered by
|
||
│ │ │ what's typed: every word, in any case, in the title, the folder's
|
||
│ │ │ name, or the session id, rows with every word in the title first;
|
||
│ │ │ a row names its folder before its note when only the folder or id
|
||
│ │ │ matched, or when another row shows the same title (suggestions
|
||
│ │ │ too). Five rows at most and never shorter than the
|
||
│ │ │ suggestions it opened from; the footer reads "Send ⏎",
|
||
│ │ │ "Remember ⇧⏎", and "New capture ⌥⇧S". Matching nothing, it reads
|
||
│ │ │ "No sessions match" and the footer drops Send and Remember. Esc or a click away goes
|
||
│ │ │ back to the suggestions; ⌥⇧S in the search records a new capture
|
||
│ │ ├── Once the capsule has gone, ⌥⇧S records as usual
|
||
│ │ ├── Esc -> not Hex's here: it goes to the app in front
|
||
│ │ └── Unpicked -> they leave with the capsule after the done hold
|
||
│ ├── Taken back (a rule's send or a pick) -> never
|
||
│ │ ends the capture: the list opens again with the same sessions, the one
|
||
│ │ it went to (by session id) once, last of all ("taken back"), with
|
||
│ │ clicks, Search, and ⇧-click as before. Open, it stays, sent nowhere; a click away
|
||
│ │ folds it to "Taken back" with "Send to…", and it leaves after the done
|
||
│ │ hold; nothing here takes Esc. ✕ (or Esc in the typed list) dismisses:
|
||
│ │ "Kept", the bundle untouched. Either way it then
|
||
│ │ waits for Send to…, even after the router said the pick went. With no
|
||
│ │ other session to offer, the list still opens with the one it went to
|
||
│ │ ("taken back"), so Search can send it elsewhere; only a rule still
|
||
│ │ resolving its folder reads "Taken back" with nothing to list.
|
||
│ │ News of the send being taken back (it went, queued, or failed) leaves
|
||
│ │ that list alone, and a pick made before the router answers the Undo
|
||
│ │ stands: the Undo's answer and the earlier send's news never take it back
|
||
│ ├── Too late to take back -> the same list, the session marked "already
|
||
│ │ sent", folded "Already sent to X"; a pick sends a copy there,
|
||
│ │ and ✕ just closes (it leaves by itself after the brief hold). Closed, it doesn't wait for a pick.
|
||
│ │ An Undo OpenCode refuses (an error) reads as too late
|
||
│ ├── Sends -> CaptureCommand::Send{to} and Undo, naming the
|
||
│ │ session by id (`RouteDest.session`), so sessions sharing a title never
|
||
│ │ collide; the capsule shows titles
|
||
│ ├── Keys (capture_host::take_keys), never taking focus:
|
||
│ │ ├── Picker -> ⌥⇧S and Esc through the listener
|
||
│ │ ├── Suggestions -> ⌥⇧S through the listener
|
||
│ │ │ (`ShortcutActivity::Capsule`), which opens the search
|
||
│ │ ├── Routed or closed -> no keys: Send to… is click-only, and never
|
||
│ │ │ Esc, so a send is taken back only by clicking Undo
|
||
│ │ ├── A send from the list still undoable -> no keys; Esc goes to the
|
||
│ │ │ app, and the Undo button is the only way back
|
||
│ │ ├── Suggestions, the pointer moved onto them -> also text typed
|
||
│ │ │ without ⌘ or ⌃ (not a leading space), which opens the search
|
||
│ │ │ with it (`CapsuleKeys::Suggest { search }`). Only a move counts:
|
||
│ │ │ suggestions that open or change under a resting pointer, or a
|
||
│ │ │ pointer that left, leave typing with the app in front, so Kit
|
||
│ │ │ typing there never loses a letter. ⌫, ↑↓, ⏎, and Esc stay the
|
||
│ │ │ app's until the search opens
|
||
│ │ │ (`typing_stays_the_apps_until_the_pointer_moves_onto_the_suggestions`)
|
||
│ │ ├── Suggestions -> at the session tap no key at all (the toast's
|
||
│ │ │ ⏎ and ⇥ aside), only a mouse press, which folds them (the click
|
||
│ │ │ still reaches the app) unless the pointer is on the pill or the
|
||
│ │ │ list under it: the tap reports the press before the overlay gets
|
||
│ │ │ it, so a click on a row would otherwise fold the row away first
|
||
│ │ │ (`clicks_on_the_capsule_do_what_they_say`, headless pointer
|
||
│ │ │ events, not native); every key, digits and Esc included, goes on
|
||
│ │ │ to the app. Shift's presses and releases (flagsChanged) reach the
|
||
│ │ │ capsule too (`CapsuleKey::Shift`) and still go on to the app
|
||
│ │ └── List open -> the session tap takes every key press without ⌘ or ⌃
|
||
│ │ that isn't a Hex shortcut: text in the active layout, ⌫, ↑↓, ⏎,
|
||
│ │ ⇧⏎ (always), Esc; their releases too. Digits are typed (session
|
||
│ │ names have them), and ⌥⇧S goes on to the listener, which records. A send, Esc, or a
|
||
│ │ press off the capsule (the tap reports mouse-downs while the list
|
||
│ │ is open, and still delivers them) closes the list and gives the
|
||
│ │ keys back; hiding drops the grab
|
||
│ │ └── Windows -> windows/input's keyboard hook stands in for the tap
|
||
│ │ (#2113): the same keys, its text read with ToUnicodeEx in the
|
||
│ │ foreground window's layout and never with the layout's state
|
||
│ │ changed (a dead key types nothing in the search, and AltGr's
|
||
│ │ Ctrl+Alt stays the app's), a press kept only while no shortcut of
|
||
│ │ Hex's claims it, and a key the app already holds left to it;
|
||
│ │ never observed on a Windows PC
|
||
│ ├── `offer` (a rule offer) -> not drawn; Scrawl's capsule doesn't draw it
|
||
│ ├── The ready capsule shows a labeled Paste button, both folded and above
|
||
│ │ the suggestions. A rule waiting to send shows Undo instead. Sending,
|
||
│ │ sent and queued results do not offer Paste. The search field replaces
|
||
│ │ the buttons while typing. Neither Paste nor Inspect binds a key.
|
||
│ │ The list hangs
|
||
│ │ under the capsule, which is never narrower than LIST_W; the capsule
|
||
│ │ widens with it. The capsule is a rounded rect with the list's 14 pt
|
||
│ │ corners (`SHELL_RADIUS`), not a pill, and its chips and icon buttons
|
||
│ │ round concentrically inside it. The line and its list are one layout:
|
||
│ │ the list's header and its names start 16 pt in from the wall, and
|
||
│ │ the reasons end 16 pt in from the other, under ✕. The preview is not
|
||
│ │ on that margin: it nests in the capsule's left corners, 10 pt from the
|
||
│ │ left, top and bottom walls, so its 4 pt corners turn about the same
|
||
│ │ centres as the shell's 14 pt ones
|
||
│ │ (`the_preview_nests_in_the_capsules_left_corners`). Paste, Search,
|
||
│ │ Inspect and ✕ are one row of 28 pt
|
||
│ │ boxes 6 pt apart and 6 pt in from the walls; Paste's is filled, and
|
||
│ │ its 32 pt chip is still what takes the click
|
||
│ │ (`the_ready_line_and_its_list_share_their_edges`). Two looks while Kit picks (tuning `top_line`):
|
||
│ │ TopLine::Quiet (default) says the label in quiet MEDIUM ink;
|
||
│ │ TopLine::Icons uses stronger ink. Both retain the annotation label
|
||
│ ├── The lead is the capture's first frame (`frames/frame-01.png`, else the
|
||
│ │ contact sheet), cropped to a small rounded card once the bundle is
|
||
│ │ written and decoded off the UI thread; until then, or without one, a
|
||
│ │ check. A warning keeps its glyph
|
||
│ │ ├── Paste -> CaptureCommand::Paste: the contact sheet, then the summary,
|
||
│ │ │ into the focused field, from the bundle shown, not the newest output.
|
||
│ │ │ No Return, send, Undo or rule change
|
||
│ │ └── Click the finished preview, when Inspect is available, or the
|
||
│ │ Inspect button on other outcomes -> opens the contact sheet (Hex
|
||
│ │ has no review window, and opening the bundle could launch
|
||
│ │ Scrawl.app)
|
||
│ ├── Undo is a click only, with no shortcut: Escape meant for another app
|
||
│ │ was taking sends back
|
||
│ ├── Search's tooltip and accessible name include the configured capture
|
||
│ │ shortcut; no keycaps appear in the ordinary completed header. The
|
||
│ │ search's footer reads "Send ⏎", "Remember ⇧⏎", "New capture ⌥⇧S"; the
|
||
│ │ cursor's and the notch's cards end in "Search ⌥⇧S" under their guesses.
|
||
│ │ The footer sits apart from the rows in one of
|
||
│ │ two looks while Kit picks (tuning `list_footer`): ListFooter::Band
|
||
│ │ (default), a full-width band on its own fill under a rule, or
|
||
│ │ ListFooter::Well, a rounded well inset like the rows
|
||
│ └── Long names give way together so the widest row stays inside the panel
|
||
├── Every change moves at once, as Dynamic Island's: the walls and what stays
|
||
│ travel on `island`, the dictation HUD's morph (`rune_hud::spring::MORPH`,
|
||
│ 0.30 s, bounce 0.15; the entry's rise and exit's fall too, with the
|
||
│ shell fading over the same time unbounced), while everything leaving and
|
||
│ everything arriving blurs (`content_blur` 3 pt), scales toward the
|
||
│ capsule's middle (`content_scale` 0.92), and fades over that same move
|
||
│ (`content_in` 0.30 s, `content_out` 0.24 s). Nothing staggers, and nothing
|
||
│ waits for a wall or for other content; only an entry takes two frames
|
||
│ (the origin pose, then rest). The list under the capsule comes and goes
|
||
│ on the same springs. The playground's Transition card tunes all of it
|
||
├── Reduce Motion -> the capsule enters at rest and leaves where it is (no
|
||
│ origin pose, menu-bar fold, or edge glow), changes jump, nothing blurs
|
||
│ or scales; the shell, at the notch too, fades on the HUD's short fade
|
||
│ (`reduce_motion_only_fades_the_capsule`)
|
||
└── After a capture, one of five flows (`capture.capsule_flow`, the
|
||
playground's "Flow after a capture"; prototypes to compare, clips in
|
||
r22-flows). Only the guesses' look changes; rules, Undo, ✕ and the
|
||
keys the capsule takes stay the same. Each capsule story's variants
|
||
pick one, so a link opens a state in a flow:
|
||
`hex://story/prototypes/capture-capsule/suggestions?variant=toast`
|
||
├── list (default) -> today's list under the capsule, as above
|
||
├── toast -> one line "Send to X ⏎" with "Next ⇥" for more than one
|
||
│ guess; ⇥/⇧⇥ cycle (CapsuleKeys::Suggest { cycle }), ⏎ or a click on
|
||
│ the line sends the one shown
|
||
├── menu_bar -> the frame plus "NAME 1/2/3" chips for MENU_HINT_S (4 s),
|
||
│ then the capsule folds into the menu-bar item
|
||
├── cursor -> recording stays under the menu bar; the "Send to" card
|
||
│ (number, name, reason) opens at the pointer and its Sent stays there
|
||
└── notch -> a black shape grows out of the notch (the measured one on
|
||
macOS's pointer screen, read when the panel may go to that display:
|
||
a new capture, an anchor change, the capsule coming back from
|
||
hidden, or while the picker is open. A capsule that is showing and
|
||
not following stays put when the pointer moves, whatever the notch
|
||
there reads; left and right edges snapped to the backing pixels, the
|
||
panel moved under its middle and pinned to the display it was
|
||
measured on, so the panel and the notch can't disagree. Pinned, Rune
|
||
doesn't follow the pointer for it; while the picker is open, Hex
|
||
re-reads the pointer's screen at the picker's display-check cadence
|
||
(`picker::DISPLAY_EVERY`, 100 ms) and re-places display and inset
|
||
together, so the capsule crosses with the picker within that
|
||
cadence. Only the fallback that couldn't read the screen keeps
|
||
`UnderPointer`, where Rune can show one frame at the previous inset
|
||
before Hex re-places it; 185 pt
|
||
on a flat display, a pretend one the menu bar's height, so its panel
|
||
lands within a point of the display's middle rather than exactly on
|
||
it) into the pill and the same card, its content under the notch,
|
||
and shrinks back into it
|
||
(capture_capsule/notch.rs). The playground's "Prototypes/Capture capsule/notch"
|
||
story plays it on a 14-inch MacBook Pro and a 27-inch display, side
|
||
by side or each whole, at their real sizes
|
||
The playground's "Prototypes/Capture capsule/family" story (playground/family.rs)
|
||
plays a dictation and then a capture in one shape on the same springs:
|
||
from the notch on a 14-inch MacBook Pro, and on a flat 27-inch display
|
||
inside the menu bar (hud_placement's 3 pt inset) or floating 6 pt under
|
||
it. Its variants draw the voice as Dot scope's braid (its real
|
||
simulation), bars, or a line; only the capture adds a red dot or a small
|
||
screen, the cue alternating each loop
|
||
Gap: only the shapes are prototyped; the menu-bar item's capture list
|
||
is not, the app reads the notch's edges but not its height (the walls
|
||
run up past the screen's top), and no flow but list has run live
|
||
|
||
The on-screen picker // capture.picker (capture_capsule/picker.rs)
|
||
├── A full-screen Rune overlay on the pointer's display, under the capsule,
|
||
│ left out of recordings (`OwnWindow::Picker`); it takes the pointer only while
|
||
│ picking and never takes focus; it follows the pointer to another display
|
||
├── Made at launch, hidden; away from macOS its window holds no GPU surface
|
||
│ while hidden (`release_gpu_when_hidden`), so a Hex that never annotates
|
||
│ holds no GPU memory for it (`perf/idle.bench.ts#window-closed-memory`,
|
||
│ `gpu.vram`)
|
||
├── Everything but the target dims; the dim moves with the target as one piece
|
||
├── Window -> the window under the pointer: tinted, ringed, its title in a badge;
|
||
│ the cursor becomes Scrawl's camera; no window under it -> the display
|
||
├── Area -> drag at least 36 pt; the badge shows its size; a click alone -> the
|
||
│ display
|
||
├── Screen -> the display, ringed
|
||
│ └── An area's ring is white with a dark hairline outside it, so it reads
|
||
│ over white content too (Scrawl's vanishes there)
|
||
├── Click or ⌥⇧S -> Action::Start(target) -> CaptureCommand::Start; the picker
|
||
│ fades; a pick that starts no capture within 2 s brings it back
|
||
└── Windows load off the UI thread (`list_selectable_windows`); nothing picks
|
||
until they arrive
|
||
|
||
Hex's own windows // capture.own-windows
|
||
├── Chrome (the capsule, ink, picker, HUDs, meeting capsules: Hex's named
|
||
│ windows and any of its windows above the normal level) -> never recorded,
|
||
│ never in the window log, never the target, clicks on it not logged
|
||
│ ├── While the pen is on, the ink overlay holds the keys, so the window
|
||
│ │ log's system-wide focused-application read names Hex itself; AppKit
|
||
│ │ answers that read on the asking `scrawl-window-log` thread, and the
|
||
│ │ vendored AccessKit macOS adapter now answers such off-main reads as a
|
||
│ │ plain view instead of borrowing its main-thread state. That borrow
|
||
│ │ aborted Hex mid-capture (#1859, Rune #1040); a test would need AppKit
|
||
│ │ and a key window, so none covers it
|
||
│ ├── Other recorders -> the capsule, picker, and ink are left out
|
||
│ │ (sharingType none) while shown; the dictation HUD and meeting
|
||
│ │ capsules show
|
||
│ │ └── Hidden -> ordered out of every Space once the hide finishes,
|
||
│ │ before sharingType returns to readOnly; prewarming keeps the
|
||
│ │ surface ready but leaves no invisible panel ordered in. This
|
||
│ │ avoids relying on macOS restoring sharing after exclusion
|
||
│ ├── Show in screen recordings on -> every recorder, Hex's captures
|
||
│ │ included, shows all of it (sharingType readOnly, no SCK exclusions);
|
||
│ │ still never in the window log or the target. Rune's
|
||
│ │ OverlayHandle::set_exclude_from_capture flips open windows, and the
|
||
│ │ sampler rebuilds its filter mid-recording (OwnWindows::recorded) and
|
||
│ │ the recording's (capture-core's Recording::set_exclusions). macOS's
|
||
│ │ pump and Windows' runtime thread follow the setting while Hex runs
|
||
│ │ (CaptureOverlays::follow_recordings), so it applies live on both, not
|
||
│ │ only to windows made after it changed
|
||
│ └── Rune's presence tests check ordering and sharing together, including
|
||
│ stale fade completions and setting changes. Its native `spaces_probe`
|
||
│ checks this process's WindowServer ordering. A native Hex smoke on
|
||
│ 2026-11-02 failed before the pin bump and passed after it: capsule,
|
||
│ Screen/Area picker, ink, dictation HUD and meeting capsule show and
|
||
│ leave no onscreen window after hiding. This used isolated test state,
|
||
│ without recording. Mission Control's actual thumbnail report and
|
||
│ third-party recording exclusion remain unverified
|
||
│ └── Click-through chrome (the full-screen ink overlay, the capsule's
|
||
│ 640×424 panel outside its pill) -> clicks there count for the app
|
||
│ beneath; only the pill's (capture_host::set_capsule_pill) are Hex's
|
||
└── Normal windows (Settings) -> like any app's: recorded, logged, pickable,
|
||
and Hex is the front app when its window is the topmost normal one, so a
|
||
capture of Settings routes by Hex's name. A capsule Paste still goes to
|
||
the last other app, never into Hex
|
||
|
||
Ink and laser // capture.ink
|
||
├── A click-through overlay over the recorded display, excluded from the
|
||
│ recording; on a Wayland monitor pick the portal names (`mapping_id`) it
|
||
│ is recreated on that output, and strokes use its xdg-output logical size
|
||
│ (see the Linux pen bullet above)
|
||
├── Shift strokes draw as ink, hold 650 ms, then fade over 650 ms
|
||
└── The laser's tail lasts 420 ms; nothing lags the pointer
|
||
|
||
While a capture records // capture.input
|
||
├── The existing session tap adds mouse down/up/drag/scroll and Shift (no second tap)
|
||
├── -> Capture::input() through a bounded queue; full -> events dropped and counted
|
||
├── Keys pressed in the app -> logged; keys Hex consumes (its shortcuts, Esc) and
|
||
│ anything while a shortcut is being recorded -> not logged
|
||
├── Scrawl's rules in the engine (hosted_input): Shift drag inks, a
|
||
│ Shift-click is part of the drawing (no pins, no ⌘Z undo), clicks on
|
||
│ Hex's capture chrome ignored; while drawing, capture_host::as_pen makes
|
||
│ every pointer event Shift ink
|
||
└── Shift still held from ⌥⇧S -> must come up once before it can ink; Shift up
|
||
at the start -> the first Shift drag inks
|
||
|
||
hex capture start // capture.start; needs a listening Hex
|
||
├── Creates <captures folder>/<unix ms>.scrawl (owner-only) -> reply: recording + bundle path
|
||
├── Two stopped captures still writing -> refused `busy`
|
||
├── Copies the dictation timeline from now into a bounded sink (no second stream)
|
||
├── In the background: Scrawl's engine starts on the active display
|
||
│ ├── Needs Screen Recording for Hex (or the terminal running it)
|
||
│ └── Fails to start -> sink closed, bundle removed, last outcome `failed`;
|
||
│ Esc and the next ⌥⇧S see nothing recording
|
||
├── Release microphone while idle -> the capture keeps the microphone open
|
||
│ (even while the display sleeps or the session is locked)
|
||
└── Already recording -> unchanged, same bundle
|
||
|
||
While it records // capture.audio
|
||
├── Dictation and meetings work as usual on the same audio
|
||
├── Sink full (collector stalled) -> capture audio dropped as a gap, dictation untouched
|
||
└── Microphone gap or recovery -> the gap is recorded, positions stay on the clock
|
||
|
||
Never on Hex's path // capture.isolation
|
||
├── No microphone, event tap, global shortcut, or model of its own
|
||
├── No ingest server (47777) or recovery workers; delivery retries only while
|
||
│ neither Scrawl.app runs nor anything serves the ingest port (capture.route)
|
||
├── No ingest token used; Scrawl's two config files read once by the import, never
|
||
│ written; nothing else of Scrawl's touched (its log goes to Hex's tracing)
|
||
└── Settings from Hex (capture.*), not Scrawl's config: auto-crop on (display
|
||
captures), the agentContext summary format off
|
||
|
||
Retention // capture.retention
|
||
├── Hex's own thread: Scrawl's retention pass with capture.*'s policy, 90 s after
|
||
│ captures turn on, then hourly; skipped while a capture records or writes,
|
||
│ or while Scrawl.app runs (it cleans the shared folder then)
|
||
├── Linux sends media to the desktop Trash itself (the freedesktop.org Trash
|
||
│ layout, no `trash` command; macOS uses /usr/bin/trash, Windows the
|
||
│ Recycle Bin), so file managers can restore it
|
||
├── `.media-cleaned` is written only after the media reaches the Trash; a
|
||
│ failed trash leaves the media in the bundle. Staging left by an earlier
|
||
│ version or an interrupted pass is put back on the next pass, and staging
|
||
│ whose bundle is gone goes to the Trash
|
||
└── Old media goes to the Trash and the prompt stays; after bundleDays the
|
||
whole bundle goes
|
||
|
||
hex capture stop // capture.stop
|
||
├── Closes the sink (≤ 100 ms wait) -> reply: idle + the bundle path, at once
|
||
├── The screen recording, window log, and input stop at once (Capture::stop)
|
||
├── recording.mp4 finishes after that on capture-core's own thread: detached
|
||
│ before its stream stops, its final frame held to the stop, capture.json
|
||
│ written beside it; only the steps that read the movie wait for it, and it
|
||
│ starts where its first frame was shown (video_offset_ms)
|
||
└── In the background, on one capture-finish thread, one capture at a time:
|
||
├── 16 kHz audio streamed to audio.wav.partial as it records, and renamed to
|
||
│ audio.wav only once this build writes the capture's unfinished.json: a
|
||
│ streamed capture's audio.wav never exists without a journal, so one
|
||
│ still recording or waiting its turn never looks recoverable (a crash
|
||
│ there leaves the partial, which retention cleans up like any other
|
||
│ media). Leading start offset and gaps are written as silence, so WAV
|
||
│ time stays capture time
|
||
├── Words: on the Parakeet and GGUF paths, audio.wav read in 30 s pieces
|
||
│ (iPhone and replay narration are decoded in memory), each a foreground
|
||
│ job on the dictation engine's model, after queued dictation, before
|
||
│ meeting chunks; waits while dictation records or transcribes; never
|
||
│ interrupts a live stream. Apple reads the file itself
|
||
├── Word times moved earlier by the model's measured lag (Parakeet Unified
|
||
│ English: starts −280 ms, −520 ms after a ≥ 250 ms pause; ends −195 ms);
|
||
│ other models and Apple keep their own times
|
||
├── A model that gives no word times (GigaAM v3) has each 30 s piece's
|
||
│ text spread over that piece's own audio (a clip held in memory, over
|
||
│ the whole of it) as estimated times, so keyframes and the editor have
|
||
│ somewhere to put it; a bundle already on disk without them counts and
|
||
│ spreads its own text
|
||
│ (`scrawl_capture::transcribe::word_count`, the Captures pane's
|
||
│ `words_for_editor`)
|
||
├── Past 8 h -> audio.wav stops growing and the rest is one gap; a disk
|
||
│ guard, never a memory bound (iPhone and replay narration are decoded in
|
||
│ memory and so are not covered by it), and scrawl.json's
|
||
│ audioStoppedAtMs makes summary.md say "Audio stopped at 8:00:00; the
|
||
│ rest of this capture has no audio" (a failed audio write says where too)
|
||
├── Cancel or a failed screen source -> the partial audio is closed and
|
||
│ removed with the bundle; a crash leaves it, and retention trashes it
|
||
│ like any other media (a movie and a partial are never a bundle recover
|
||
│ rebuilds)
|
||
├── recording.mp4, capture.json, audio.wav, frames/, scrawl.json,
|
||
│ summary.md; then every file 0600 and folder 0700
|
||
├── Keyframes decoded from recording.mp4 by scrawl-helper (beside `hex`: in
|
||
│ Hex.app's Contents/MacOS or target/<profile>; or $SCRAWL_HELPER); helper
|
||
│ missing -> the 480 px sampled frames
|
||
├── Spoken cues (look, now, here, this…) pick the keyframe the screen
|
||
│ changes to; they also count in Spanish (mira, ahora, aquí, esto…),
|
||
│ French, German, Portuguese, and Italian, whatever the dictation
|
||
│ language. Check: scrawl-capture keyframes
|
||
│ spoken_cues_count_in_other_languages
|
||
├── Transcription fails -> logged; the bundle has no words, and the outcome says why
|
||
├── Written -> image and text copied to the clipboard (Scrawl's handoff),
|
||
│ or the light zip alone with Labs › Copy a capture as one file (capture.copy)
|
||
└── Sent -> the capture-route thread (capture.route)
|
||
|
||
Copying a written capture // capture.copy; Labs › Copy a capture as one file
|
||
├── Off (default) -> the bundle's summary text on the clipboard, as before:
|
||
│ macOS and Windows through Scrawl's handoff (Windows may also put the
|
||
│ contact sheet PNG when the clipboardImage experiment is on), Linux the
|
||
│ summary text through Hex's paster
|
||
├── On -> crates/capture/src/slim_zip.rs writes <stem>.scrawl.slim.zip inside
|
||
│ the bundle (a temp name, then rename) and puts only that file on the
|
||
│ clipboard: macOS a file URL through handoff::copy_file_to_clipboard (the
|
||
│ screenshot card's file copy uses it too), Windows CF_HDROP (a DROPFILES
|
||
│ header and a double-NUL-terminal UTF-16 path list), Linux text/uri-list
|
||
│ (the X11 `text/uri-list` target, or wl-copy --type text/uri-list)
|
||
│ ├── Contents: the bundle's files recursively, paths relative to its root,
|
||
│ │ STORED (no compression), UTF-8 names, a DOS timestamp
|
||
│ ├── Left out: HEAVY_MEDIA (recording.mp4, recording.mov, audio.wav,
|
||
│ │ frames-full/, contact-sheet-full.png), .media-pending, .media-sending,
|
||
│ │ .deliver.json and its .tmp, and any earlier *.slim.zip or temp file;
|
||
│ │ symlinks are skipped
|
||
│ ├── summary.md inside the zip is generated with the bundle folder as `.`,
|
||
│ │ so its header reads anywhere it is unzipped; the summary.md on disk
|
||
│ │ is unchanged
|
||
│ ├── A failed build or clipboard write is logged, and the capture copies
|
||
│ │ as it did before
|
||
│ ├── Refused rather than written as ZIP64 at 0xFFFF files or 0xFFFFFFFF
|
||
│ │ bytes
|
||
│ ├── A terminal paste gets no text: the clipboard holds only the file
|
||
│ ├── scrawl.json and summary.md still name media the zip leaves out
|
||
│ │ (recording.mp4, audio.wav, frames-full/), because they describe the
|
||
│ │ whole capture
|
||
│ └── The capsule's Paste, Inspect, Send to…, routing, and retention still
|
||
│ see the same bundle; the zip goes when the bundle does
|
||
│ // checks: `cargo test --manifest-path crates/capture/Cargo.toml --lib slim_zip`
|
||
│ // (CRC-32 vector, central-directory read-back, excludes, header path,
|
||
│ // a second copy); a_file_is_one_percent_encoded_uri_list_line (linux/paste.rs);
|
||
│ // copy_capture_zip_is_its_own_switch_and_everywhere (labs.rs)
|
||
│ // not verified: pasting the zip into a real chat or agent app on any OS,
|
||
│ // and the Windows clipboard write (cross-compiled only)
|
||
|
||
What a send carries // capture.send (router opencode::send_bundle)
|
||
├── One prompt: the summary (transcript, marks, notes) as text, then the
|
||
│ contact sheet as one inline `data:image/png;base64` file part
|
||
│ (12 MiB encoded at most; past it, no image). The keyframes are not
|
||
│ attached; the text says they are in `frames/` and `frames-full/` of the
|
||
│ capture folder. The picture does not depend on the folder, so an OpenCode
|
||
│ server on another machine reads it; the text only names where it was recorded
|
||
├── Every request goes to the running local service over HTTP
|
||
│ (service.json); a service that isn't up is never started. On Windows with
|
||
│ Labs › OpenCode in WSL on, a WSL service that answers is used ahead of a
|
||
│ Windows one; the Windows service is used when WSL answers nothing
|
||
│ (capture.route). The prompt's "Recorded at" line names the
|
||
│ capture folder as its Linux `/mnt/…` path; the summary header keeps the
|
||
│ Windows path
|
||
├── With an agent host (routing's agentHost), the request body travels on SSH
|
||
│ stdin to a Python 3 standard-library HTTP client. The contact sheet stays
|
||
│ inline; the bundle copy follows the accepted prompt in the background
|
||
│ (the prompt says the folder may still be arriving). Credentials stay remote.
|
||
│ An explicitly configured agentOpencode keeps the CLI and synchronous copy;
|
||
│ without Python 3, the default path falls back to the CLI before any HTTP send
|
||
│ The light-files tar leaves out any `*.slim.zip` and its temp file too:
|
||
│ that copy is this Mac's clipboard attachment, not the capture
|
||
│ └── macOS/Linux reuse the SSH connection for 60 s after its last use,
|
||
│ through an owner-only control socket; Windows opens each connection.
|
||
│ API reads now honor their caller's deadline over SSH too, including
|
||
│ the optional 600 ms idle-status check before a prompt. A failed idle
|
||
│ check records unknown status and still sends. Neither CLI startup nor
|
||
│ the light-bundle copy is on the default prompt's critical path
|
||
├── No video: a capture's movie is too large for a prompt
|
||
└── Check: router opencode tests; a send from apollo to a throwaway session
|
||
on OpenCode V2 stored the contact sheet and keyframe as inline image/png (before one-sheet)
|
||
(2026-10-29)
|
||
|
||
Where it goes // capture.route; capture.routing's autoSend on
|
||
├── Once recording starts -> a capture-look thread reads the tab,
|
||
│ terminal, and sessions in front (Scrawl's router, in Hex's process, with
|
||
│ Scrawl's own router folders, so `scrawl router undo` and its evals see Hex's
|
||
│ sends as Scrawl's). The whole look has 15 s: steps not begun by then are
|
||
│ skipped, and routing waits 1 s more for the one under way, then goes on
|
||
│ without it. The newest-session list starts beside local sensing; exact
|
||
│ folders are read in the other lane as soon as the clues name them (at most
|
||
│ two API reads in flight, one shared 10 s session budget). Results merge in
|
||
│ the same order; missing session IDs are read afterward. Saved rules match
|
||
│ locally from the captured bundle ID or observed URL; suggestions read the
|
||
│ list afterward. A saved rule takes precedence over a spoken session name.
|
||
│ An app rule's folder is read with the sessions, and a failed
|
||
│ page of sessions keeps what was read and still reads the folders and
|
||
│ sessions that rules name
|
||
│ └── Timings in process.log: local sensing, total lookup, page publisher,
|
||
│ written-to-look wait, judging, and sending (sync, idle check, post).
|
||
│ Fake-dependency benchmarks and their proof limits:
|
||
│ [Annotate routing latency](../../perf/annotate-routing.md)
|
||
├── Written -> the one capture-route thread (queue 16) decides; the snapshot's
|
||
│ `last.route` follows: Looking -> Rule | ResolvingRule | Guess | Nothing |
|
||
│ NoAnswer | Unreachable. Background refinements preserve an open search and
|
||
│ selected session; a pick or Undo cancels them
|
||
│ ├── Rule -> only a rule the person saved (`routing.sites`, `routing.apps`,
|
||
│ │ `routing.terminals`, a Shift pick's): a three-second countdown begins
|
||
│ │ when the rule is shown. A folder
|
||
│ │ rule shows its repository while its session resolves; it sends once both
|
||
│ │ the countdown and resolution finish. Undo during the countdown posts
|
||
│ │ nothing. The capsule's `Hold` pauses the countdown while the pointer is
|
||
│ │ on it and resumes it with what it had left (at least 1 s; a hold the
|
||
│ │ capsule never ends lapses after 2 min, and the next capture resumes
|
||
│ │ it). Sending -> Sent, or Queued when OpenCode didn't take it. Whatever
|
||
│ │ autoSend, smartRouting, or jevRouting say, no page or publisher can
|
||
│ │ authorize a send or override a saved rule. smartRouting off decides with the port
|
||
│ │ ladder, after rules and addresses, on the same look
|
||
│ ├── Guess -> everything else the router would send (Jev's pick, HOME when
|
||
│ │ Jev couldn't tell, a dev server's port, a page's opencode-session tag,
|
||
│ │ `.opencode-session` file, data-session or
|
||
│ │ publisher, a name that was said, the captured app's checkout (the one
|
||
│ │ open session folder named like the app, such as `…/hex` for Hex's own
|
||
│ │ Settings; smartRouting only), the port ladder) is suggestion #1, its
|
||
│ │ reason as the row's word ("dev server", "same page", "mentioned"; none
|
||
│ │ for Jev's pick or fallback), then
|
||
│ │ the runners-up and recent sessions; nothing sent until one is picked.
|
||
│ │ The deterministic guess publishes first. Model refinement has one
|
||
│ │ three-second budget including recent messages and credential lookup;
|
||
│ │ a timeout keeps the first guess. A later result cannot override a pick or Undo.
|
||
│ │ `site` names what a Shift pick (Shift+click, ⇧⏎ in the list,
|
||
│ │ or `send --always`) would save, after which that site's captures are Rules;
|
||
│ │ ⇧⏎ in a Rule's list saves the rule for the session picked instead
|
||
│ ├── Each Rule or Guess -> routing.json's `shown`: `as` rule or guess, and
|
||
│ │ `why`, the router's leading reason; `hex capture route` prints it
|
||
│ ├── Nothing / NoAnswer -> recent sessions only
|
||
│ └── Unreachable -> the look read no sessions (OpenCode slow past the look's
|
||
│ deadline, down, or refusing); a warning in process.log with the error, and
|
||
│ the capsule reads "Couldn't reach OpenCode" with why ("OpenCode didn't
|
||
│ answer"), or "OpenCode isn't running" alone, keeping Paste. A reason is
|
||
│ shown only in plain words: one with a path, URL, OS error or code goes
|
||
│ to process.log and the capsule shows none
|
||
│ └── OpenCode refusing Hex's password (HTTP 401/403) -> "OpenCode needs a
|
||
│ password Hex doesn't have", for the look and for a send, and
|
||
│ process.log says where Hex reads it (service.json under this user's
|
||
│ ~/.local/state/opencode, or OPENCODE_PASSWORD), never the password.
|
||
│ The capture stays saved and copied. Check: capture_host::routing
|
||
│ a_look_that_read_no_sessions_answers_unreachable,
|
||
│ a_failed_send_says_what_went_wrong. On Windows with Labs ›
|
||
│ OpenCode in WSL on, the capture router reaches the WSL service
|
||
│ too (capture.send)
|
||
├── An answer for a capsule already closed (dismissed, or past its hold)
|
||
│ leaves it closed; `hex capture list` and `send` still offer the capture
|
||
├── Starting another capture preserves an older rule's accepted pending send;
|
||
│ its eventual answer does not replace the newest capsule
|
||
├── The capsule's commands act only on `last`, the newest routed capture:
|
||
│ ├── Send{to} -> the Rule's own session, already sent, does nothing; any
|
||
│ │ other pick sends there (taking back a send already elsewhere first) and logs
|
||
│ │ the correction; `always` also saves a rule for its site or app
|
||
│ └── Undo -> a sent one is withdrawn unless the
|
||
│ session acted on it (TooLate); an undo the router fails is TooLate too
|
||
├── Two sends to one place agree -> `offer` names the site or app a rule could
|
||
│ send straight there
|
||
├── Queued sends -> retried every 15 s, unless Scrawl.app runs or something
|
||
│ serves the ingest port (that process retries); a retried send can't be
|
||
│ undone, so its "Sent to X" has no Undo. A queued or too-late-to-undo send
|
||
│ leaves after the brief hold (4 s) rather than the 25 s done hold; a taken
|
||
│ back capture with nothing to list ('Taken back', a rule still resolving)
|
||
│ does too
|
||
└── autoSend off -> no look, no route; the capture is only copied
|
||
|
||
hex capture list|show|send|route // capture.cli; docs/cli.md
|
||
├── list [--limit N] -> the newest bundles in the captures folder: mode, words,
|
||
│ where it went; words count a model's own word times, else the words in the
|
||
│ transcript's text
|
||
├── show <capture> -> one capture's words, files, and send; a capture whose
|
||
│ audio stopped early (the 8 h disk guard, or a failed write) says where
|
||
├── send <capture> [--to <session>] [--always] -> needs a listening Hex
|
||
│ ├── The last routed capture -> as the capsule's pick (no --to: the Rule's
|
||
│ │ session, a Guess's first suggestion, or the recorded destination), so
|
||
│ │ the capsule's Undo follows it; --always also saves the site or app rule
|
||
│ │ for the session it goes to. Also when a newer capture was written first
|
||
│ └── Older -> --to must be a session ID; without it, its recorded
|
||
│ destination; --always is refused
|
||
├── route <capture> [--ask-jev] -> what the router sees and decides; sends nothing
|
||
└── <capture> = bundle name, path, or `latest`; unknown -> `not-found`
|
||
|
||
hex annotate inspect|zoom|frames|skill // capture.agents; labs.annotate_agent_tools
|
||
├── Off by default (`Audience::Everyone`, every OS); a CLI read in the `hex`
|
||
│ process, never over the control socket: Hex needn't run, and nothing
|
||
│ records, sends, or writes a setting. `frames` extracts with `ffmpeg` from
|
||
│ PATH when one is installed (the engine's own choice), else the engine's
|
||
│ decoder. Off -> refused naming `hex labs on annotate-agent-tools`
|
||
├── inspect [<capture>=latest] -> `summary.md` (sketch::summary_markdown)
|
||
│ ├── --marks -> one `agent_context::MarkEntry::line()` per mark, time order,
|
||
│ │ or `No marks`; --marks --json -> the entries as a JSON array
|
||
│ └── --json -> `scrawl.json` pretty-printed as written
|
||
├── zoom [<capture>=latest] (--at <M:SS|secs|phrase> | --mark <n|id> [--frame N])
|
||
│ [--span secs=2] [--frames N=12] [--full] -> `zoom::zoom` into a sheet at
|
||
│ `<bundle>/zoom/zoom-<from>-<to>.png`; prints the sheet's path,
|
||
│ `0:09.500 → 0:11.500, 12 frames: #1 0:09.500, …`, and `Said: “…”` when
|
||
│ the window has words. A mark prints `find_mark`'s line first, and its
|
||
│ `--frame` picks the keyframe whose ✎① the number counts
|
||
├── frames [<capture>=latest] --from <M:SS|secs> --to <M:SS|secs> [--fps 4] ->
|
||
│ `handoff::extract_frames_range`'s PNGs under `<bundle>/zoom/range-*/`,
|
||
│ each path printed. `--to` is clamped to the bundle's `durationMs`, and a
|
||
│ range over 240 frames is refused `invalid`, counted as the larger of
|
||
│ `(to − from) * fps / 1000 + 1` and the engine decoder's inclusive
|
||
│ `(to − from) / max(1000 / fps, 1) + 1`; without a video, or over an empty
|
||
│ range, it prints the keyframes in `frames/` that fall in the range instead
|
||
├── skill -> `include_str!` of `skills/hex-annotate/SKILL.md`, the guide an
|
||
│ agent installs with `npx skills add anomalyco/hex --skill hex-annotate`
|
||
├── Every command resolves `<capture>` as `hex capture list` does (bundle name,
|
||
│ path, or `latest`, in `capture.folder`); an unknown bundle -> `not-found`.
|
||
│ `zoom` and `frames` write only into `<bundle>/zoom/`; nothing else in the
|
||
│ bundle changes
|
||
└── Proof limits: unit tests cover the mark lines, both JSON shapes, a sheet's
|
||
printed lines, and every refusal; decoding a bundle's video into a sheet or
|
||
frames is the engine's own `zoom`/`movie_frames_at` path, unverified on a
|
||
real bundle here, and no capture was recorded for this change
|
||
|
||
hex capture cancel -> engine cancelled, sink closed, bundle removed
|
||
hex capture status -> recording (bundle, elapsed), bundles still writing, last outcome
|
||
(words, audio gaps, or why it has no transcript, and where
|
||
its audio stopped if it did)
|
||
Hex quits mid-capture -> cancelled, bundle removed; captures still writing stop
|
||
transcribing between chunks and keep their audio (shutdown waits ≤ 2 s)
|
||
|
||
15 s replay buffer (Labs, macOS 27+) // capture.replay; labs.annotate_replay
|
||
├── Off by default (`Audience::Everyone`); shown and runnable only when macOS 27's
|
||
│ `SCClipBufferingOutput` and `SCStream`'s clip-buffering selectors exist at
|
||
│ run time (weak Objective-C lookup; older macOS and Linux/Windows neither
|
||
│ link nor offer it). Locked off by `DisableAnnotate` and `DisableLabs`
|
||
├── Off -> no `SCStream`, no worker thread, no display observer, no consent prompt
|
||
├── On (with Annotate on and Scrawl.app not running) -> one worker thread waits
|
||
│ until Screen Recording is already granted (never prompts in the background),
|
||
│ then runs one 15 fps `SCStream` with `SCClipBufferingOutput` on the pointer's
|
||
│ display (≤ 2560 px long edge, cursor hidden, audio off, Hex's `OWN_WINDOWS`
|
||
│ and non-layer-0 overlays excluded unless Show in screen recordings is on)
|
||
│ ├── `replayd` encodes the rolling 15 s H.264 ring out-of-process; macOS keeps
|
||
│ │ the menu-bar screen-recording indicator lit while the stream is active
|
||
│ └── Display plug/unplug (`CGDisplayRegisterReconfigurationCallback`) or a
|
||
│ stopped stream -> reopens on the active display with bounded backoff
|
||
├── `hex capture replay` (`hex annotate replay`) -> exports up to 15 s (`> 15 s`
|
||
│ is rejected by ScreenCaptureKit `-3812`) into `<captures>/<unix ms>.scrawl`
|
||
│ without stopping the buffer, probes and decodes keyframes with `scrawl-helper`,
|
||
│ writes `capture.json` and `scrawl.json` (`missing: "no_audio"`), copies to
|
||
│ the clipboard, and feeds `capture.route`
|
||
└── `capture.replay_hotkey` (unbound by default; recordable in
|
||
Settings › Annotate while the lab is on, or `hex settings set capture.replay_hotkey`)
|
||
├── Tap (< 300 ms) -> exports the silent 15 s clip at once, no mic opened
|
||
└── Hold (≥ 300 ms) -> exports the frozen 15 s clip at the press and records
|
||
Hex's microphone timeline with "While dictating" until release + tail;
|
||
narration is shifted onto the bundle clock after the clip (`video_ms..`),
|
||
transcribed with word timestamps, and anchored onto the final keyframe
|
||
```
|
||
|
||
`scrawl.json` is Scrawl's bundle manifest. Its `transcript.words` are
|
||
`{t0, t1, text}` in milliseconds from the capture's start (`t0_boot_ns`, the
|
||
boot-time clock audio and shortcuts share), and `audio.wav` starts at that
|
||
start too, padded with silence when the microphone began late or dropped
|
||
audio.
|
||
|
||
`hex dev capture-replay <wav>…` (debug, with the feature) runs the same
|
||
capture host on a fresh transcription engine with the selected model, feeding
|
||
WAV files instead of the microphone, and prints the bundle path. It needs
|
||
Annotate on too, and records the screen meanwhile.
|
||
|
||
Sources: [capture_host.rs](../../src/capture_host.rs), the tap in
|
||
[suppression.rs](../../src/suppression.rs), the shortcut and Escape in
|
||
[recognition.rs](../../src/recognition.rs), `CaptureSettings` and the import in
|
||
[app_settings.rs](../../src/app_settings.rs), the pane in
|
||
[rune_ui/scrawl.rs](../../src/rune_ui/scrawl.rs), the input rules in
|
||
[hosted_input.rs](../../crates/capture/src/hosted_input.rs), the sink in
|
||
[dictation/audio.rs](../../src/dictation/audio.rs),
|
||
`transcribe_with_words` in [parakeet.rs](../../src/parakeet.rs), routing in
|
||
[capture_host/routing.rs](../../src/capture_host/routing.rs), the CLI's reads in
|
||
[capture_host/library.rs](../../src/capture_host/library.rs), the 15 s replay
|
||
buffer in [capture_host/replay.rs](../../src/capture_host/replay.rs) and
|
||
[capture_host/replay/sck.rs](../../src/capture_host/replay/sck.rs), the phone finish
|
||
in [phone.rs](../../crates/capture/src/phone.rs), and the `scrawl-capture`
|
||
control commands in [control.rs](../../src/control.rs).
|
||
|
||
Checks:
|
||
|
||
- Gate: `the_host_is_absent_while_the_experiment_is_off` in capture_host.rs;
|
||
[test-app-identity.sh](../../scripts/test-app-identity.sh) for the
|
||
release-path refusals.
|
||
- Sink, in dictation/audio.rs: `capture_sink_shares_the_timeline_with_dictation`,
|
||
`a_full_capture_sink_drops_audio_as_a_gap_and_never_stalls_dictation`,
|
||
`capture_sink_records_microphone_gaps`,
|
||
`capture_sink_holds_a_released_microphone_open_until_it_closes`,
|
||
`capture_sink_keeps_every_frame_through_timestamp_jitter`, and
|
||
`a_dropped_capture_receiver_closes_the_sink`.
|
||
- Words, in parakeet.rs: `chunked_word_transcription_keeps_word_times_across_chunks`
|
||
and `word_transcription_is_a_foreground_request_on_the_shared_engine`;
|
||
`foreground_work_waits_for_an_open_stream_instead_of_interrupting_it` in
|
||
[transcription/engine.rs](../../src/transcription/engine.rs);
|
||
`a_transcript_without_word_times_counts_the_words_it_says` in
|
||
[transcribe.rs](../../crates/capture/src/transcribe.rs).
|
||
- Host, in capture_host.rs (a fake engine that writes through Scrawl's real
|
||
`finalize_capture_bundle_with_flags`, without a screen):
|
||
`controls_are_idempotent_and_stop_writes_a_scrawl_bundle`,
|
||
`an_engine_that_cannot_start_leaves_nothing_behind`,
|
||
`a_screen_source_that_gives_up_ends_the_capture`,
|
||
`scrawls_engine_claims_nothing_machine_wide`,
|
||
`the_timeline_resamples_pads_gaps_and_ends_at_the_stop`,
|
||
`a_pending_gap_and_a_stalled_tail_come_back_from_the_closed_sink`,
|
||
`a_gap_is_digital_silence_where_it_happened`,
|
||
`a_capture_longer_than_twenty_minutes_keeps_all_of_its_audio`,
|
||
`past_the_guard_the_file_stops_and_the_rest_is_one_gap`,
|
||
`the_guard_records_where_the_audio_stopped`,
|
||
`a_write_failure_stops_the_audio_where_the_file_stopped`,
|
||
`a_mid_capture_write_failure_stops_where_the_file_stopped`,
|
||
`captures_finish_one_at_a_time_and_starts_wait_for_room`,
|
||
`a_capture_queued_behind_a_slow_finish_has_already_stopped`,
|
||
`a_failed_start_raced_by_stop_never_wedges_later_starts`,
|
||
`the_engine_config_comes_from_hex`, and
|
||
`shutdown_cancels_the_recording_capture_and_waits_for_its_cleanup`.
|
||
In capture_host/wav_file.rs: `a_finished_file_reads_back_and_can_be_trimmed`,
|
||
`a_keep_past_the_written_samples_keeps_them_all`, and
|
||
`the_file_is_owner_only`.
|
||
In capture_host/words.rs: `pieces_past_the_first_window_keep_their_place_on_the_clock`,
|
||
`an_untimed_piece_spreads_over_its_own_audio_only`, and
|
||
`a_model_without_word_times_spreads_its_text_over_the_clip`;
|
||
`an_untimed_capture_gets_its_words_spread_over_its_length` in
|
||
[rune_ui/captures.rs](../../src/rune_ui/captures.rs).
|
||
- The audio-stopped notice: `summary_says_where_a_captures_audio_stopped` in
|
||
[sketch.rs](../../crates/capture/src/sketch.rs);
|
||
`lengths_read_as_clocks_at_an_hour` and
|
||
`old_bundles_load_without_app_context_and_new_ones_round_trip` (which proves
|
||
`audioStoppedAtMs` is absent until it happens) in
|
||
[model.rs](../../crates/router/src/model.rs);
|
||
`the_guard_records_where_the_audio_stopped` above.
|
||
- Shortcut and input, in capture_host.rs: `the_host_is_absent_while_captures_are_off`,
|
||
`the_shortcut_starts_and_a_later_press_stops`,
|
||
`input_reaches_only_the_capture_that_is_recording`, `tap_flags_become_modifiers`,
|
||
`the_shortcut_records_once_its_keys_come_up`,
|
||
`a_failed_start_ends_before_escape_or_the_shortcut_sees_it`;
|
||
`the_capture_shortcut_is_suppressed_and_never_chord_evidence` and
|
||
`a_capture_logs_only_keys_that_reach_the_app` in suppression/tests.rs;
|
||
`only_the_capsules_pill_and_not_its_panel_takes_clicks` in capture_host.rs;
|
||
`a_paste_goes_to_an_external_app_never_the_host` in capture/macos.rs;
|
||
`the_hosts_normal_windows_count_and_its_chrome_does_not` in
|
||
[window_log.rs](../../crates/capture/src/window_log.rs) and
|
||
`helper_windows_are_scrawl_s_except_the_capture_source` in
|
||
[capture.rs](../../crates/capture/src/capture.rs) for Hex's own windows;
|
||
the ink and click rules in hosted_input.rs, including
|
||
`a_capture_started_by_its_shortcut_waits_for_shift_to_come_up`,
|
||
`a_capture_started_without_shift_held_inks_from_the_first_stroke`, and
|
||
`a_click_with_the_chords_shift_still_held_is_no_pin`, and in
|
||
capture.rs `a_shift_click_is_part_of_the_drawing`.
|
||
- iPhone: `an_iphone_capture_narrates_with_hex_audio_and_finishes_when_its_mirror_ends`
|
||
and `a_mirror_ending_never_stops_a_screen_capture` in capture_host.rs;
|
||
`since_moves_audio_and_gaps_onto_a_later_clock`,
|
||
`audio_streamed_to_disk_counts_its_samples_and_can_be_read_back` (reading a
|
||
streamed recording consumes its partial), and
|
||
`a_streamed_recording_whose_file_is_gone_has_no_samples` in
|
||
[audio.rs](../../crates/capture/src/audio.rs). The mirror, its
|
||
`--no-microphone`, and the camera prompt have not run in Hex.
|
||
- Streaming and recovery, in [recover.rs](../../crates/capture/src/recover.rs):
|
||
`a_capture_that_has_not_been_journaled_is_never_owed_a_rebuild` (and the
|
||
live case
|
||
of `recover_picks_up_old_video_and_audio_without_a_journal_but_never_a_live_capture`);
|
||
`a_streamed_audio_takes_its_final_name_only_when_its_build_settles_it` in
|
||
[pipeline.rs](../../crates/capture/src/pipeline.rs);
|
||
`a_journaled_capture_adopts_its_streamed_audio` in
|
||
[recover.rs](../../crates/capture/src/recover.rs) (a crash between the
|
||
journal and `audio.wav`);
|
||
`a_crashed_capture_with_only_partial_audio_is_cleaned_up` in
|
||
[retention.rs](../../crates/capture/src/retention.rs).
|
||
- Routing, in [routing.rs](../../src/capture_host/routing.rs):
|
||
`only_a_saved_rule_sends_on_its_own`,
|
||
`page_metadata_is_only_a_suggestion_and_never_overrides_a_saved_rule`
|
||
(early and late page reads, a fake recipient, the send deadline advanced),
|
||
`a_rule_sends_and_lists_the_rest`,
|
||
`a_guess_sends_nothing_and_a_failure_still_lists_recent_sessions`, and
|
||
its `looked` tests, which write a bundle and route it through `look` and
|
||
`Worker::written` with fake sessions and tab:
|
||
`a_saved_site_or_app_rule_sends_whatever_the_routing_flags`,
|
||
`the_port_ladder_is_a_guess`, `the_captured_apps_checkout_is_a_guess`,
|
||
`a_look_past_its_deadline_is_routed_without`,
|
||
`an_always_rule_names_the_session_it_sends_to`,
|
||
`an_always_rule_keeps_sending_to_the_session_picked`, and
|
||
`a_capture_routed_after_a_newer_one_settled_is_still_sendable_by_name`;
|
||
`an_always_rule_makes_its_sites_next_capture_a_saved_rule`,
|
||
`a_failing_page_keeps_what_was_read_and_still_reads_what_rules_name`,
|
||
`an_app_rules_folder_is_read_with_the_sessions`, and
|
||
`a_look_past_its_deadline_skips_what_it_hadnt_begun`
|
||
in [look.rs](../../crates/router/src/routing/look.rs); the CLI's
|
||
`a_capture_on_disk_summarizes_its_mode_words_and_send` and
|
||
`a_capture_without_word_times_counts_the_words_it_says` in
|
||
[library.rs](../../src/capture_host/library.rs), the protocol round trips in
|
||
control.rs, and `control_commands_parse` in main.rs.
|
||
- Switching, in capture_host.rs:
|
||
`a_discarded_capture_leaves_no_trace_and_a_pick_takes_its_place`,
|
||
`a_switch_never_shows_nothing_recording`,
|
||
`a_switch_with_no_room_to_start_keeps_the_recording`,
|
||
`a_switch_that_cannot_start_after_its_discard_settles_failed`,
|
||
`discarded_captures_never_count_against_the_writers`,
|
||
`a_pick_that_cannot_start_says_why`,
|
||
`the_first_click_or_scroll_means_work_started`, and
|
||
`the_newest_grab_takes_only_its_keys_until_it_drops`. Paste and Inspect
|
||
need a written capture. `paste_only_hands_off_the_shown_written_annotation_once`
|
||
and `clicks_on_the_capsule_do_what_they_say` distinguish local Paste from
|
||
picking a row by pointer or Enter in search. They record actions, not native
|
||
clipboard, focus or insertion in a receiving application.
|
||
- Capsule, in rune_ui/capture_capsule (feature builds only): the flow tests
|
||
(`chips_fold_once_seen_when_work_starts_but_never_under_the_pointer`,
|
||
`the_shortcut_records_the_pickers_screen_and_a_chip_discards_the_recording`,
|
||
…), feed tests
|
||
(`the_shortcuts_recording_shows_no_target_chips`,
|
||
`the_pen_lights_while_drawing`,
|
||
`a_rule_shows_once_and_only_news_the_capsule_cant_know_moves_it`,
|
||
`a_queued_send_says_so_and_says_sent_when_it_goes`,
|
||
`a_cabled_phone_shows_its_chip`, …),
|
||
`paste_takes_back_a_send_still_undoable_and_the_capsule_stays_gone`,
|
||
`a_rule_reads_sent_at_once_and_sounds_when_it_went`,
|
||
`undoing_a_rule_send_reopens_the_list_until_a_pick`,
|
||
`undoing_a_pick_reopens_the_list_it_came_from`,
|
||
`a_pick_during_an_undo_stands_when_the_undo_answers`,
|
||
`a_send_undone_while_under_way_keeps_the_list_however_it_ends`,
|
||
`the_undo_window_wakes_the_capsule_once_at_its_end`,
|
||
`every_rule_send_sounds_sent_once`,
|
||
`picking_the_rules_own_session_just_closes_the_list`,
|
||
`too_late_to_take_back_offers_the_list_for_a_copy`,
|
||
`shift_return_in_a_rules_list_saves_a_rule_elsewhere`,
|
||
`shift_shows_the_rule_and_a_shift_pick_saves_it`,
|
||
`an_answer_after_the_capsule_closed_leaves_it_closed`,
|
||
`the_capsule_takes_only_the_keys_its_state_needs`,
|
||
`shortcuts_read_as_one_quiet_word`, `the_capsule_draws_no_keycaps`,
|
||
`a_written_copy_offers_paste_and_inspect_as_icons`,
|
||
`every_change_moves_at_once`,
|
||
`entry_lands_the_pose_instantly_then_springs_to_rest_with_its_content`,
|
||
`the_toast_cycles_its_guesses_and_return_sends_the_one_shown`,
|
||
`the_menu_bar_hint_is_brief_and_its_chips_send`,
|
||
`the_cursor_card_lists_its_guesses_at_the_pointer`,
|
||
`the_notch_hangs_from_the_top_edge`,
|
||
`the_default_notch_rests_where_it_always_has`,
|
||
`the_notch_shell_fits_a_measured_notch`,
|
||
`notch_content_centres_on_the_notch`,
|
||
`a_wide_recording_row_stays_inside_the_notch_shell`,
|
||
`a_measured_notch_refreshes_the_entry_and_the_rest`,
|
||
`a_measured_notch_never_tucks_under_reduce_motion`,
|
||
`the_menu_bar_flow_folds_up_toward_the_menu_bar`, goldens
|
||
`tests/golden/capture-capsule-*.png`, and
|
||
`the_capsule_moves_inside_its_panel_and_comes_to_rest`, which traces every
|
||
scripted state change per frame, Calm and Reduce Motion, for no pops,
|
||
jumps, or jiggle. `HEX_CAPSULE_MOTION` also writes rune-motion traces for
|
||
motion-lint. `cues_mark_start_stop_sent_and_taking_back`,
|
||
`nothing_heard_says_so`, and `a_capture_with_no_words_says_nothing_was_heard`
|
||
cover the tones and the empty transcript. Observed (Oct 27, worker, r19 motion): all 28
|
||
capsule runs, Calm and Reduce Motion, pass lint. In
|
||
[notch.rs](../../src/rune_ui/capture_capsule/notch.rs):
|
||
`the_default_geometry_is_the_pretend_notch`,
|
||
`a_measured_notch_moves_the_panel_with_its_edges`,
|
||
`a_two_x_notch_lands_on_whole_pixels`, and
|
||
`edges_between_pixels_snap_outward_or_inward`. Flat displays go through
|
||
the same measured path with the pretend notch, so their panel is
|
||
pixel-snapped within a point of the display's middle, not exactly on it.
|
||
Which poll re-measures and re-places is decided by `move_of` in
|
||
[mod.rs](../../src/rune_ui/capture_capsule/mod.rs), tested by
|
||
`a_showing_capsule_that_is_not_following_stays_put`,
|
||
`a_new_capture_places_the_capsule_on_the_pointers_display`,
|
||
`an_anchor_change_places_the_capsule_again`,
|
||
`a_capsule_coming_back_from_hidden_places_on_the_pointers_display`, and
|
||
`following_places_only_for_a_changed_notch`: a showing capsule that isn't
|
||
following never moves because the pointer's notch changed, a capsule coming
|
||
back from hidden is placed before it shows rather than on the display it
|
||
last hid on, and following moves it only for a changed notch. Whether a
|
||
poll looks at the pointer's screen on its own (an open picker over a
|
||
pinned capsule with an overlay) is `follows_display`, tested by
|
||
`only_an_open_picker_over_a_pinned_capsule_looks_again`. The native
|
||
read, the re-place, and the 100 ms wake itself have no test: they need two
|
||
real displays.
|
||
- Playground: `hex capsule-playground` (feature builds; hidden), or
|
||
`scripts/build-capsule-playground.sh` for an ad-hoc signed
|
||
`Hex Playground.app`, runs the real capsule in a window of its own against a
|
||
fake host, with state buttons, the whole flow, slow motion, Reduce Motion,
|
||
the tones, and live tuning ([tuning.rs](../../src/rune_ui/capture_capsule/tuning.rs);
|
||
"Copy values" copies its `DEFAULT`). It reads and writes no Hex data and
|
||
records or sends nothing. The whole flow picks the first suggestion;
|
||
its "Saved rule" toggle plays a rule's send, sent at once, instead.
|
||
`--snap --scene suggestions|shift|sent-rule|search|typed|picked|rule|recording`
|
||
renders it. The tones were checked by waveform and level only, never on
|
||
speakers.
|
||
- Picker, in capture_capsule/picker.rs and tests.rs: geometry, label, and
|
||
target tests, `the_picker_records_what_it_shows_and_comes_to_rest`, and
|
||
`picker_stills_and_motion`, which scripts window hover, an area drag, and
|
||
mode switches per frame (Calm and Reduce Motion). Observed (Oct 26, pro):
|
||
all 6 runs pass motion-lint with two excuses: the target growing from a
|
||
zero-size point, and the camera stopping where the scripted pointer stops.
|
||
Not proven natively end to end: the overlays have been driven only by the
|
||
scripted scenes, not a live capture.
|
||
- Pane: `captures_default_off_and_refuse_a_shortcut_in_use` and
|
||
`retention_names_presets_and_imported_days` in rune_ui/scrawl.rs; scenes
|
||
`mac-scrawl-off`, `mac-scrawl`, `mac-scrawl-running`, `mac-scrawl-permission`
|
||
(feature builds only, so no goldens: those run on Linux).
|
||
- Settings: every `capture.*` key in `hex settings`
|
||
(`every_persisted_setting_has_a_cli_key`).
|
||
- Replay, in [capture_host/replay.rs](../../src/capture_host/replay.rs) and
|
||
[capture_host/replay/sck.rs](../../src/capture_host/replay/sck.rs):
|
||
`empty_audio_produces_no_wav_and_narrated_audio_shifts_after_video`,
|
||
`capture_manifest_matches_v1_schema_with_and_without_audio`,
|
||
`encoded_dimensions_stay_even_and_capped_at_2560`,
|
||
`replay_shortcut_begins_on_press_finishes_on_release_and_cancels_on_escape`,
|
||
and `error_completion_block_carries_xpc_signature`.
|
||
|
||
Observed 2026-10-25 on pro (M2 Max), before the real engine, with the
|
||
stand-in: `HEX_EXPERIMENTS=capture hex dev capture-replay` over three
|
||
vocabulary fixtures (6.5 s) loaded Parakeet unified English on Metal and
|
||
wrote 18 words with plausible times. The non-replay `hex capture` engine has
|
||
not run end to end in automated tests (unit tests use a fake engine over
|
||
Scrawl's real bundle writer). Observed 2026-11-04 in a macOS 27.0
|
||
(`26A428`) guest VM on Apple Silicon with Developer ID-signed `Hex Test.app`:
|
||
`labs.annotate_replay` opened a 15 fps `1920×1080` `SCClipBufferingOutput`
|
||
stream, exported 15,000 ms silent and 19,581 ms narrated bundles (13 Parakeet
|
||
words anchored on the final keyframe, cursor hidden, Hex chrome excluded,
|
||
clipboard handoff populated) via `hex capture replay`, an `⌥⇧R` tap, and a
|
||
3.5 s `⌥⇧R` hold, recovered across `CGVirtualDisplay` plug/unplug and
|
||
display sleep/wake, and rejected a 16 s request with
|
||
`SCStreamErrorDomain -3812`. The menu-bar screen-recording indicator stays lit
|
||
for the entire lifetime of the rolling buffer.
|
||
# Paste Last provenance on macOS
|
||
|
||
A successful Annotate finish marks its own bundle with `.hex-annotate-v1`
|
||
(owner-only identity and ready time, no words or images). Failed, cancelled,
|
||
and Screenshots recordings do not publish this mark. Existing ambiguous
|
||
bundles, including captures made by standalone Scrawl, are not backfilled
|
||
from a folder name. They remain available through their existing actions.
|
||
The [Mac picker](dictation.md#paste-last-picker-labs-macos) (Labs › Paste Last
|
||
picker) reads the mark; failure to write it does not fail the capture or affect
|
||
its send.
|