mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
A TestFlight tester asked for a clipboard manager. Records it with iOS's per-read paste prompt as the main constraint. Docs only.
332 lines
19 KiB
Markdown
332 lines
19 KiB
Markdown
# Roadmap
|
||
|
||
Hex ships the macOS product loop: configurable hold-to-talk local dictation,
|
||
optional context-aware OpenCode post-processing, a Metal HUD, bounded paste
|
||
workers, settings, and a signed app bundle. Meetings (beta) is a
|
||
persisted opt-in (`meetings.enabled`, default off). Windows (x64 and ARM64) and
|
||
x86_64 Linux (i3/X11 and compatible wlroots Wayland compositors) betas provide
|
||
hotkey dictation, Modes, History, the shared Rune window, and signed user-local
|
||
updates; they do not yet include meetings. Nix packaging for
|
||
Linux is implemented and awaiting supported-host validation before release.
|
||
The authenticated local transcription service now implements discovery,
|
||
direct-child embedding, model preparation, and bounded host-audio transcription;
|
||
`@kitlangton/hex@0.3.0` publishes ready transcribers through Promise and Effect
|
||
entrypoints. Local Electron capture and native transcription have been exercised;
|
||
signed helper packaging and a signed first-consumer integration remain. Follow the
|
||
[embedded desktop delivery plan](docs/plans/typescript-sdk.md); Apple Silicon
|
||
macOS is the proposed first target, not a cross-platform availability promise.
|
||
|
||
## Validate The Rune Desktop App
|
||
|
||
The window, menu-bar and tray hosts, HUDs, meeting capsules, and onboarding now
|
||
run only on Rune (GPUI is gone). Linux has been run in a real session; macOS
|
||
has been compiled, tested, and launched without being seen on screen; Windows
|
||
has only been cross-compiled. Before releasing `dev`, a person runs each
|
||
OS through the checklist in [the Rune port plan](docs/plans/rune-ui.md#needs-a-person-to-look), including
|
||
the Windows x64 and ARM64 tray app, the macOS HUD and meeting capsules, and
|
||
shortcut capture on every OS. Restore application icons in the Modes picker
|
||
(removed with GPUI) on Rune's raster images, and add native file dialogs on
|
||
macOS and Windows.
|
||
|
||
## Validate The macOS Release
|
||
|
||
Runtime logs now live in Application Support, first-run setup installs the
|
||
selected dictation model, and the release flow signs, notarizes,
|
||
staples, Gatekeeper-assesses, and publishes Sparkle updates through R2.
|
||
|
||
The quarantined public DMG and a genuine automatic 2.0.0-to-2.0.1 Sparkle update
|
||
have been validated on the development Mac. Before broadly sharing it, validate
|
||
the DMG from a clean macOS account, including every TCC permission, model
|
||
installation, and source-free launch. Preserve the stable bundle identity that
|
||
owns Microphone, Accessibility, and Automation permissions.
|
||
|
||
## Finish Settings And First-Run Health
|
||
|
||
Later, split features from settings into two windows (Kit, 2026-10-24; deferred
|
||
in favour of the one-window feature-owned panes): the Hex window keeps what you
|
||
use, History and Meetings recordings, and a separate Settings window (⌘, and
|
||
the menu bar's Settings…) holds General, Modes,
|
||
Meetings settings, and Experiments.
|
||
|
||
Microphone selection, idle-safe live switching, unavailable-device fallback,
|
||
sound-effect volume, application/browser-host processing modes, and the native
|
||
`SMAppService` login-item toggle now ship. Runtime settings changes project live
|
||
and persist atomically; launch-at-login remains owned by macOS rather than the
|
||
settings file. A persistent menu-bar item keeps Settings, Paste Last, updates,
|
||
and Quit reachable when the Dock icon is hidden. Settings now surfaces revoked
|
||
Microphone or Accessibility access after onboarding, and
|
||
supported transcription runtimes expose automatic language detection. Modes own
|
||
corrections, optional OpenCode rewriting, and ordered text transformations;
|
||
stage failures preserve the preceding pipeline output. Do not add a generalized
|
||
processor framework without a concrete need beyond this shipped sequence.
|
||
|
||
Keep Swift and Rust separate. Prefer the old app's existing Sparkle update window
|
||
with a website-only informational item to install the new app manually, with no
|
||
data import or automatic replacement.
|
||
Follow [`docs/plans/swift-app-handoff.md`](docs/plans/swift-app-handoff.md).
|
||
The new app uses identity `com.kitlangton.hex2`, display name `Hex`, and filename
|
||
`Hex.app`, with data in `hex`. Use Kit's approved
|
||
Developer ID signing team and validate the new identity before release. The legacy
|
||
S3 feed must never install a Rust payload. Publish
|
||
the updated Rust artifact first, then the separately validated legacy feed notice.
|
||
No Swift rebuild should be necessary for the notice.
|
||
|
||
## Validate The Linux Beta
|
||
|
||
Run a genuine signed update between two published versions on the target
|
||
Arch/i3 machine. Verify exact artifact validation, atomic activation, restart,
|
||
and retained-version rollback. Smoke-test installation, model download, the CLI
|
||
microphone override, hotkey rebinding, locked capture, cancellation, automatic
|
||
paste, systemd autostart, launch forwarding (`Activate`), and closing the window
|
||
without stopping accepted work. Linux no longer uses a tray.
|
||
|
||
Keep app-managed updates limited to the user-local direct-install layout. A
|
||
future Arch package must leave updates to the package manager. Nix owns its
|
||
packaged installation and optional Home Manager service. Validate the Wayland
|
||
beta on a compatible compositor with physical portal shortcuts, the `hex`
|
||
commands bound in a desktop without the portal, microphone, foreground paste,
|
||
click-through HUD, and window close leaving the process running while `hex quit`
|
||
drains and exits. See [`docs/plans/linux.md`](docs/plans/linux.md). Add
|
||
context, meetings, or broader Wayland support only as explicit later capability
|
||
slices.
|
||
|
||
macOS, Windows, and Linux now share one window and one `settings.json` (see
|
||
[`docs/plans/shared-desktop-ui.md`](docs/plans/shared-desktop-ui.md)). Remaining
|
||
desktop parity: website mode rules and the focused application on Wayland and
|
||
for Linux browsers.
|
||
|
||
## Harden The Real-Time Boundary
|
||
|
||
Append-only observation serialization and flushing now run on a bounded writer
|
||
instead of the semantic coordinator. Replaceable partial transcript and context
|
||
observations may be dropped under pressure; completed transcripts, state
|
||
transitions, and failures retain bounded backpressure.
|
||
|
||
The authoritative dictation timeline now drains timestamped CoreAudio on its own
|
||
thread. Physically smoke-test delayed
|
||
shortcut boundaries, custom modifier and key chords, lock behavior,
|
||
cancellation, foreground insertion, and both reserved paste shortcuts.
|
||
|
||
The macOS stop tone now sounds at the release, from the audio owner before it
|
||
drains the tail, after a muted output is restored. That change was written
|
||
without a Mac: compile it, run the `recording_environment` and `dictation_audio`
|
||
tests, and hear the tone at key-up with While dictating set to Mute.
|
||
|
||
The first installed timestamped-timeline build regressed physical dictation:
|
||
shortcut input arrived, but long captures discarded and one accepted job reached
|
||
inference with zero milliseconds of audio. At the HID tap, physical timestamps
|
||
arrived as raw Mach ticks; on the development Mac's `125/3` timebase, treating
|
||
them as nanoseconds made a three-second hold appear to last roughly 72 ms.
|
||
The 2.1.7 move to annotated-session taps retained that conversion even though
|
||
timestamps at this boundary are already nanoseconds. This stretched short taps
|
||
by 41.67 times, breaking double-tap lock and admitting accidental short captures.
|
||
A simultaneous passive HID/annotated probe confirmed the different units for
|
||
real modifier events. Annotated-session timestamps now enter the audio timeline
|
||
without scaling; callback-level regressions cover lock, short-tap discard, and
|
||
exact delayed release trimming. Keep physical shortcuts, assistive modifier and
|
||
key chords, non-empty captures, and immediate HUD onset in the release smoke
|
||
checks; changing tap location requires revalidating the clock as well as routing.
|
||
|
||
Version 2.1.11 also recovers blocked modifier gestures after missed releases and
|
||
navigation-key metadata. Recovery requires a neutral keyboard and emits no
|
||
capture actions. Keep the callback regressions for delayed releases and Fn
|
||
evidence alongside physical Option-hold checks; a physical Fn/Globe smoke test
|
||
and general cross-tap delivery-order validation remain outstanding.
|
||
|
||
Settings > Shortcut mode now offers Toggle (press to start, press again to
|
||
finish) beside Hold on macOS, Linux, and Windows; see
|
||
[dictation](docs/features/dictation.md#shortcut-mode). Physically check it with
|
||
modifier-only, key-chord, Fn/Globe, and standalone function-key bindings, and
|
||
with cold-microphone capture, before calling #77 done.
|
||
|
||
Track the remaining shortcut requests explicitly:
|
||
|
||
- [#82](https://github.com/anomalyco/hex/issues/82): menu-bar Pause/Resume ships
|
||
as the `pause-menu` Lab ([labs.md](docs/features/labs.md)), off by default.
|
||
It suspends every capture shortcut and the secure-input Carbon hotkeys, passes
|
||
every key event through, cancels only active capture, keeps accepted output,
|
||
and requires a neutral keyboard before shortcuts go live again. App exclusions
|
||
(pausing only in chosen apps) are a later slice.
|
||
- [#77](https://github.com/anomalyco/hex/issues/77): Toggle mode is implemented;
|
||
it still needs physical chord-discard and cold-microphone checks.
|
||
- [#67](https://github.com/anomalyco/hex/issues/67): standalone keypad/character
|
||
bindings need explicit key-consumption, cancellation, and rebinding semantics.
|
||
- [#37](https://github.com/anomalyco/hex/issues/37): Caps Lock remapped to
|
||
Control is read through a raw HID modifier tracker below macOS's keyboard
|
||
filter, so a Right Control shortcut no longer starts from it while Either
|
||
still does; the shortcut recorder applies the same normalization while the
|
||
listener's input monitor is running. Hex never requests Input Monitoring:
|
||
the tracker runs only when Hex already has it (turn Hex on under System
|
||
Settings › Privacy & Security › Input Monitoring and restart Hex), skips
|
||
itself otherwise, and gives up rather than delay listener startup. Without
|
||
it, Right Control still starts from a remapped Caps Lock, and correctness
|
||
depends on the raw Caps Lock callback arriving before its flags event, which
|
||
is unproven. Needs a physical remap run on hardware. CI covers constructed
|
||
HID snapshots only.
|
||
|
||
On-demand rewrite [PR #63](https://github.com/anomalyco/hex/pull/63) remains under
|
||
review: publish the shared last transcript only after successful paste, require
|
||
explicit shortcut opt-in, prove generation while automatic processing is off at
|
||
the generation boundary, and reuse the isolated native keyboard test harness.
|
||
Do not pre-publish an uncommitted transcript and attempt to repair it afterward.
|
||
|
||
The microphone and dictation model remain warm by default. The shipped
|
||
`Release when idle` microphone option opens the device on shortcut press, removes
|
||
pre-roll, and adds first-capture latency. `Do nothing` is the default
|
||
recording-audio behavior. Idle-sleep prevention is automatic and scoped to
|
||
intentional dictation and active meeting capture rather than persisted as a
|
||
user setting.
|
||
|
||
Add end-of-speech detection only if it preserves explicit Send, Cancel, and
|
||
locked-capture controls.
|
||
|
||
## Make Diagnostics Incremental
|
||
|
||
Ratatui and the Activity pane now share a bounded incremental event reader with session and
|
||
partial-write handling, and observation writes run on a bounded background
|
||
writer. Add heartbeat or process-liveness evidence so a crash cannot leave an
|
||
old `Listening` event looking live.
|
||
|
||
Add an inspectable audit that separates capture failures, recognition errors,
|
||
sleeping or discarded utterances, context mismatches, resolution misses, queue
|
||
pressure, and action failures. Recommendations may propose command-taxonomy
|
||
changes but must not mutate compiled configuration automatically.
|
||
|
||
Authenticated developer control now rides the existing loopback API through the
|
||
debug-only `hex dev` subcommand: it inspects app state, drives semantic HUD
|
||
scenarios, opens/focuses panes, and toggles command mode.
|
||
Extend the same bounded typed channel with fixture-audio capture, microphone
|
||
recovery, model switching, permission snapshots, deterministic screenshots,
|
||
and assertions over capture/job/UI state. MCP may wrap this protocol later; it
|
||
must not become a second control server or settings authority.
|
||
When that wrapper exists, publish it as a standard npm stdio package with a
|
||
`bin` entry so clients can run it through `npx -y`, and publish matching MCP
|
||
Registry metadata. Do not publish an MCP package before the protocol exposes a
|
||
real supported user capability.
|
||
|
||
## Generalize Context From Evidence
|
||
|
||
Foreground application identity now comes from
|
||
`NSWorkspace.frontmostApplication`, and focused window titles come from the
|
||
bounded Accessibility bridge rather than repeated System Events scripts. Keep
|
||
command and dictation-profile semantics expressed in application and browser-host
|
||
terms rather than Brave-specific types.
|
||
|
||
Add one real second browser adapter before extracting a generalized adapter
|
||
framework. Candidate implementations include Chromium-family AppleScript,
|
||
Safari AppleScript, browser extensions, or Accessibility APIs. Diagnose Slack
|
||
Huddle detection separately from generic browser context. Capture failures
|
||
already log a warning and invalidate the cached context. Surface context age and
|
||
failure health in the UI rather than relying only on diagnostics.
|
||
|
||
Keep installed-application discovery lazy. Opening Settings must not recursively
|
||
scan application bundles or rasterize their icons; start that work only after
|
||
the user opens `Add application`, and verify ordinary startup requests no
|
||
Downloads, Calendar, or other unrelated TCC access.
|
||
|
||
## Complete Meeting Lifecycle Recovery
|
||
|
||
Add EventKit correlation for titles and reminders, explicit input-device
|
||
selection, structured capture-gap observations, active-WAV repair, and resumable
|
||
final transcription. Status, final-publication recovery, the call-ended prompt,
|
||
and the menu-bar recording state already exist; do not replace them.
|
||
|
||
Search over titles, notes, and final transcripts, scrolling, and follow-live
|
||
are implemented. Echo suppression is tuned on one real speaker-mode call
|
||
(`hex meeting echo <dir>`); validate it on more rooms, volumes, and
|
||
headphones, and record capture arrival times so offline replays need no
|
||
`--microphone-offset-ms`.
|
||
|
||
Acoustic echo cancellation (Voice Processing IO) was evaluated and deferred:
|
||
it would replace ScreenCaptureKit's raw microphone with an AVAudioEngine or
|
||
`kAudioUnitSubType_VoiceProcessingIO` input, which ducks other audio, switches
|
||
Bluetooth headsets to the low-quality call profile, applies its own gain and
|
||
noise suppression to the transcribed track, runs a second voice processor
|
||
beside the call app's, and drops ScreenCaptureKit's shared capture clock. Revisit
|
||
only if signal-level suppression proves insufficient, with a per-device opt-in.
|
||
|
||
`hex mcp` serves meetings to agents read-only over stdio (full-text BM25
|
||
with stemming and fuzzy matching, transcripts, stored notes). Next: on-device
|
||
semantic search only when full-text misses real paraphrased queries
|
||
(NaturalLanguage embeddings through a Swift bridge, cached under
|
||
`~/Library/Caches/hex/meeting-index`), EventKit attendees in
|
||
`list_meetings`, and a one-shot Granola CSV import
|
||
([granola-import.md](docs/plans/granola-import.md)). Keep it a CLI subcommand;
|
||
the npm-package guidance above applies only to a developer-control wrapper.
|
||
|
||
Give durable transcript entries stable identities so live-to-final reconciliation,
|
||
incremental UI refresh, and meeting-delta paste can advance without rereading
|
||
broad history. Keep `You` and `Them` source labels and the recoverable rolling
|
||
transcript when final transcription fails.
|
||
|
||
Add end-to-end coverage for voice-triggered start/stop, shutdown during capture,
|
||
forced final-transcription failure, and live provider detection. Evaluate
|
||
synchronized playback, echo annotation, summarization, and person-level
|
||
diarization only against real meeting recordings.
|
||
|
||
## Recommend Better Models To Existing Users
|
||
|
||
After the Rune port is fully done (the macOS app has passed its smoke test),
|
||
add a gentle, dismissible "recommended update" hint in the model picker and
|
||
Settings for users whose saved model is no longer recommended for their
|
||
language, explaining how to switch. Language routing already keeps their model
|
||
and lists it in the picker ([plan](docs/plans/language-recommendations.md)); the
|
||
hint never switches or downloads anything itself.
|
||
|
||
## Shared Engine Service
|
||
|
||
Later, not started. Move the engine into a launchd service
|
||
(`anomaly.hex.engine`) serving the same `/streams` and `/transcriptions` API;
|
||
the Hex UI becomes a client. Today the running app serves
|
||
[`/streams`](docs/streams.md) on dictation's loaded model, with dictation
|
||
first; the service keeps that contract, so stream clients such as Seer don't
|
||
change.
|
||
|
||
## Grow Labs › Screenshots
|
||
|
||
The ⌘⇧4 lab (docs/features/labs.md) shoots an area, a window, or a display
|
||
to the clipboard, a folder, and a corner thumbnail. Next, once a person has
|
||
used it on a real Mac:
|
||
|
||
- A markup editor from the thumbnail: arrows, text, boxes, blur, crop.
|
||
- Scrolling capture of a long window.
|
||
- Copy the text in a shot (OCR, Apple's Vision).
|
||
- A self-timer before the shot.
|
||
- Pin a shot to the screen as a floating window.
|
||
|
||
## Grow Hex Mobile (iOS)
|
||
|
||
Hex Mobile is in TestFlight with an on-device dictation keyboard. These are the
|
||
larger asks from testers (X replies and TestFlight feedback, Nov 2026) that
|
||
aren't being built yet. Everything stays on device and commercially safe.
|
||
|
||
- **Keys that learn how you type:** per-user key offsets learned on device from
|
||
confirmed corrections, feeding the touch decoder before visible hit areas
|
||
change; bounded, decaying and resettable. Research is under way; ship as an
|
||
Experiment first.
|
||
- **Chinese input:** Pinyin and Shuangpin layouts with candidate selection.
|
||
- **Themes:** keyboard colour themes (for example a carbon theme), each matched
|
||
in light and dark.
|
||
- **GIFs:** only with a design that keeps the keyboard offline by default,
|
||
such as an opt-in search the user starts.
|
||
- **Choose the speech model:** pick between the local models Hex supports
|
||
(Apple Speech, Parakeet). No cloud models.
|
||
- **Grammar-aware autocorrect:** use the whole phrase, not just the word, beyond
|
||
the its/it's and their/they're pairs.
|
||
- **Clipboard history:** recent copies to paste again from the keyboard, kept on device. iOS asks before each clipboard read, so it can only remember copies the keyboard sees.
|
||
- **More translation:** detect the language you speak instead of picking it, and translate a copied message back into your own language from the keyboard, all on device.
|
||
- **Interface languages:** translate the app into Ukrainian and more, with help
|
||
from testers who offered.
|
||
- **Start dictating on switch:** the Dictation bar can start recording when you
|
||
switch to Hex (built as an Experiment, `ios-dictation-bar-autostart`); needs
|
||
real-iPhone checks before it ships.
|
||
|
||
## Deliberate Deferrals
|
||
|
||
Do not generalize the `~/.config/hex` TypeScript transformation config into a
|
||
plugin framework.
|
||
|
||
Do not generalize the concrete macOS and Linux X11 adapters into a platform
|
||
framework. Add seams only when a second implemented adapter requires one.
|
||
Follow the remaining capability contracts and exit criteria in
|
||
[`docs/plans/linux.md`](docs/plans/linux.md).
|