hex/ios/docs/typing-logs.md
Dax 2d52c720cb
iOS: two-thumb selection on the space bar (experiment, off by default) (#2265)
While the space bar is the trackpad, a second finger can pick a range the
keyboard can't set in the host: each thumb moves one end with its own
TrackpadCursor.State, the strip shows the text with the selected part
highlighted, and both thumbs up offers Delete, Cut, Copy, Dictate and Cancel.
Off, a second finger is ignored exactly as before.

Settings > Experiments > Two-thumb selection, experiments.twoThumbSelection,
off by default.

Idea from @sieversjosua in #2255.

Co-authored-by: Hex <hex@anoma.ly>
2026-11-21 20:44:15 -05:00

310 lines
17 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Typing logs (Record typing logs)
Settings › Privacy › Advanced › **Record typing logs** (off by default, asks for
consent in a dialog before it turns on, App Group key
`experiments.typingLogs`) makes Hex's keyboard write raw logs of everything it
sees: every touch with its position and timing, the key it pressed, what that
did to the text, the text around the cursor, and dictations. They are for
testing and training keyboard prediction and per-user touch models on a Mac.
Nothing is sent anywhere; Privacy › Advanced › Typing logs shares a day's file (or
all of them) through the share sheet, for example by AirDrop.
Once per App Group, the app and the keyboard turn it off at launch for anyone who
had it on (`migrations.typingLogsOff.v1`, `TypingLogSettings.turnOffOnce`). The
logs already recorded stay, and turning it back on afterwards keeps it on.
It is separate from Learn how I type, which keeps only word pairs and per-key
averages and is unchanged.
## Files
- `TypingLogs/yyyy-MM-dd.ndjson` in the App Group container
(`group.anomaly.hex.mobile`), one per local day (Gregorian date in the
iPhone's time zone). Each event goes to the file of the day its `t` falls in.
- NDJSON: one JSON object per line, UTF-8, `\n` after every line.
- Kept 14 days (today and the 13 before). Older files are deleted when the
keyboard starts a logger and when the Typing logs screen opens.
- At most 64 MB a day. The first event that would cross the cap is replaced by
one `cap_reached` line and nothing more is written that day.
- The keyboard writes nothing without Full Access (it can't reach the App Group)
and nothing while the experiment is off. The app's keyboard preview and
simulator fixtures never write logs.
Writing: events are appended to an in-memory buffer on the main thread and
handed to one serial utility queue every 2 s (and when the keyboard goes away,
on a memory warning, and at 1,500 buffered events), which encodes them and
appends them with `FileHandle`. Lines from one session are in order; a session
ends with `session_end` unless the keyboard process was killed first.
## Common fields
Every line starts with these, in this order:
| key | type | meaning |
| --- | --- | --- |
| `v` | int | Schema version, `1`. |
| `t` | int | Wall clock, ms since 1970 (UTC). |
| `mt` | number | Monotonic ms (`CACurrentMediaTime`, time since boot without sleep). Use it for intervals; `UITouch.timestamp` (`ts`) is on the same clock. |
| `type` | string | Event type, below. |
| `session` | string | 12 hex digits, new each time the keyboard appears or moves to another field. |
| `seq` | int | 1, 2, 3… within a session. A gap means lines were lost (only the cap does that). |
Units and spaces:
- Positions and sizes are points in the keyboard view's coordinates: origin at
the top-left of Hex's keyboard view (the suggestion strip's top edge), x to
the right, y down. The strip is `y` 0–44 in portrait, the keys below it.
- A rect is `[x, y, width, height]`. Insets are `[top, left, bottom, right]`.
Points have 2 decimals, ms 3.
- Character counts are Swift `Character`s (grapheme clusters), what one
backspace removes.
- A key id is the character itself for character keys (always one character:
`"a"`, `"7"`, `"@"`; letters lowercase) and a word for the others: `shift`,
`delete`, `space`, `return`, `hex`, `globe`, `layer:numbers`,
`layer:symbols`, `layer:letters`.
## Events
### `session_start`
The keyboard appeared (`trigger`: `appear`), the field changed while it showed
(`field`), or the experiment was turned on while it showed (`enabled`).
- `surface`: `full` (Hex's keys) or `bar` (the dictation bar; no touches).
- `document`: `UITextDocumentProxy.documentIdentifier`, which changes with the
field. The host app is not knowable from a keyboard and is never guessed.
- `traits`: `keyboardType`, `returnKeyType`, `autocapitalization`,
`autocorrection`, `spellChecking`, `smartQuotes`, `smartDashes`,
`smartInsertDelete`, `inlinePrediction`, `keyboardAppearance` (UIKit case
names, e.g. `"emailAddress"`, `"send"`, `"no"`), `textContentType` (its raw
value or null), `secure`, `enablesReturnKeyAutomatically`.
- `bounds` `[w, h]` and `safeArea` of the keyboard's root view when the session
started (may be before layout; `layout` lines carry the real size),
`orientation` (`portrait`/`landscape`), `appearance` (`light`/`dark`).
- `device` (`iPhone17,1`; `Simulator:…` in the simulator), `system`
(`iOS 26.1`), `version` and `build` of Hex, `locale`, `languages` (first
four preferred), `keyboardLanguage` (the keyboard's `primaryLanguage`),
`inputMode` (the field's input mode language), `fullAccess`.
- `settings`: `autocorrect` (Hex's), `spatialAutocorrect`, `key_popups`, `haptics`, `learnHowIType`, `cleanupTagger`,
`hexKeyPlacement`, `dictationLanguage` (BCP 47, or `automatic`), `keyboardLayout`
(`en`/`nb`/`es`/`de`, empty for the dictation bar), `style` (dictation style).
A `context` line follows at once with the text already in the field.
### `skipped`
Instead of `session_start` in a field that must not be logged; nothing else is
written for that session, not even `session_end`. `reason` is
`secureTextEntry` or `textContentType:<raw value>` for `password`,
`newPassword`, `oneTimeCode` and every `creditCard…` content type. Also
`trigger`, `surface`, `keyboardType`, `textContentType`, `secure`. Every other
field is logged, including ones with autocorrection off.
### `session_end`
`reason`: `disappear`, `field_changed`, `disabled`, or `replaced`.
`durationMS`, and counts of `touches` (began), `edits`, `dictations`. A
`context` line (when the text changed since the last one) and any unwritten
`candidates` come just before it.
### `layout`
Written when the key layout or shift changes, and when a session starts.
- `hash`: 16 hex digits over every key's id, frame and hit frame and the strip.
Touches name it in `layout`.
- `cause`: `session`, `field`, `layout` (width, orientation, globe), `geometry`
(view size or safe area only), `layer_key` (123/#+=/ABC pressed),
`layer_auto` (space or return went back to letters), `layer_slide` (hold 123,
slide, let go), `shift_key`, `typed` (a one-shot shift turned off), `auto`
(auto-capitalization), `cancel`, `language` (the keyboard layout changed),
`letters` (its letters' arrangement changed).
- `layer` (`letters`/`numbers`/`symbols`), `shift` (`off`/`on`/`locked`),
`shiftHeld` (only when true), `field` (`text`/`url`/`email`), `orientation`,
`width`, `height` (the keys' layout), `view` `[w, h]`, `safeArea`,
`horizontalInset` (landscape), `globe`, `hexPlacement`, `language`
(`en`/`nb`/`es`/`de`), `letters` (`own`/`qwerty`/`qwertz`/`azerty`/`dvorak`,
Settings › Keyboard › Languages › Layout).
- The first time a hash appears in a day's file the line also has:
- `letterWidth` (a 10-key row's; 11-key Norwegian and German letters are
narrower, see `keys`), `metrics` (`side`, `gap`, `rowGap`, `keyHeight`,
`stripHeight`, `keysBottom`, `cornerRadius`),
- `strip`: the suggestion strip's rect, and `slots`: its three suggestion
slots (left, centre, right),
- `keys`: `[{id, label, row, frame, hitFrame}]`. `frame` is the drawn keycap,
`hitFrame` the touch target; hit frames tile the key area (row 0 to 3) with
no gaps. The Hex key in the strip has row `-1`.
Later lines with the same hash in that file say `"repeat": true` instead, so
look the keys up by hash. Every touch's hash appears in full earlier in the
same file.
### `touch`
One finger sample on the keys (not the strip, whose buttons log `ui`/`edit`).
- `phase`: `began`, `moved`, `ended`, `cancelled`. `id`: the finger, stable
from began to ended.
- `x`, `y`: the raw touch location. `r`: `majorRadius`, `rt`:
`majorRadiusTolerance` (pt). `force` and `maxForce` only on devices with
force. `ts`: `UITouch.timestamp` in ms (monotonic, same clock as `mt`).
- `layout`: the layout hash at the time.
- `key`: the key the engine resolves. For `began`, the key the engine presses:
the key under the point, or a neighbouring letter the next-letter odds gave it
(`FullKeyboardEngine.letterOdds`). For `moved` and `ended`, the key the finger is held
on: sliding across character keys moves it; any other key keeps the key it
went down on. `null` when the finger went down outside every key.
- `under`: the key under the raw point when it differs from `key` (`null`:
outside every key, on `began`/`ended`/`cancelled`). On `began` it appears
only when the next-letter odds moved the press to a neighbouring letter, or
(`null`) when a finger on the strip's lower half while typing went to the row-1
key below it (`FullKeyboardEngine.takesStripTouch`; `y` is then above the
keys); otherwise the engine uses the hit frames as logged, so `key` on
`began` is the key under the point.
- `moved` is throttled: a sample is written once the finger has moved at least
2 pt and 8 ms since the last line for it, or when it slides onto another key.
Order: `began` and `ended` lines come before the edits they cause (a character
types on release; delete and shift act on press; a press commits the
characters of fingers still down, which is rollover). A `moved` line comes
after what the move did (trackpad cursor steps, accent selection).
### `edit`
What the keys did to the text, with `op`. To rebuild the text before the
cursor, for every `edit` (and `dictation`) line remove `deleted` (a string;
`deletedCount` characters when the text wasn't known) from the end, then append
`inserted`. `cursor` moves need the `context` lines to resync.
| `op` | fields |
| --- | --- |
| `insert` | `inserted`, `src`: `key` (a character or space), `accent` (long-press pick), `slide` (hold 123, slide, let go), `bar` (the dictation bar's space) |
| `delete` | `deleted` (the character), `src`: `key` (on press), `repeat` (held, 75 ms), `repeat_fast` (after 0.85 s, two at a time), `bar`, `selection` (Two-thumb selection's Delete), `selection_cut`, `selection_dictate` (its Dictate, when the dictation's words arrive and replace the range; `deletedCount` when the range is longer than 80 characters) |
| `delete_word` | `deleted`, `src`: `repeat_word` (held past 1.4 s) |
| `autocorrect` | `from`, `to`, `deleted` (= `from`), `inserted` (`to` + space). Applied on space. |
| `autocorrect_undo` | `deleted` (the correction, without its space or mark), `inserted` (what was typed), `src`: `strip`. The strip's revert, offered once delete took the correction's space or mark (or the cursor came back to its end), put back what was typed. |
| `suggestion` | `slot` (0 left, 1 centre, 2 right), `text` (as shown), `deleted` (the word being typed), `inserted` (the replacement + space), `candidates` (as in `candidates`) |
| `double_space_period` | `deleted`: `" "`, `inserted`: `". "` |
| `space_swallowed` | `after`: `suggestion`. A space right after a suggestion typed its own changed nothing. |
| `return` | `inserted`: `"\n"`, `returnKeyType`, `send` (the key said Send); `ignored: true` and no `inserted` when the key was disabled (empty field) |
| `paste` | `inserted`: the strip's Paste chip |
| `undo_dictation` | `deleted` (the dictation), or `deletedCount` when the host showed only its end |
| `cursor` | `offset`: UTF-16 units the cursor moved (negative is left); `src`: absent for the spacebar trackpad (including its settle when a selection starts), `selection` (Two-thumb selection's Copy, or its Delete moving to the range's end), `selection_cut` and `selection_dictate` (the same move before their delete) |
### `candidates`
The suggestions the strip offers: `items`: `[{slot, text, replacement?, kind,
auto?}]`, `kind` `literal` (left, what you typed), `primary` (centre; `auto`:
space will apply it), `secondary`. Written only when they changed, lazily: just
before the next touch goes down and at session end, so each line is what was on
screen when a finger next went down. The strip hides them while it shows Undo
dictation, a status, or the clipboard chips.
### `context`
The host's text around the cursor, as `documentContextBeforeInput` /
`AfterInput` give it. `why`: `session_start`, `space`, `return`,
`suggestion`, `delete_word`, `autocorrect_undo`, `revert`, `paste`, `trackpad`,
`selection`, `undo_dictation`, `dictation`, `session_end`, or `text_did_change` (the host
changed the text or moved the cursor: written once things settle for 0.3 s, at
once when the field emptied). `before` (the last 200 characters), `after` (the
first 200), `beforeLength` and `afterLength` (before capping; hosts often cut
the context short themselves), `hasText`. `cleared: true` when the field had
text and now has none (most apps clear it when they send). An unchanged context
is not written again.
### `ui`
Taps that aren't text: `action`:
- `accent_open` (`id`, `key`, `options`), `accent_pick` (`id`, `index`, `chosen`,
`rollover` when another key committed it), `accent_cancel`;
- `trackpad_start` (`id`; `hold: true`, as only holding the space bar starts it;
older logs without it had a 10 pt slide start it), `trackpad_end` (`id`, `offset`: total
UTF-16 units; with Two-thumb selection on, the space bar's own lift writes it even while the
other thumb keeps selecting), followed by a `context` (`trackpad`);
- `selection_start` (`id`, the second finger; `thumb: second`, `anchor`: the cursor's
UTF-16 offset into the trackpad's known text, which is what the selection lays out),
`selection_end` (`id`, `empty: true` when both
thumbs lifted with no range and the selection ended like a trackpad lift; when `false`,
`start` and `end`: the range's UTF-16 offsets from that anchor), `selection_action`
(`action`: `delete`, `cut`, `copy`, `dictate`, `cancel`, `dismiss` (another key closed the
actions, or the text no longer matched: `reason` is `key`, `host_edit`, `text` or
`context`; no `start` or `end`); `start` and `end` as `selection_end` for the others.
A Dictate writes a second `action: dictate` line when its words arrive: `result` is
`replaced` (the range was deleted and the words go in there) or `inserted` (the field
changed, with a `reason`: `field`, `text` or `context`, and the words go in at the
cursor);
- `selection_cancel` (`id`: the system took a finger away, nothing changed);
- `language_swipe` (`id`, `step`: 1 the next language, -1 the one before) and
`keyboard_language` (`language`: the layout it switched to), with two or more
keyboard languages;
- `hex_key` (`dictation`: `warm`, `keyboard`, `open_hex`), `hex_menu`, `style`
(`style`, `from`: `hex_menu` or `recording_panel`),
`dictation_language` (`language`: BCP 47 or `automatic`, from the Hex menu),
`dictation_stop` (`from: hex_key_release` when lifting a push-to-talk hold
ends it), `dictation_cancel`;
- `next_keyboard` (strip chip), `globe`, `undo_dictation` (`applied: false` when
the text had changed), `touches_cancelled` (`count`).
### `dictation`
Hex inserted a dictation (from the full keyboard or the dictation bar).
- `source`: `keyboard` (in-keyboard dictation) or `app` (a warm session or a
handoff to Hex). `resultID`, `style`.
- Each step, when it ran (a step that didn't run is missing): `engine`, `locale`
(in-keyboard) or `language` (the app), `recordedMS`, `transcribeMS`, `raw`
(as heard), `fixed` (after Learn how I type's fixes), `spelled` (after custom
vocabulary, app), `tagged` (after Clean up what I say), `cleaned` (after
Apple Intelligence, app), `translated`, `formatted` (after the style, app).
- `text`: the transcript Hex delivered; `inserted`: what went into the field
(with the spacing `TranscriptFormatter.insertionText` added).
A `context` line follows.
### `dictation_outcome`
What became of a dictation, from `DictationEditTracker` (the same inference
Learn how I type uses, run separately). `dictationSession` and `dictation` (its
`seq`) name the `dictation` line it is about. `outcome`: `kept`, `edited`,
`rewritten`, `undone`, `deleted`, `redictated`. `words`, `changed`, `pairs`:
`[{kind, heard, inserted, meant, before, after}]`. It settles within 60 s, when
the keyboard goes away or moves to another field, or on a Send return key, so it
can be in a later session than its dictation.
### `cap_reached`
The day's file hit 64 MB. `bytes` (file size before this line), `cap`; `seq` and
`session` are those of the first event not written.
## Size and cost
Measured in the simulator (Debug build, `TypingLogTests.testLoggingOverheadAndSize`,
3,000 taps of ordinary words): about 1 KB per tap, about 5.2 lines (a `began`,
one `moved`, an `ended`, an `edit`, a `candidates`, plus a `context` per
word and a `layout` when shift changes). Steady typing at 200 characters a
minute is about 12 MB an hour, so the 64 MB cap is about 5 hours of nonstop
typing in a day. On the main thread logging added about 12 µs per tap (35 µs
without it, 48 µs with); encoding and appending run on the utility queue.
Real fingers may produce more `moved` lines than the test's one per tap.
## Reading a day on a Mac
```python
import json
layouts, text = {}, ""
for line in open("2026-11-19.ndjson"):
e = json.loads(line)
if e["type"] == "layout" and "keys" in e:
layouts[e["hash"]] = e["keys"]
elif e["type"] in ("edit", "dictation"):
if "deleted" in e: text = text[: len(text) - len(e["deleted"])]
elif "deletedCount" in e: text = text[: len(text) - e["deletedCount"]]
text += e.get("inserted", "")
elif e["type"] == "touch" and e["phase"] == "ended":
keys = layouts[e["layout"]] # where the finger lifted, against every key's frame
```
(Python counts code points, not grapheme clusters; use a grapheme library for
emoji and combining marks.)