hex/ios/HEXMobileTests
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Ryan Vogel ef133b8a4f
iOS keyboard: smoother, calmer motion for the live dictation preview (#2424)
* iOS keyboard: the live preview's motion, smooth and calm

What: one spring family (duration and bounce 0, UIKit's springDuration curve) for every
change while dictating, all of it drawn by one view so no memory comes back:
- Keys out, words in as recording starts: the keys fade (.2) as the words fade in rising
  6 pt (.3), the same frame. Words out, keys back as the dictation goes in: the words fade
  drifting 6 pt up toward the field (.15) as the keys fade in (.3); the words stop taking
  touches at once, so the keys work the moment the dictation ends.
- Words arriving (LiveWordsView): each new word fades in rising 4 pt (.3), a burst's words
  40 ms apart and all of them within 240 ms, so the last is in long before Apple's next
  burst (about once a second). "Listening…" fades out a beat (60 ms) ahead of the first words.
- Tentative to settled: the muted tone eases to the keys' ink (.3).
- A new line: the block moves up by a critically damped spring (.3) that carries on from
  where it is when another line comes, and the line leaving the top fades through a 14 pt band
  instead of being cut.
- A changed word ("Okay" to "Hey", "730." to "7.30."): the old word fades out where it was
  (.15) as the new one fades in (.2); words keep their place by position, so the words after
  it never re-fade, and a word that only gains a mark stays as it is.
- The recording bar's morph has no bounce any more (was damping .82) and its fades are the
  same spring (.2), so nothing bounces.
- Reduce Motion: fades only (.2), nothing rises, drifts or slides.
LiveWordsView draws the words with one TextKit 1 layout into one backing store the size of the
box, from a display link that runs only while something moves, and drops whole lines well out
of view so the lines that stay never re-break. It replaces the preview's label.

Why: Ryan asked for smoother motion with better easing after watching the phone recordings
(keys and words swapped in 180 ms with a double exposure; a burst's words popped in at once).

Verified: LiveDictationTests (10) on the Studio simulator; the Studio simulator with the 60 s
replay: footprint +2.7 MB at most while recording (+5.7 with #2410's label), 1,868 frames
drawn while moving, none late; frames of a recording checked for the start, a burst, a new
line and the end. code-shape.py passes.

* iOS keyboard: the preview's words move by Core Animation, not a display link

What: LiveWordsView no longer redraws every frame. It lays the words out and draws them once
per update, in one layer, and Core Animation does every frame of the motion: what changed
cross-fades in place (0.3 s, the cubic closest to a no-bounce spring: new words, a settled
tail turning to ink, a word the recogniser changed), the block moves up by whole lines with a
no-bounce spring (0.3 s) inside a box whose 14 pt band fades the line leaving the top, and once
at rest that line fades out of the band and the lines above the box are dropped from the layout
with nothing on screen changing. The per-word stagger and rise are gone.

Why: on the iPhone 17 Pro the display link drawing the words got only a quarter of its frames
on time while recording (intervals 12-20 ms 26%, 20-30 ms 23%, 30-45 ms 27%, longer 24%; the
drawing itself took 0.3 ms): the main thread is shared with the capture, the recogniser and the
Dot Scope. Core Animation's own animations keep the display's pace regardless.

Verified: LiveDictationTests (9, including whole-line scrolling and dropping the lines above the
box) and recordings on the Studio simulator: a burst, settling, a new line with the band's fade,
the drop at rest, the start and the end; 60 s replay +5.9 MB at most against +7.0 MB for the
same recording without the preview; 1.4 ms of main thread per update.

* iOS README: the live preview's motion, as it now runs

What: the live-words bullet in ios/README.md describes the motion as it ships: one family (a
no-bounce spring by its duration, and its closest cubic for the cross-fade), Core Animation
running every frame, what changes cross-fading in place while the rest stays exactly as it is,
whole-line scrolling through a 14 pt band, and Reduce Motion as cross-fades only. Replaces the
paragraph about per-word staggers and a display link, which the previous commit removed.

Why: the README is where the keyboard's behaviour is written down for the next person.

Verified on the iPhone 17 Pro with 9869899c installed, warm (Parakeet compiled before measuring):
- 60 s dictation, preview on: 59.1 MB at the end, 62.9 MB peak while recording, 76.3 MB at
  transcription (peak 77.3); #2410 was 60.1 / 66.5 / 78.7. Preview off: 58.3 / 60.0 / 75.3.
- The preview's main-thread share: 1.69 ms per update on average, 3.40 ms at most (242 updates).
- In the 30 fps phone recordings, words that didn't change hold the same brightness to the
  decimal through every cross-fade near them; a changing line eases over about 0.2 s with no
  stall; the field's text appears on the frame the preview's words go.

Correction to 9869899c's message: its "+7.0 MB for the same recording without the preview" on
the simulator wasn't measured (the simulator's memory script only runs with the preview); the
phone numbers above are the comparison.
2026-11-22 14:49:09 -05:00
..
Fixtures iOS: emoji panel with on-device suggestions (Experiments › Emoji) (#2051) 2026-11-20 15:53:24 +00:00
AppleSpeechAssetReservationsTests.swift iOS: speech model preparation that never answers fails with Try again (#2372) 2026-11-22 12:31:22 -05:00
AutocorrectEngineTests.swift iOS keyboard: French reads Hex's own word list and capitals, not the phone's dictionary (#2404) 2026-11-22 13:34:40 -05:00
CleanupTaggerTests.swift iOS dictation: spoken numbers written in digits for every engine (Numbers as digits) (#2414) 2026-11-22 13:55:16 -05:00
CustomVocabularyTests.swift iOS: custom vocabulary writes a name heard with another spelling the listed way (#2416) 2026-11-22 14:10:01 -05:00
DictationControllerTests.swift iOS keyboard: a long dictation goes to Apple's engine instead of Parakeet, which holds it all in memory (#2412) 2026-11-22 13:47:38 -05:00
DictationLanguageTests.swift iOS: dictate Latin American Spanish with Latin America's model, and keep a picked variant when the keyboard switches (#2193) 2026-11-21 09:53:34 -05:00
DictationLearningTests.swift iOS keyboard: smoother, calmer motion for the live dictation preview (#2424) 2026-11-22 14:49:09 -05:00
DotScopeTests.swift fix(ios): eliminate handoff frame stalls and add overlapping ballistic swipe throw (t1153) (#757) 2026-10-31 22:14:22 -07:00
EmojiPanelLettersKeyTests.swift iOS dictation: no spaces or stray full stops around Chinese and Japanese; the bar's emoji panel says Back (#2347) 2026-11-22 03:47:53 -05:00
EmojiSuggesterTests.swift iOS keyboard: stop leaking views, capture web views, and emoji indices across shows (#2352) 2026-11-22 06:45:38 -05:00
FullKeyboardEngineTests.swift iOS keyboard: Polish autocorrects with Hex's own model, word list and capitals (#2423) 2026-11-22 14:45:55 -05:00
GreeklishRomanizerTests.swift iOS: add a Greeklish experiment for Greek dictation (#2366) 2026-11-22 08:51:48 -05:00
HangulComposerTests.swift iOS keyboard: Korean two-set (두벌식) layout with Hangul composition (experiment, off by default) (#2339) 2026-11-22 04:39:53 -05:00
HexKeyGestureTests.swift iOS keyboard: Paste alone takes a tap anywhere on the bar (#2386) 2026-11-22 11:36:18 -05:00
KeyboardBridgeTests.swift iOS: tappable Paste from Other Apps notice and app Settings card (#2394) 2026-11-22 11:13:25 -08:00
KeyboardDuoCoverTests.swift ios: smaller key labels on the duo's cover screen (#2346) 2026-11-22 04:35:44 -05:00
KeyboardLayoutTests.swift iOS keyboard: English and German together type on the first one's keys, as Apple's bilingual keyboard (#2402) 2026-11-22 12:55:14 -05:00
KeyboardSwitchGuessTests.swift iOS: the dictation bar can start dictating by itself (Auto-start: never, on switch, always) (#2267) 2026-11-21 17:08:50 -05:00
KeyboardThemeTests.swift iOS keyboard: add a keyboard themes experiment (#2365) 2026-11-22 08:24:11 -05:00
LaggingField.swift iOS keyboard: Korean two-set (두벌식) layout with Hangul composition (experiment, off by default) (#2339) 2026-11-22 04:39:53 -05:00
LanguageLearningTests.swift ios: learn each language separately (experiment) (#2188) 2026-11-21 09:00:58 -05:00
ShiftStateMachineTests.swift ios: add Dvorak punctuation keys experiment (#2373) 2026-11-22 09:31:25 -05:00
SwipeDecoderTests.swift iOS: swipe reads the letters under the path, and the emoji bar scrubs (#2189) 2026-11-21 08:47:45 -05:00
TranscriptCleanupTests.swift iOS dictation: numbers convert whole or not at all; names and the person's words stay (#2420) 2026-11-22 14:21:20 -05:00
TwoLanguageMergeTests.swift iOS: two-language dictation experiment (#2348) 2026-11-22 04:58:27 -05:00
TwoThumbSelectionTests.swift iOS: two-thumb selection on the space bar (experiment, off by default) (#2265) 2026-11-21 20:44:15 -05:00
TypingGateReplayTests.swift iOS keyboard: Polish autocorrects with Hex's own model, word list and capitals (#2423) 2026-11-22 14:45:55 -05:00
TypingLogTests.swift iOS: hold the bar to dictate, Learn how I type on, a plain centred Paste, and typing logs off (#2210) 2026-11-21 12:11:58 -05:00
TypingReplayTests.swift iOS keyboard: the typing replay reports all-words accuracy, contractions and taps read as swipes (#2287) 2026-11-21 20:32:00 -05:00