mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* iOS keyboard: the live preview's motion, smooth and calm
What: one spring family (duration and bounce 0, UIKit's springDuration curve) for every
change while dictating, all of it drawn by one view so no memory comes back:
- Keys out, words in as recording starts: the keys fade (.2) as the words fade in rising
6 pt (.3), the same frame. Words out, keys back as the dictation goes in: the words fade
drifting 6 pt up toward the field (.15) as the keys fade in (.3); the words stop taking
touches at once, so the keys work the moment the dictation ends.
- Words arriving (LiveWordsView): each new word fades in rising 4 pt (.3), a burst's words
40 ms apart and all of them within 240 ms, so the last is in long before Apple's next
burst (about once a second). "Listening…" fades out a beat (60 ms) ahead of the first words.
- Tentative to settled: the muted tone eases to the keys' ink (.3).
- A new line: the block moves up by a critically damped spring (.3) that carries on from
where it is when another line comes, and the line leaving the top fades through a 14 pt band
instead of being cut.
- A changed word ("Okay" to "Hey", "730." to "7.30."): the old word fades out where it was
(.15) as the new one fades in (.2); words keep their place by position, so the words after
it never re-fade, and a word that only gains a mark stays as it is.
- The recording bar's morph has no bounce any more (was damping .82) and its fades are the
same spring (.2), so nothing bounces.
- Reduce Motion: fades only (.2), nothing rises, drifts or slides.
LiveWordsView draws the words with one TextKit 1 layout into one backing store the size of the
box, from a display link that runs only while something moves, and drops whole lines well out
of view so the lines that stay never re-break. It replaces the preview's label.
Why: Ryan asked for smoother motion with better easing after watching the phone recordings
(keys and words swapped in 180 ms with a double exposure; a burst's words popped in at once).
Verified: LiveDictationTests (10) on the Studio simulator; the Studio simulator with the 60 s
replay: footprint +2.7 MB at most while recording (+5.7 with #2410's label), 1,868 frames
drawn while moving, none late; frames of a recording checked for the start, a burst, a new
line and the end. code-shape.py passes.
* iOS keyboard: the preview's words move by Core Animation, not a display link
What: LiveWordsView no longer redraws every frame. It lays the words out and draws them once
per update, in one layer, and Core Animation does every frame of the motion: what changed
cross-fades in place (0.3 s, the cubic closest to a no-bounce spring: new words, a settled
tail turning to ink, a word the recogniser changed), the block moves up by whole lines with a
no-bounce spring (0.3 s) inside a box whose 14 pt band fades the line leaving the top, and once
at rest that line fades out of the band and the lines above the box are dropped from the layout
with nothing on screen changing. The per-word stagger and rise are gone.
Why: on the iPhone 17 Pro the display link drawing the words got only a quarter of its frames
on time while recording (intervals 12-20 ms 26%, 20-30 ms 23%, 30-45 ms 27%, longer 24%; the
drawing itself took 0.3 ms): the main thread is shared with the capture, the recogniser and the
Dot Scope. Core Animation's own animations keep the display's pace regardless.
Verified: LiveDictationTests (9, including whole-line scrolling and dropping the lines above the
box) and recordings on the Studio simulator: a burst, settling, a new line with the band's fade,
the drop at rest, the start and the end; 60 s replay +5.9 MB at most against +7.0 MB for the
same recording without the preview; 1.4 ms of main thread per update.
* iOS README: the live preview's motion, as it now runs
What: the live-words bullet in ios/README.md describes the motion as it ships: one family (a
no-bounce spring by its duration, and its closest cubic for the cross-fade), Core Animation
running every frame, what changes cross-fading in place while the rest stays exactly as it is,
whole-line scrolling through a 14 pt band, and Reduce Motion as cross-fades only. Replaces the
paragraph about per-word staggers and a display link, which the previous commit removed.
Why: the README is where the keyboard's behaviour is written down for the next person.
Verified on the iPhone 17 Pro with 9869899c installed, warm (Parakeet compiled before measuring):
- 60 s dictation, preview on: 59.1 MB at the end, 62.9 MB peak while recording, 76.3 MB at
transcription (peak 77.3); #2410 was 60.1 / 66.5 / 78.7. Preview off: 58.3 / 60.0 / 75.3.
- The preview's main-thread share: 1.69 ms per update on average, 3.40 ms at most (242 updates).
- In the 30 fps phone recordings, words that didn't change hold the same brightness to the
decimal through every cross-fade near them; a changing line eases over about 0.2 s with no
stall; the field's text appears on the frame the preview's words go.
Correction to 9869899c's message: its "+7.0 MB for the same recording without the preview" on
the simulator wasn't measured (the simulator's memory script only runs with the preview); the
phone numbers above are the comparison.
235 lines
10 KiB
Swift
235 lines
10 KiB
Swift
import Combine
|
||
import os
|
||
import UIKit
|
||
|
||
/// The words so far in place of the full keyboard's keys while it records (Settings ›
|
||
/// Dictation › Show what I say): the keys go and the words have their space, the settled
|
||
/// words in the keys' ink and the tentative tail muted, the newest line always in view with
|
||
/// whole lines above it. It takes every touch where the keys were, so nothing types while it
|
||
/// shows; the recording bar above and the globe stay as they are.
|
||
@MainActor
|
||
final class LiveDictationPreviewView: UIView {
|
||
var palette = FullKeyboardPalette(isDark: false) { didSet { if palette != oldValue { render() } } }
|
||
var orientation: KeyboardOrientation = .portrait { didSet { if orientation != oldValue { render() } } }
|
||
/// Where the words sit inside this view: the keys' own edges.
|
||
var textInsets: UIEdgeInsets = .zero { didSet { if textInsets != oldValue { setNeedsLayout() } } }
|
||
/// Where the keyboard has its own globe key (an iPhone with a Home button), its keycap in
|
||
/// this view: drawn whole over the words, since it still switches keyboards.
|
||
var globeFrame: CGRect? {
|
||
didSet {
|
||
guard globeFrame != oldValue else { return }
|
||
globe.isHidden = globeFrame == nil
|
||
globe.frame = globeFrame ?? .zero
|
||
globe.setNeedsDisplay()
|
||
}
|
||
}
|
||
var globeRadius: CGFloat = FullKeyboardPalette.cornerRadius { didSet { globe.cornerRadius = globeRadius } }
|
||
private(set) var transcript = LiveTranscript()
|
||
/// The words, drawn by one view (`LiveWordsView`), which also moves them.
|
||
let words = LiveWordsView()
|
||
private let globe = GlobeKeycapView()
|
||
|
||
static let placeholder = String(localized: "Listening…")
|
||
|
||
override init(frame: CGRect) {
|
||
super.init(frame: frame)
|
||
backgroundColor = .clear
|
||
isOpaque = false
|
||
words.placeholder = Self.placeholder
|
||
addSubview(words)
|
||
globe.isHidden = true
|
||
globe.isUserInteractionEnabled = false
|
||
addSubview(globe)
|
||
// One element for VoiceOver, read when it's focused, never announced as words change.
|
||
isAccessibilityElement = true
|
||
accessibilityTraits = .staticText
|
||
render()
|
||
}
|
||
|
||
@available(*, unavailable)
|
||
required init?(coder: NSCoder) { fatalError() }
|
||
|
||
func show(_ next: LiveTranscript) {
|
||
guard next != transcript else { return }
|
||
transcript = next
|
||
accessibilityLabel = next.isEmpty ? Self.placeholder : next.text
|
||
words.show(text: next.text, settled: next.settled.count)
|
||
}
|
||
|
||
override func layoutSubviews() {
|
||
super.layoutSubviews()
|
||
// The words' box, with the band above it where a line leaving the top fades out.
|
||
let box = bounds.inset(by: textInsets)
|
||
words.frame = CGRect(x: box.minX, y: box.minY - LiveWordsView.edge, width: box.width, height: box.height + LiveWordsView.edge)
|
||
}
|
||
|
||
/// A touch anywhere on it stays with it.
|
||
override func point(inside point: CGPoint, with event: UIEvent?) -> Bool {
|
||
bounds.contains(point)
|
||
}
|
||
|
||
private func render() {
|
||
globe.palette = palette
|
||
globe.isLandscape = orientation == .landscape
|
||
words.palette = palette
|
||
words.font = .systemFont(ofSize: orientation == .landscape ? 19 : 23, weight: .regular)
|
||
accessibilityLabel = transcript.isEmpty ? Self.placeholder : transcript.text
|
||
}
|
||
}
|
||
|
||
/// The globe key as the key plane draws it, for the preview to keep where it was.
|
||
@MainActor
|
||
private final class GlobeKeycapView: UIView {
|
||
var palette = FullKeyboardPalette(isDark: false) { didSet { if palette != oldValue { setNeedsDisplay() } } }
|
||
var isLandscape = false { didSet { if isLandscape != oldValue { setNeedsDisplay() } } }
|
||
var cornerRadius = FullKeyboardPalette.cornerRadius { didSet { if cornerRadius != oldValue { setNeedsDisplay() } } }
|
||
|
||
override init(frame: CGRect) {
|
||
super.init(frame: frame)
|
||
backgroundColor = .clear
|
||
isOpaque = false
|
||
contentMode = .redraw
|
||
}
|
||
|
||
@available(*, unavailable)
|
||
required init?(coder: NSCoder) { fatalError() }
|
||
|
||
override func draw(_ rect: CGRect) {
|
||
guard let context = UIGraphicsGetCurrentContext() else { return }
|
||
context.fillKeycap(bounds, fill: palette.keyFill, shadow: palette.keyShadow, radius: cornerRadius)
|
||
let config = UIImage.SymbolConfiguration(pointSize: isLandscape ? 16 : 19, weight: .regular)
|
||
guard let image = UIImage(systemName: "globe", withConfiguration: config)?
|
||
.withTintColor(palette.textBase, renderingMode: .alwaysOriginal) else { return }
|
||
image.draw(at: CGPoint(x: bounds.midX - image.size.width / 2, y: bounds.midY - image.size.height / 2 + (isLandscape ? 0 : -1)))
|
||
}
|
||
}
|
||
|
||
/// Puts `LiveDictationPreviewView` in place of a full keyboard's keys while it records with
|
||
/// live words (`KeyboardViewModel.liveText`), and takes it away, with the keys back, once
|
||
/// the dictation is in: the preview fades with the words it had while the keys and the
|
||
/// dictation's own text come in, so different words never flash in it.
|
||
@MainActor
|
||
final class LiveDictationPreviewHost {
|
||
private weak var keyboard: FullKeyboardView?
|
||
private var preview: LiveDictationPreviewView?
|
||
private var subscription: AnyCancellable?
|
||
|
||
/// The keys out and the words in as recording starts, and back as the dictation goes in, in
|
||
/// one change each, never waiting for the other half: springs with no bounce. The words
|
||
/// settle from a few points below as they come, and drift a few points up, toward the field
|
||
/// their text goes into, as they go. With Reduce Motion, fades only.
|
||
enum Motion {
|
||
static let keysOut: TimeInterval = 0.2
|
||
static let wordsIn: TimeInterval = 0.3
|
||
static let wordsOut: TimeInterval = 0.15
|
||
static let keysIn: TimeInterval = 0.3
|
||
static let travel: CGFloat = 6
|
||
}
|
||
|
||
func attach(to keyboard: FullKeyboardView) {
|
||
self.keyboard = keyboard
|
||
subscription = keyboard.model.liveText.$transcript.sink { [weak self] transcript in
|
||
self?.sync(transcript)
|
||
}
|
||
}
|
||
|
||
var palette = FullKeyboardPalette(isDark: false) {
|
||
didSet { preview?.palette = palette }
|
||
}
|
||
|
||
/// The recording state changed.
|
||
func sync() {
|
||
sync(keyboard?.model.liveText.transcript)
|
||
}
|
||
|
||
private func sync(_ transcript: LiveTranscript?) {
|
||
guard let keyboard else { return }
|
||
let live = keyboard.model.liveText
|
||
guard keyboard.showsRecording, live.showsPreview, let transcript else {
|
||
hide()
|
||
return
|
||
}
|
||
let preview = self.preview ?? show(in: keyboard)
|
||
preview.show(transcript)
|
||
}
|
||
|
||
private func show(in keyboard: FullKeyboardView) -> LiveDictationPreviewView {
|
||
let preview = LiveDictationPreviewView()
|
||
preview.palette = keyboard.palette
|
||
keyboard.insertSubview(preview, aboveSubview: keyboard.plane)
|
||
self.preview = preview
|
||
layout()
|
||
guard keyboard.window != nil else {
|
||
keyboard.plane.alpha = 0
|
||
return preview
|
||
}
|
||
let moves = !UIAccessibility.isReduceMotionEnabled
|
||
preview.alpha = 0
|
||
preview.transform = moves ? CGAffineTransform(translationX: 0, y: Motion.travel) : .identity
|
||
Self.spring(moves ? Motion.wordsIn : 0.2) {
|
||
preview.alpha = 1
|
||
preview.transform = .identity
|
||
}
|
||
Self.spring(moves ? Motion.keysOut : 0.2) {
|
||
keyboard.plane.alpha = 0
|
||
}
|
||
return preview
|
||
}
|
||
|
||
/// The keys take touches again at once: the words stop taking them as they start to go.
|
||
private func hide() {
|
||
guard let preview, let keyboard else { return }
|
||
self.preview = nil
|
||
preview.isUserInteractionEnabled = false
|
||
let cost = preview.words.takeCost()
|
||
if cost.updates > 0 {
|
||
Self.log.notice("live: preview drew \(cost.updates, privacy: .public) updates, \(String(format: "%.2f", cost.averageMS), privacy: .public) ms on average, \(String(format: "%.2f", cost.slowestMS), privacy: .public) at most")
|
||
}
|
||
guard keyboard.window != nil else {
|
||
keyboard.plane.alpha = 1
|
||
preview.removeFromSuperview()
|
||
return
|
||
}
|
||
let moves = !UIAccessibility.isReduceMotionEnabled
|
||
Self.spring(moves ? Motion.wordsOut : 0.2) {
|
||
preview.alpha = 0
|
||
if moves { preview.transform = CGAffineTransform(translationX: 0, y: -Motion.travel) }
|
||
} completion: {
|
||
preview.removeFromSuperview()
|
||
}
|
||
Self.spring(moves ? Motion.keysIn : 0.2) {
|
||
keyboard.plane.alpha = 1
|
||
}
|
||
}
|
||
|
||
private static let log = Logger(subsystem: "anomaly.hex.mobile.keyboard", category: "capture-probe")
|
||
|
||
private static func spring(_ duration: TimeInterval, _ changes: @escaping () -> Void, completion: (() -> Void)? = nil) {
|
||
UIView.animate(
|
||
springDuration: duration, bounce: 0, initialSpringVelocity: 0, delay: 0,
|
||
options: [.beginFromCurrentState, .allowUserInteraction],
|
||
animations: changes,
|
||
completion: { _ in completion?() }
|
||
)
|
||
}
|
||
|
||
/// Where the keys were, under the recording bar to the keyboard's bottom.
|
||
func layout() {
|
||
guard let preview, let keyboard else { return }
|
||
let layout = keyboard.engine.layout
|
||
preview.frame = layout.keysAreaFrame.offsetBy(dx: keyboard.plane.frame.minX, dy: keyboard.plane.frame.minY)
|
||
let side = layout.horizontalInset + layout.metrics.side + 4
|
||
var bottom: CGFloat = 10
|
||
// The globe keeps working where the keyboard has one: its button is above the words.
|
||
if let globe = layout.key(.globe), !keyboard.globeButton.isHidden {
|
||
let frame = globe.frame.offsetBy(dx: keyboard.plane.frame.minX - preview.frame.minX, dy: keyboard.plane.frame.minY - preview.frame.minY)
|
||
preview.globeFrame = frame
|
||
bottom = max(bottom, preview.bounds.height - frame.minY + 4)
|
||
preview.globeRadius = layout.metrics.cornerRadius
|
||
} else {
|
||
preview.globeFrame = nil
|
||
}
|
||
preview.textInsets = UIEdgeInsets(top: layout.metrics.rowGap + 4, left: side, bottom: bottom, right: side)
|
||
preview.orientation = keyboard.orientation
|
||
}
|
||
}
|