mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* iOS keyboard: the live preview's motion, smooth and calm
What: one spring family (duration and bounce 0, UIKit's springDuration curve) for every
change while dictating, all of it drawn by one view so no memory comes back:
- Keys out, words in as recording starts: the keys fade (.2) as the words fade in rising
6 pt (.3), the same frame. Words out, keys back as the dictation goes in: the words fade
drifting 6 pt up toward the field (.15) as the keys fade in (.3); the words stop taking
touches at once, so the keys work the moment the dictation ends.
- Words arriving (LiveWordsView): each new word fades in rising 4 pt (.3), a burst's words
40 ms apart and all of them within 240 ms, so the last is in long before Apple's next
burst (about once a second). "Listening…" fades out a beat (60 ms) ahead of the first words.
- Tentative to settled: the muted tone eases to the keys' ink (.3).
- A new line: the block moves up by a critically damped spring (.3) that carries on from
where it is when another line comes, and the line leaving the top fades through a 14 pt band
instead of being cut.
- A changed word ("Okay" to "Hey", "730." to "7.30."): the old word fades out where it was
(.15) as the new one fades in (.2); words keep their place by position, so the words after
it never re-fade, and a word that only gains a mark stays as it is.
- The recording bar's morph has no bounce any more (was damping .82) and its fades are the
same spring (.2), so nothing bounces.
- Reduce Motion: fades only (.2), nothing rises, drifts or slides.
LiveWordsView draws the words with one TextKit 1 layout into one backing store the size of the
box, from a display link that runs only while something moves, and drops whole lines well out
of view so the lines that stay never re-break. It replaces the preview's label.
Why: Ryan asked for smoother motion with better easing after watching the phone recordings
(keys and words swapped in 180 ms with a double exposure; a burst's words popped in at once).
Verified: LiveDictationTests (10) on the Studio simulator; the Studio simulator with the 60 s
replay: footprint +2.7 MB at most while recording (+5.7 with #2410's label), 1,868 frames
drawn while moving, none late; frames of a recording checked for the start, a burst, a new
line and the end. code-shape.py passes.
* iOS keyboard: the preview's words move by Core Animation, not a display link
What: LiveWordsView no longer redraws every frame. It lays the words out and draws them once
per update, in one layer, and Core Animation does every frame of the motion: what changed
cross-fades in place (0.3 s, the cubic closest to a no-bounce spring: new words, a settled
tail turning to ink, a word the recogniser changed), the block moves up by whole lines with a
no-bounce spring (0.3 s) inside a box whose 14 pt band fades the line leaving the top, and once
at rest that line fades out of the band and the lines above the box are dropped from the layout
with nothing on screen changing. The per-word stagger and rise are gone.
Why: on the iPhone 17 Pro the display link drawing the words got only a quarter of its frames
on time while recording (intervals 12-20 ms 26%, 20-30 ms 23%, 30-45 ms 27%, longer 24%; the
drawing itself took 0.3 ms): the main thread is shared with the capture, the recogniser and the
Dot Scope. Core Animation's own animations keep the display's pace regardless.
Verified: LiveDictationTests (9, including whole-line scrolling and dropping the lines above the
box) and recordings on the Studio simulator: a burst, settling, a new line with the band's fade,
the drop at rest, the start and the end; 60 s replay +5.9 MB at most against +7.0 MB for the
same recording without the preview; 1.4 ms of main thread per update.
* iOS README: the live preview's motion, as it now runs
What: the live-words bullet in ios/README.md describes the motion as it ships: one family (a
no-bounce spring by its duration, and its closest cubic for the cross-fade), Core Animation
running every frame, what changes cross-fading in place while the rest stays exactly as it is,
whole-line scrolling through a 14 pt band, and Reduce Motion as cross-fades only. Replaces the
paragraph about per-word staggers and a display link, which the previous commit removed.
Why: the README is where the keyboard's behaviour is written down for the next person.
Verified on the iPhone 17 Pro with 9869899c installed, warm (Parakeet compiled before measuring):
- 60 s dictation, preview on: 59.1 MB at the end, 62.9 MB peak while recording, 76.3 MB at
transcription (peak 77.3); #2410 was 60.1 / 66.5 / 78.7. Preview off: 58.3 / 60.0 / 75.3.
- The preview's main-thread share: 1.69 ms per update on average, 3.40 ms at most (242 updates).
- In the 30 fps phone recordings, words that didn't change hold the same brightness to the
decimal through every cross-fade near them; a changing line eases over about 0.2 s with no
stall; the field's text appears on the frame the preview's words go.
Correction to 9869899c's message: its "+7.0 MB for the same recording without the preview" on
the simulator wasn't measured (the simulator's memory script only runs with the preview); the
phone numbers above are the comparison.
277 lines
13 KiB
Swift
277 lines
13 KiB
Swift
import os
|
||
import QuartzCore
|
||
import UIKit
|
||
|
||
/// How the preview's words move: Core Animation does every frame, so the motion keeps the
|
||
/// display's pace however busy the keyboard's main thread is with the capture and the
|
||
/// recogniser (on an iPhone 17 Pro a display link drawing the words got only a quarter of its
|
||
/// frames on time while recording). One family: a spring with no bounce by its duration, and
|
||
/// for the cross-fade the curve closest to it.
|
||
nonisolated enum LiveWordsMotion {
|
||
/// What changed in the words (new words, a settled tail turning to ink, a word the
|
||
/// recogniser changed) cross-fades in place; what didn't change stays exactly as it is.
|
||
static let change: TimeInterval = 0.3
|
||
/// A new line: the block moves up by a line, the line leaving the top fading through `edge`.
|
||
static let scroll: TimeInterval = 0.3
|
||
/// With Reduce Motion: the cross-fade only, shorter, and lines jump.
|
||
static let reducedChange: TimeInterval = 0.2
|
||
/// The cubic closest to `springDuration: d, bounce: 0` (1 − (1 + ωt)e^(−ωt), ω = 2π/d):
|
||
/// 47%, 82% and 95% of the way at a quarter, half and three quarters of the time.
|
||
static var curve: CAMediaTimingFunction { CAMediaTimingFunction(controlPoints: 0.25, 0.6, 0.3, 1) }
|
||
}
|
||
|
||
/// The preview's words in a box: one layer drawn from a TextKit layout only when the words
|
||
/// change, inside a clip whose bounds scroll by whole lines. Lines well above the box are
|
||
/// dropped, whole, so a minute's dictation draws no more than a few lines more than the box.
|
||
@MainActor
|
||
final class LiveWordsView: UIView {
|
||
var palette = FullKeyboardPalette(isDark: false) { didSet { if palette != oldValue { relayout(animated: false) } } }
|
||
var font = UIFont.systemFont(ofSize: 23) { didSet { if font != oldValue { relayout(animated: false) } } }
|
||
var placeholder = "" { didSet { if placeholder != oldValue { relayout(animated: false) } } }
|
||
/// The band above the box where a line leaving the top fades out.
|
||
static let edge: CGFloat = 14
|
||
|
||
private let canvas = LiveWordsCanvas()
|
||
/// Scrolls (its bounds' origin moves) under the box and the band, which stay put.
|
||
private let scroller = UIView()
|
||
private let edgeMask = CAGradientLayer()
|
||
private var text = ""
|
||
private var settled = 0
|
||
/// Characters of `text` dropped from the top, whole lines that left the box.
|
||
private var dropped = 0
|
||
private var scrolled: CGFloat = 0
|
||
private var trimWork: DispatchWorkItem?
|
||
/// Changes with every new layout, so a trim begun before it never lands after it.
|
||
private var generation = 0
|
||
/// The main thread's share of each update (laying out and drawing), for the probe log.
|
||
private var updates = 0
|
||
private var updateTime: CFTimeInterval = 0
|
||
private var slowestUpdate: CFTimeInterval = 0
|
||
private let reduceMotion = UIAccessibility.isReduceMotionEnabled
|
||
|
||
var lineHeight: CGFloat { (font.lineHeight * 1.08).rounded(.up) }
|
||
|
||
override init(frame: CGRect) {
|
||
super.init(frame: frame)
|
||
backgroundColor = .clear
|
||
isUserInteractionEnabled = false
|
||
clipsToBounds = true
|
||
scroller.isUserInteractionEnabled = false
|
||
addSubview(scroller)
|
||
scroller.addSubview(canvas)
|
||
// The band above the first line: clear at the top, opaque by the line.
|
||
edgeMask.colors = [UIColor.clear.cgColor, UIColor.black.cgColor]
|
||
edgeMask.startPoint = CGPoint(x: 0.5, y: 0)
|
||
edgeMask.endPoint = CGPoint(x: 0.5, y: 1)
|
||
layer.mask = edgeMask
|
||
}
|
||
|
||
@available(*, unavailable)
|
||
required init?(coder: NSCoder) { fatalError() }
|
||
|
||
/// Where the words are now (their top in this view's scrolled coordinates), for tests.
|
||
var scrollOffset: CGFloat { scroller.bounds.origin.y }
|
||
var drawnHeight: CGFloat { canvas.bounds.height }
|
||
|
||
func takeCost() -> (updates: Int, averageMS: Double, slowestMS: Double) {
|
||
defer { updates = 0; updateTime = 0; slowestUpdate = 0 }
|
||
return (updates, updates > 0 ? updateTime / Double(updates) * 1000 : 0, slowestUpdate * 1000)
|
||
}
|
||
|
||
func show(text: String, settled: Int) {
|
||
guard text != self.text || settled != self.settled else { return }
|
||
let started = CACurrentMediaTime()
|
||
defer {
|
||
// The drawing it asked for happens before the frame is shown: counted with it.
|
||
canvas.layer.displayIfNeeded()
|
||
let took = CACurrentMediaTime() - started
|
||
updates += 1
|
||
updateTime += took
|
||
slowestUpdate = max(slowestUpdate, took)
|
||
}
|
||
if !text.hasPrefix(String(self.text.prefix(dropped))) || text.count < dropped { dropped = 0 }
|
||
self.text = text
|
||
self.settled = settled
|
||
relayout(animated: window != nil)
|
||
}
|
||
|
||
override func layoutSubviews() {
|
||
super.layoutSubviews()
|
||
CATransaction.begin()
|
||
CATransaction.setDisableActions(true)
|
||
edgeMask.frame = bounds
|
||
edgeMask.locations = [0, NSNumber(value: Double(Self.edge / max(1, bounds.height)))]
|
||
scroller.bounds.size = bounds.size
|
||
scroller.center = CGPoint(x: bounds.midX, y: bounds.midY)
|
||
CATransaction.commit()
|
||
if canvas.bounds.width != bounds.width { relayout(animated: false) }
|
||
}
|
||
|
||
/// Draws the words as they are now. Animated: the change cross-fades, and the block moves
|
||
/// up by whole lines when the last line no longer fits.
|
||
private func relayout(animated: Bool) {
|
||
guard bounds.width > 0 else { return }
|
||
generation += 1
|
||
setBand(clear: false)
|
||
let moves = animated && !reduceMotion
|
||
let attributed = attributedWords()
|
||
let height = canvas.layOut(attributed, width: bounds.width, lineHeight: lineHeight)
|
||
let visible = max(lineHeight, bounds.height - Self.edge)
|
||
let overflow = height - visible
|
||
let target = overflow > 0 ? (overflow / lineHeight).rounded(.up) * lineHeight : 0
|
||
if animated {
|
||
let fade = CATransition()
|
||
fade.type = .fade
|
||
fade.duration = reduceMotion ? LiveWordsMotion.reducedChange : LiveWordsMotion.change
|
||
fade.timingFunction = LiveWordsMotion.curve
|
||
canvas.layer.add(fade, forKey: "words")
|
||
}
|
||
CATransaction.begin()
|
||
CATransaction.setDisableActions(true)
|
||
canvas.frame = CGRect(x: 0, y: Self.edge, width: bounds.width, height: max(height, lineHeight))
|
||
CATransaction.commit()
|
||
canvas.setNeedsDisplay()
|
||
if target != scrolled {
|
||
scrolled = target
|
||
let place = { self.scroller.bounds.origin.y = target }
|
||
if moves {
|
||
UIView.animate(springDuration: LiveWordsMotion.scroll, bounce: 0, initialSpringVelocity: 0, delay: 0,
|
||
options: [.beginFromCurrentState, .allowUserInteraction], animations: place)
|
||
} else {
|
||
place()
|
||
}
|
||
}
|
||
scheduleTrim()
|
||
}
|
||
|
||
private func attributedWords() -> NSAttributedString {
|
||
let shown = Array(text).dropFirst(dropped)
|
||
let paragraph = NSMutableParagraphStyle()
|
||
paragraph.minimumLineHeight = lineHeight
|
||
paragraph.maximumLineHeight = lineHeight
|
||
paragraph.lineBreakMode = .byWordWrapping
|
||
let base: [NSAttributedString.Key: Any] = [.font: font, .paragraphStyle: paragraph]
|
||
guard !shown.isEmpty else {
|
||
return NSAttributedString(string: placeholder, attributes: base.merging([.foregroundColor: palette.textMuted]) { $1 })
|
||
}
|
||
let ink = max(0, min(shown.count, settled - dropped))
|
||
let result = NSMutableAttributedString(string: String(shown.prefix(ink)), attributes: base.merging([.foregroundColor: palette.textBase]) { $1 })
|
||
result.append(NSAttributedString(string: String(shown.dropFirst(ink)), attributes: base.merging([.foregroundColor: palette.textMuted]) { $1 }))
|
||
return result
|
||
}
|
||
|
||
/// Once the words are at rest, the lines above the box go, whole: the words are laid out
|
||
/// from the top line's first word and the block moves down by as much in the same frame,
|
||
/// so nothing on screen changes, the lines that stay break as they did, and nothing waits in
|
||
/// the band above the box but the line the next move takes out through it.
|
||
private func scheduleTrim() {
|
||
trimWork?.cancel()
|
||
let work = DispatchWorkItem { [weak self] in self?.trim() }
|
||
trimWork = work
|
||
DispatchQueue.main.asyncAfter(deadline: .now() + max(LiveWordsMotion.change, LiveWordsMotion.scroll) + 0.05, execute: work)
|
||
}
|
||
|
||
/// The band's lines fade out first (what's left of the line that just left the top), then
|
||
/// they go from the layout with nothing on screen changing.
|
||
private func trim() {
|
||
let above = Int((scrolled / lineHeight).rounded())
|
||
guard above > 0, canvas.characterStartingLine(above) != nil else { return }
|
||
let started = generation
|
||
CATransaction.begin()
|
||
CATransaction.setAnimationDuration(reduceMotion ? 0 : LiveWordsMotion.reducedChange)
|
||
CATransaction.setAnimationTimingFunction(LiveWordsMotion.curve)
|
||
CATransaction.setCompletionBlock { [weak self] in
|
||
guard let self, self.generation == started else { return }
|
||
self.dropLines(above)
|
||
}
|
||
setBand(clear: true)
|
||
CATransaction.commit()
|
||
}
|
||
|
||
/// The band above the box: a fade from clear at its top to opaque at the first line, or
|
||
/// clear all through once the words are at rest.
|
||
private func setBand(clear: Bool) {
|
||
let band = Double(Self.edge / max(1, bounds.height))
|
||
let locations: [NSNumber] = clear ? [NSNumber(value: band * 0.99), NSNumber(value: band)] : [0, NSNumber(value: band)]
|
||
guard edgeMask.locations != locations else { return }
|
||
if !clear {
|
||
CATransaction.begin()
|
||
CATransaction.setDisableActions(true)
|
||
}
|
||
edgeMask.locations = locations
|
||
if !clear { CATransaction.commit() }
|
||
}
|
||
|
||
private func dropLines(_ above: Int) {
|
||
guard let start = canvas.characterStartingLine(above) else { return }
|
||
CATransaction.begin()
|
||
CATransaction.setDisableActions(true)
|
||
dropped += start
|
||
let shift = CGFloat(above) * lineHeight
|
||
let height = canvas.layOut(attributedWords(), width: bounds.width, lineHeight: lineHeight)
|
||
canvas.frame = CGRect(x: 0, y: Self.edge, width: bounds.width, height: max(height, lineHeight))
|
||
canvas.setNeedsDisplay()
|
||
scrolled -= shift
|
||
scroller.bounds.origin.y = scrolled
|
||
setBand(clear: false)
|
||
CATransaction.commit()
|
||
}
|
||
}
|
||
|
||
/// The words' one drawn layer: a TextKit 1 layout, drawn only when the words change.
|
||
@MainActor
|
||
private final class LiveWordsCanvas: UIView {
|
||
private let storage = NSTextStorage()
|
||
private let layout = NSLayoutManager()
|
||
private let container = NSTextContainer(size: .zero)
|
||
private var lineHeight: CGFloat = 0
|
||
|
||
override init(frame: CGRect) {
|
||
super.init(frame: frame)
|
||
backgroundColor = .clear
|
||
isOpaque = false
|
||
isUserInteractionEnabled = false
|
||
contentMode = .topLeft
|
||
container.lineFragmentPadding = 0
|
||
layout.addTextContainer(container)
|
||
storage.addLayoutManager(layout)
|
||
}
|
||
|
||
@available(*, unavailable)
|
||
required init?(coder: NSCoder) { fatalError() }
|
||
|
||
/// Lays `text` out at `width` and returns its height in whole lines.
|
||
func layOut(_ text: NSAttributedString, width: CGFloat, lineHeight: CGFloat) -> CGFloat {
|
||
self.lineHeight = lineHeight
|
||
container.size = CGSize(width: width, height: .greatestFiniteMagnitude)
|
||
storage.setAttributedString(text)
|
||
layout.ensureLayout(for: container)
|
||
var lines = 0
|
||
if layout.numberOfGlyphs > 0 {
|
||
layout.enumerateLineFragments(forGlyphRange: NSRange(location: 0, length: layout.numberOfGlyphs)) { _, _, _, _, _ in lines += 1 }
|
||
}
|
||
return CGFloat(lines) * lineHeight
|
||
}
|
||
|
||
/// How many characters come before line `line` (0 is the first).
|
||
func characterStartingLine(_ line: Int) -> Int? {
|
||
var index = 0
|
||
var found: Int?
|
||
guard layout.numberOfGlyphs > 0 else { return nil }
|
||
layout.enumerateLineFragments(forGlyphRange: NSRange(location: 0, length: layout.numberOfGlyphs)) { _, _, _, glyphs, stop in
|
||
if index == line {
|
||
found = self.layout.characterIndexForGlyph(at: glyphs.location)
|
||
stop.pointee = true
|
||
}
|
||
index += 1
|
||
}
|
||
guard let found else { return nil }
|
||
// In characters of the string, as `dropped` counts them.
|
||
return (storage.string as NSString).substring(to: found).count
|
||
}
|
||
|
||
override func draw(_ rect: CGRect) {
|
||
guard layout.numberOfGlyphs > 0 else { return }
|
||
layout.drawGlyphs(forGlyphRange: NSRange(location: 0, length: layout.numberOfGlyphs), at: .zero)
|
||
}
|
||
}
|