hex/ios/HEXShared/KeyboardsOnScreen.swift
Ryan Vogel 994b346982
iOS keyboard: stop leaking views, capture web views, and emoji indices across shows (#2352)
What was wrong:
iOS creates a new KeyboardViewController almost every time the keyboard comes up inside
one HEXKeyboard process. Three things stayed behind after each presentation (~0.44 MB
per empty show, ~0.82 MB once a word was typed) until iOS killed the keyboard over its
memory limit and fell back to the system keyboard:

1. KeyboardLocalDictation built a new WKWebView (KeyboardWebCapture) for every controller
   and never detached it from KeyboardInputView when the keyboard hid or deallocated.
   WebKit's process-wide observers kept each WKWebView alive, and its Auto Layout engine
   kept the whole dismissed KeyboardInputView and FullKeyboardView alive with it.
2. On iOS 27, UIKit keeps every UIInputView shell alive in ObjC associated storage after
   its controller deallocates, so any subviews Hex left attached (FullKeyboardView, the
   bar, KeyboardBackdropView's UIVisualEffectView) stayed in memory too. In hosts with an
   autofocusing inputAccessoryView (#2302), iOS also keeps earlier controllers alive
   without calling viewWillDisappear, leaving their views and 4 Hz poll timers running.
3. EmojiLibrary.load built a new ~0.8 MB index per FullKeyboardView instead of sharing
   one copy per process like TypingDecoderModel and GermanCapitals.

What changed (and what was replaced):
- Replaced per-controller WKWebView creation with one shared KeyboardWebCapture per
  extension process, guarded by KeyboardCaptureOwnership so only the current owner (or
  any caller while parked with no owner) can detach, park, or evict it on a memory
  warning. A stale controller that hid or received a memory warning drops only its own
  pointer instead of stopping a newer keyboard's active recording.
- Replaced eager KeyboardBackdropView allocation with a lazy backdrop created only while
  KeyboardsOnScreen.isOverAnother is true. KeyboardInputView records weak references to
  only the subviews Hex mounted (surface, backdrop, barEmojiPanel) and removes those
  exact views in viewDidDisappear and deinit without touching UIKit's own views.
- Superseded controllers in the same named field (KeyboardsOnScreen.isSuperseded) stop
  their poll timer and unmount their surface; if the newer keyboard above leaves while
  the older one is still on screen, restoreRetiredKeyboard() rebuilds only its surface
  and poll timer without handing off number pads, auto-starting dictation, or resetting
  its cold-start job. Cleared handOffWaitsForPosture on appear, disappear, teardown, and
  retire.
- Added EmojiLibrary.bundled(language:bundle:) and EmojiLibrary.evictCached() to share
  one index across presentations until a memory warning.

Checked:
- Release on iPhone Air iOS 27.2 simulator vs origin/dev:
  - 30 hide/shows + 20 Apple switches: 46.35 MB (peak 54.71 MB) -> 25.25 MB (peak
    33.57 MB); FullKeyboardView 51 -> 1, WKWebView 51 -> 1, KeyboardBackdropView 51 -> 0.
  - 6 switches in autofocus inputAccessoryView host: 28.50 MB -> 24.96 MB (1 live
    FullKeyboardView, 1 WKWebView, 1 backdrop on the top keyboard).
  - First-frame commit time (20 shows): 1.19 ms median (new controller) vs 1.25 ms on
    dev; 15.66 ms when forcibly reusing one controller; keys present across Messages,
    Safari address bar and web fields, Spotlight, two-field focus, and rotation.
  - 48 kHz mock-mic dictation across first show, 10 hide/shows, keyboard switches, bar,
    auto-start, hide mid-recording/transcribing, active recording surviving an older
    controller's memory warning/hide/deinit, and number-pad restore without hand-off or
    auto-start.
- xcodebuild test -scheme HEXMobile: 808 executed, 3 skipped, 0 failures; code-shape and
  hex-agent-policy clean.
2026-11-22 06:45:38 -05:00

137 lines
6.4 KiB
Swift

import Foundation
/// Hex's keyboards that iOS has on screen at once. It should be one.
///
/// iOS makes a new input view controller each time the keyboard comes up, tells the one before
/// that it disappeared, and frees it. In one case it doesn't (iOS 27.0 and 27.2 simulators,
/// every time): a field that takes focus as its screen appears and has an input accessory
/// view, opened with Hex. There, switching to Apple's keyboard never tells Hex it disappeared
/// and keeps its remote view in the app's view tree; Apple's keyboard is added beside it, and
/// switching back puts a second Hex on top without taking Apple's away (a
/// `UIKeyboardAutomatic` between two remote views in `_UIKBCompatInputView`). Hex's keys are
/// translucent on iOS's glass, so Apple's keys showed through a row above its own (TestFlight
/// build 56, iPhone 17, iOS 27.0.1). Opened with Apple's keyboard, the same field hides and
/// frees Hex as usual, and nothing is left behind.
///
/// A keyboard can't see the app's views, but it can see this: an earlier Hex keyboard that
/// was never told to disappear, still on screen in the same field. That is exactly when
/// Apple's keys may be under the new one, and the only time Hex draws its backdrop
/// (`KeyboardBackdropView`). Otherwise Hex draws nothing behind its keys and is iOS's glass.
nonisolated struct KeyboardsOnScreen: Equatable, Sendable {
private struct Entry: Equatable, Sendable {
let keyboard: Int
/// The field the keyboard is in, nil until UIKit names it.
var field: UUID?
}
/// In the order they appeared.
private var entries: [Entry] = []
var count: Int { entries.count }
/// `keyboard` came on screen in `field`, after every keyboard already there.
mutating func appeared(_ keyboard: Int, in field: UUID?) {
entries.removeAll { $0.keyboard == keyboard }
entries.append(Entry(keyboard: keyboard, field: field))
}
/// UIKit named `keyboard`'s field, or it moved to another. Nothing for a keyboard off screen.
mutating func moved(_ keyboard: Int, to field: UUID?) {
guard let index = entries.firstIndex(where: { $0.keyboard == keyboard }) else { return }
entries[index].field = field
}
/// iOS said `keyboard` disappeared, or freed it.
mutating func disappeared(_ keyboard: Int) {
entries.removeAll { $0.keyboard == keyboard }
}
/// Whether an earlier keyboard is still on screen in `keyboard`'s field: iOS kept it, and
/// with it whatever it showed in between. A field UIKit hasn't named counts as the same
/// one, so the doubt covers.
func isOverAnother(_ keyboard: Int) -> Bool {
guard let index = entries.firstIndex(where: { $0.keyboard == keyboard }) else { return false }
let field = entries[index].field
return entries[..<index].contains { $0.field == nil || field == nil || $0.field == field }
}
/// Whether a later keyboard is currently on screen in `keyboard`'s own named field: `keyboard`
/// sits underneath it and can retire its surface until no later keyboard remains in that field.
func isSuperseded(_ keyboard: Int) -> Bool {
guard let index = entries.firstIndex(where: { $0.keyboard == keyboard }),
let field = entries[index].field else { return false }
return entries[(index + 1)...].contains { $0.field == field }
}
}
/// Single-owner bookkeeping for the keyboard process's shared capture web view. Only the
/// current owner—or any caller while the capture is parked with no owner—may park, evict on a
/// memory warning, or tear down the shared web view; a former owner that still held a pointer
/// drops only its own reference so an active recording on a newer keyboard is never disturbed.
nonisolated struct KeyboardCaptureOwnership: Equatable, Sendable {
private(set) var hasSharedCapture = false
private(set) var owner: Int?
private(set) var isCapturing = false
/// Whether `requester` may attach or start the shared capture. False while another keyboard
/// is actively recording with it.
func canAcquire(by requester: Int) -> Bool {
!isCapturing || owner == requester
}
/// `requester` attached the shared capture while idle (creating it if needed).
mutating func acquireIdle(by requester: Int) {
hasSharedCapture = true
owner = requester
isCapturing = false
}
/// `requester` armed or started microphone capture on the shared web view.
mutating func beginCapture(by requester: Int) {
hasSharedCapture = true
owner = requester
isCapturing = true
}
/// `requester`'s capture stopped, failed or was discarded while its keyboard stays on screen.
mutating func endCapture(by requester: Int) {
guard owner == requester else { return }
isCapturing = false
}
/// `requester`'s keyboard hid: true when `requester` owned the shared capture (or nobody did),
/// so the caller detaches and parks it; false when another keyboard now owns it.
mutating func park(by requester: Int) -> Bool {
guard hasSharedCapture, owner == nil || owner == requester else { return false }
owner = nil
isCapturing = false
return true
}
/// `requester` deallocated: true when `requester` still owned the shared capture (or nobody
/// did), so the caller stops, detaches and parks it; false when another keyboard owns it.
mutating func ownerDidDeinit(_ requester: Int) -> Bool {
guard hasSharedCapture, owner == nil || owner == requester else { return false }
owner = nil
isCapturing = false
return true
}
/// `requester` received a memory warning while idle: true only when the shared capture is not
/// capturing and either nobody owns it or `requester` itself owns it.
mutating func evictForMemoryWarning(by requester: Int) -> Bool {
guard hasSharedCapture, !isCapturing, owner == nil || owner == requester else { return false }
hasSharedCapture = false
owner = nil
return true
}
/// WebKit's content process terminated under `requester` (or `nil` while parked).
mutating func dropTerminated(by requester: Int?) -> Bool {
guard hasSharedCapture, requester == nil || owner == nil || owner == requester else { return false }
hasSharedCapture = false
owner = nil
isCapturing = false
return true
}
}