hex/ios/HEXShared/KeyboardSelfDictation.swift
Ryan Vogel 62ba0b4984
iOS: a warm session dictates in the background on iOS 27 again, and a keyboard that opens Hex brings the warm default back (#2406)
TestFlight build 129 (iPhone 17 Pro, iOS 27.3): Dictate opened Hex every time, and back
in the chat app the Hex key said "Another app is using audio. Try again."

Cause, reproduced on an iPhone 17 Pro on iOS 27.2 with build 128's code (84915a9f,
before #2393) as well as dev: a warm session's dictation fails as soon as Hex is in the
background. beginCapture calls configureAudioSession, which set the category again even
though it was unchanged (the keepalive's .playAndRecord/.measurement/same options). On iOS
27 that asks to interrupt others, which audiomxd refuses for a background app that isn't
Now Playing ("can't stop others because it is in the background", "BeginInterruption
returned an error"), so setCategory throws cannotInterruptOthers ('!int'), which Hex shows
as "Another app is using audio". The keepalive restart after a capture fails the same way
in the background, so the warm session silently ends. #2393 didn't cause this, but its
Off default meant a keyboard that opened Hex for a reason it left no record of got a one
shot each time and no warm session.

What changed
- AudioBackend.configureAudioSession sets the category only when category, mode or
  options differ (AudioSessionControlling now exposes category, mode and categoryOptions,
  which AVAudioSession already has). A Bluetooth choice, which needs HFP, still sets it.
- Hex records every keyboard launch for a dictation as "the keyboard didn't dictate by
  itself" (KeyboardSelfDictation.recordOpenedHex, reason openedHex, keeping a reason the
  keyboard wrote), before reading the warm default, so the 15-minute default, the row and
  a warm cold start come back from that launch whatever the keyboard's reason was.
- The record is cleared when the keyboard delivers a dictation it made itself (insertResult
  with source "keyboard"), not when one merely starts (replaced: the clear in the
  onLocalDictateTap closure).
- Removed: "Off where the keyboard dictates by itself: a launch that missed its recording
  warms nothing" in activateWarmSessionAndRunHandoff. Every keyboard launch now records,
  so it could no longer apply; a late handoff warms as it did before 129.
- README's warm-session paragraph says so.

Verified
- iPhone 17 Pro, iOS 27.2 (UDID 00008150-00163C5C0A47801C), builds signed for the device:
  - build 128 code: Keep mic warm 15m, then the Hex key in another app shows "Another app
    is using audio. Try again." (the report's screenshot); the next tap dictates in the
    keyboard.
  - this branch: a keyboard cold start (hex-dictation://keyboard/record) records, Done in
    the keyboard keeps the warm session (Hex key orange), the next Hex key records through
    Hex in the background with no error, and the warm session stays after it. The App Group
    then holds keyboard.handedDictationToHex = openedHex and Settings shows Keep mic warm
    at 15m.
  - control: with Hex in the back and no warm session, the keyboard dictates by itself,
    also with Safari playing audio.
- xcodebuild test on an iOS 27.2 simulator: 878 tests, 3 skipped, 0 failures. New:
  AudioSessionSetupTests.testTheCategoryIsSetOnlyWhenItChanges; updated
  testKeepMicWarmFollowsWhetherTheKeyboardDictatesItself (a late handoff records and warms)
  and KeyboardBridgeTests (recordOpenedHex keeps the keyboard's reason).
- Not checked: iOS 27.3 itself, iOS 26, and a real phone whose keyboard can't dictate by
  itself (memory, missing model); a keyboard-made dictation clearing the record was only
  unit-tested.
- python3 scripts/code-shape.py passes; git diff --check clean.
2026-11-22 13:12:24 -05:00

75 lines
3.9 KiB
Swift
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import Foundation
/// Whether the keyboard dictates by itself, in its own process with Hex closed (iOS 26,
/// `KeyboardLocalDictation`), so Hex needn't keep the mic warm for it: there Keep mic warm
/// defaults to Off and its row hides (`KeyboardBridge.warmDuration`). Not before iOS 26, and
/// not while the keyboard hands its dictations to Hex: it couldn't get its speech model
/// (`LanguageSettings.keyboardNeedsHex`), Experiments › Multilingual model takes the language
/// and only Hex loads it, or the keyboard opened Hex for a dictation in the last
/// `recentWindow` (recorded by the keyboard, and by Hex whatever the reason), until the
/// keyboard next finishes a dictation by itself.
nonisolated struct KeyboardSelfDictation {
enum Reason: String {
case memory
case speechModel
case multilingual
/// Hex saw the keyboard open it for a dictation, whatever the keyboard's reason.
case openedHex
}
static let handedToHexKey = "keyboard.handedDictationToHex"
static let handedToHexAtKey = "keyboard.handedDictationToHexAt"
/// How long one dictation the keyboard handed to Hex keeps the warm mic's default, unless
/// the keyboard dictates by itself sooner: a week, as memory runs short with how the iPhone
/// is used (older 4 GB models, many apps open), which comes back from one day to the next.
static let recentWindow: TimeInterval = 7 * 24 * 60 * 60
static let firstVersion = 26
private let defaults: UserDefaults?
init(defaults: UserDefaults? = UserDefaults(suiteName: KeyboardBridge.appGroup)) {
self.defaults = defaults
}
/// Written by the keyboard as Dictate opens Hex because it can't dictate by itself.
func recordHandedToHex(_ reason: Reason, at now: Date = Date()) {
defaults?.set(reason.rawValue, forKey: Self.handedToHexKey)
defaults?.set(now.timeIntervalSince1970, forKey: Self.handedToHexAtKey)
}
/// Written by Hex as the keyboard opens it for a dictation, for any reason (one the keyboard
/// didn't record, such as a dictation it couldn't start), keeping the keyboard's reason.
func recordOpenedHex(at now: Date = Date()) {
recordHandedToHex(handedToHex?.reason ?? .openedHex, at: now)
}
/// The keyboard finished a dictation by itself again.
func clearHandedToHex() {
guard defaults?.object(forKey: Self.handedToHexKey) != nil else { return }
defaults?.removeObject(forKey: Self.handedToHexKey)
defaults?.removeObject(forKey: Self.handedToHexAtKey)
}
var handedToHex: (reason: Reason, at: Date)? {
guard let raw = defaults?.string(forKey: Self.handedToHexKey), let reason = Reason(rawValue: raw),
let at = defaults?.object(forKey: Self.handedToHexAtKey) as? Double else { return nil }
return (reason, Date(timeIntervalSince1970: at))
}
func dictatesItself(
osMajorVersion: Int = ProcessInfo.processInfo.operatingSystemVersion.majorVersion,
now: Date = Date()
) -> Bool {
guard osMajorVersion >= Self.firstVersion else { return false }
if let handed = handedToHex, now.timeIntervalSince(handed.at) < Self.recentWindow { return false }
let languages = LanguageSettings(defaults: defaults)
if languages.keyboardNeedsHex(now: now) { return false }
// Speech model in keyboard (on unless turned off) dictates the multilingual model's
// language in the keyboard only once the keyboard has that model loaded; until then
// Dictate opens Hex for it.
let multilingual = MultilingualParakeetSettings(defaults: defaults)
let language = multilingual.language(set: languages.dictationLanguage, keyboard: languages.keyboardDictationLanguage())
return !multilingual.keyboardOpensHex(language: language)
|| KeyboardParakeetSettings(defaults: defaults).keyboardHasMultilingualModel
}
}