hex/ios/HEXMobile/AudioBackend.swift
Ryan Vogel 62ba0b4984
iOS: a warm session dictates in the background on iOS 27 again, and a keyboard that opens Hex brings the warm default back (#2406)
TestFlight build 129 (iPhone 17 Pro, iOS 27.3): Dictate opened Hex every time, and back
in the chat app the Hex key said "Another app is using audio. Try again."

Cause, reproduced on an iPhone 17 Pro on iOS 27.2 with build 128's code (84915a9f,
before #2393) as well as dev: a warm session's dictation fails as soon as Hex is in the
background. beginCapture calls configureAudioSession, which set the category again even
though it was unchanged (the keepalive's .playAndRecord/.measurement/same options). On iOS
27 that asks to interrupt others, which audiomxd refuses for a background app that isn't
Now Playing ("can't stop others because it is in the background", "BeginInterruption
returned an error"), so setCategory throws cannotInterruptOthers ('!int'), which Hex shows
as "Another app is using audio". The keepalive restart after a capture fails the same way
in the background, so the warm session silently ends. #2393 didn't cause this, but its
Off default meant a keyboard that opened Hex for a reason it left no record of got a one
shot each time and no warm session.

What changed
- AudioBackend.configureAudioSession sets the category only when category, mode or
  options differ (AudioSessionControlling now exposes category, mode and categoryOptions,
  which AVAudioSession already has). A Bluetooth choice, which needs HFP, still sets it.
- Hex records every keyboard launch for a dictation as "the keyboard didn't dictate by
  itself" (KeyboardSelfDictation.recordOpenedHex, reason openedHex, keeping a reason the
  keyboard wrote), before reading the warm default, so the 15-minute default, the row and
  a warm cold start come back from that launch whatever the keyboard's reason was.
- The record is cleared when the keyboard delivers a dictation it made itself (insertResult
  with source "keyboard"), not when one merely starts (replaced: the clear in the
  onLocalDictateTap closure).
- Removed: "Off where the keyboard dictates by itself: a launch that missed its recording
  warms nothing" in activateWarmSessionAndRunHandoff. Every keyboard launch now records,
  so it could no longer apply; a late handoff warms as it did before 129.
- README's warm-session paragraph says so.

Verified
- iPhone 17 Pro, iOS 27.2 (UDID 00008150-00163C5C0A47801C), builds signed for the device:
  - build 128 code: Keep mic warm 15m, then the Hex key in another app shows "Another app
    is using audio. Try again." (the report's screenshot); the next tap dictates in the
    keyboard.
  - this branch: a keyboard cold start (hex-dictation://keyboard/record) records, Done in
    the keyboard keeps the warm session (Hex key orange), the next Hex key records through
    Hex in the background with no error, and the warm session stays after it. The App Group
    then holds keyboard.handedDictationToHex = openedHex and Settings shows Keep mic warm
    at 15m.
  - control: with Hex in the back and no warm session, the keyboard dictates by itself,
    also with Safari playing audio.
- xcodebuild test on an iOS 27.2 simulator: 878 tests, 3 skipped, 0 failures. New:
  AudioSessionSetupTests.testTheCategoryIsSetOnlyWhenItChanges; updated
  testKeepMicWarmFollowsWhetherTheKeyboardDictatesItself (a late handoff records and warms)
  and KeyboardBridgeTests (recordOpenedHex keeps the keyboard's reason).
- Not checked: iOS 27.3 itself, iOS 26, and a real phone whose keyboard can't dictate by
  itself (memory, missing model); a keyboard-made dictation clearing the record was only
  unit-tested.
- python3 scripts/code-shape.py passes; git diff --check clean.
2026-11-22 13:12:24 -05:00

357 lines
No EOL
13 KiB
Swift
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import AVFAudio
import Foundation
nonisolated enum AudioSystemEvent: Equatable, Sendable {
case interruptionBegan
case interruptionEnded(shouldResume: Bool)
case routeOldDeviceUnavailable
case routeNewDeviceAvailable
case mediaServicesReset
}
@MainActor
protocol ActiveAudioCapture: AnyObject {
var url: URL { get }
var isRecording: Bool { get }
var currentTime: TimeInterval { get }
func normalizedPowerLevel() -> Float
func averagePowerDB() -> Float
func stop()
}
extension ActiveAudioCapture {
func averagePowerDB() -> Float {
DotScopeSimulation.loudness(fromNormalized: normalizedPowerLevel())
}
}
@MainActor
protocol AudioManaging: AnyObject {
var isKeepaliveRunning: Bool { get }
var onSystemEvent: ((AudioSystemEvent) -> Void)? { get set }
var onUnexpectedCaptureEnd: ((URL, Bool) -> Void)? { get set }
func requestPermission() async -> Bool
/// Holds the iPhone's microphone between dictations (`MicrophoneRoute.keepalive`).
func startKeepalive() throws
func stopKeepalive()
func deactivateSession()
/// Records from `microphone`'s input for this route (`MicrophoneRoute.capture`).
func beginCapture(at url: URL, maxDuration: TimeInterval, microphone: MicrophoneChoice) throws -> any ActiveAudioCapture
}
/// Keeps AVAudioRecorders that stopped or failed to start alive until their audio queue is
/// done calling back.
///
/// A recorder's AudioQueue input callback reads the recorder (a weak reference inside it) on
/// the queue's own thread. When the queue fails or stops from inside that callback (no
/// microphone, a lost route), it is disposed asynchronously and callbacks it already fetched
/// are still delivered. Freeing the recorder in that window crashed Hex in `objc_loadWeak`
/// (`AudioRecorderAQInputCallback`) when the microphone failed during the keyboard's switch to
/// Hex. Nothing reports when the last callback has run, so a retired recorder is held for
/// `holdTime`: the queue takes about a millisecond to wind down, and a stopped recorder costs
/// next to nothing to keep.
@MainActor
final class RetiredRecorders {
static let shared = RetiredRecorders()
static let holdTime: Duration = .seconds(5)
private var held: [UInt64: AnyObject] = [:]
private var nextToken: UInt64 = 0
/// Recorders still held.
var count: Int { held.count }
func hold(_ recorder: AnyObject, for duration: Duration = holdTime) {
nextToken &+= 1
let token = nextToken
held[token] = recorder
Task { [weak self] in
try? await Task.sleep(for: duration)
self?.held[token] = nil
}
}
}
@MainActor
private final class RecorderCaptureAdapter: NSObject, ActiveAudioCapture, AVAudioRecorderDelegate {
let url: URL
private let recorder: AVAudioRecorder
private let onFinish: @MainActor (URL, Bool) -> Void
private var explicitlyStopped = false
init(
url: URL,
recorder: AVAudioRecorder,
onFinish: @escaping @MainActor (URL, Bool) -> Void
) {
self.url = url
self.recorder = recorder
self.onFinish = onFinish
super.init()
recorder.delegate = self
}
var isRecording: Bool {
recorder.isRecording
}
var currentTime: TimeInterval {
recorder.currentTime
}
func averagePowerDB() -> Float {
recorder.updateMeters()
return recorder.averagePower(forChannel: 0)
}
func normalizedPowerLevel() -> Float {
let decibels = averagePowerDB()
return max(0, min(1, (decibels + 48) / 48))
}
func stop() {
guard !explicitlyStopped else { return }
explicitlyStopped = true
recorder.delegate = nil
recorder.stop()
// The controller drops this capture next; its queue may still be calling back.
RetiredRecorders.shared.hold(recorder)
}
nonisolated func audioRecorderDidFinishRecording(
_ recorder: AVAudioRecorder,
successfully flag: Bool
) {
Task { @MainActor [weak self] in
guard let self, !self.explicitlyStopped else { return }
self.onFinish(self.url, flag)
}
}
}
@MainActor
final class SystemAudioManager: AudioManaging {
private let session: any AudioSessionControlling
private var engine = AVAudioEngine()
private var tapInstalled = false
nonisolated(unsafe) private var observers: [NSObjectProtocol] = []
var onSystemEvent: ((AudioSystemEvent) -> Void)?
var onUnexpectedCaptureEnd: ((URL, Bool) -> Void)?
var isKeepaliveRunning: Bool {
engine.isRunning
}
init(session: any AudioSessionControlling = AVAudioSession.sharedInstance()) {
self.session = session
installNotifications()
}
deinit {
for token in observers {
NotificationCenter.default.removeObserver(token)
}
}
func requestPermission() async -> Bool {
await withCheckedContinuation { continuation in
AVAudioApplication.requestRecordPermission { granted in
continuation.resume(returning: granted)
}
}
}
func startKeepalive() throws {
stopKeepalive()
try configureAudioSession(recordingFrom: nil)
// Changing the session (a dictation's own input, or a route change) can move the
// hardware to another sample rate. An engine built before that keeps the old input
// format, and installTap raises an Objective-C exception Swift can't catch, so build a
// fresh engine for the current route and tap with the format the hardware reports.
engine = AVAudioEngine()
let input = engine.inputNode
let format = input.inputFormat(forBus: 0)
guard format.sampleRate > 0, format.channelCount > 0,
input.outputFormat(forBus: 0).sampleRate == format.sampleRate
else {
throw CocoaError(.coderInvalidValue)
}
input.installTap(
onBus: 0,
bufferSize: 4_096,
format: format,
block: Self.discardSamples
)
tapInstalled = true
engine.prepare()
do {
try engine.start()
} catch {
// Leave nothing half-installed: no tap on an engine that never ran.
stopKeepalive()
throw error
}
}
func stopKeepalive() {
if tapInstalled {
engine.inputNode.removeTap(onBus: 0)
tapInstalled = false
}
if engine.isRunning {
engine.stop()
}
engine.reset()
}
func deactivateSession() {
stopKeepalive()
try? session.setActive(false, options: .notifyOthersOnDeactivation)
}
func beginCapture(
at url: URL,
maxDuration: TimeInterval,
microphone: MicrophoneChoice
) throws -> any ActiveAudioCapture {
stopKeepalive()
try configureAudioSession(recordingFrom: microphone)
let settings: [String: Any] = [
AVFormatIDKey: kAudioFormatLinearPCM,
AVSampleRateKey: 16_000,
AVNumberOfChannelsKey: 1,
AVLinearPCMBitDepthKey: 16,
AVLinearPCMIsBigEndianKey: false,
AVLinearPCMIsFloatKey: false,
]
let recorder = try AVAudioRecorder(url: url, settings: settings)
recorder.isMeteringEnabled = true
recorder.prepareToRecord()
guard recorder.record(forDuration: maxDuration) else {
// The microphone didn't start (none in the simulator, or one another app holds).
// The recorder's queue made it this far and is still winding down: freeing the
// recorder here crashed Hex on the queue's thread.
recorder.stop()
RetiredRecorders.shared.hold(recorder)
throw CocoaError(.fileWriteUnknown)
}
return RecorderCaptureAdapter(url: url, recorder: recorder) { [weak self] finishedURL, ok in
self?.onUnexpectedCaptureEnd?(finishedURL, ok)
}
}
/// Captures and the warm keepalive record alongside other audio: `.mixWithOthers` never
/// interrupts another app, so its video or music keeps playing (over A2DP in a car or
/// headphones) with nothing to resume, and no `.duckOthers`.
///
/// A dictation records from Settings › Microphone's input for this route
/// (`MicrophoneRoute.capture`): with Automatic (the default) a wired headset's
/// microphone, the car's in CarPlay, else the iPhone's; Bluetooth hands-free microphones
/// only with Bluetooth, the one choice that adds `.allowBluetoothHFP`. The keepalive
/// (`choice` nil) stays on the iPhone's microphone without HFP, so a warm session never
/// holds a car or headset in call audio between dictations. The input is always set,
/// never left to an earlier preference or to iOS, which routes HFP and CarPlay first.
///
/// No preferred sample rate: `.playAndRecord` runs the microphone and the output at one
/// hardware rate, so asking for 16 kHz moved everything the iPhone played and recorded (a
/// screen recording's audio too) to 16 kHz for as long as Hex held the session, and back
/// at its end. The hardware keeps its own rate: the recorder writes 16 kHz itself, and the
/// keepalive taps whatever format the hardware reports.
///
/// The category is set only when it changes: on iOS 27 setting it again, even unchanged,
/// while Hex is in the background asks iOS to interrupt others, which it refuses there
/// ("can't stop others because it is in the background"), so a warm session's dictation
/// failed with "Another app is using audio" (cannotInterruptOthers, TestFlight build 129).
func configureAudioSession(recordingFrom choice: MicrophoneChoice?) throws {
let options = MicrophoneRoute.categoryOptions(allowingBluetooth: choice?.allowsBluetooth ?? false)
if session.category != .playAndRecord || session.mode != .measurement || session.categoryOptions != options {
try session.setCategory(.playAndRecord, mode: .measurement, options: options)
}
try session.setActive(true, options: [])
let inputs = session.microphones
let input = choice.map { MicrophoneRoute.capture($0, from: inputs) } ?? MicrophoneRoute.keepalive(from: inputs)
try? session.prefer(input)
}
nonisolated private static func discardSamples(
_ buffer: AVAudioPCMBuffer,
at time: AVAudioTime
) {}
private func installNotifications() {
let center = NotificationCenter.default
observers.append(
center.addObserver(
forName: AVAudioSession.interruptionNotification,
object: AVAudioSession.sharedInstance(),
queue: .main
) { [weak self] notification in
let typeRaw = notification.userInfo?[AVAudioSessionInterruptionTypeKey] as? UInt
let optionsRaw = notification.userInfo?[AVAudioSessionInterruptionOptionKey] as? UInt
MainActor.assumeIsolated {
guard let self,
let typeRaw,
let type = AVAudioSession.InterruptionType(rawValue: typeRaw) else {
return
}
switch type {
case .began:
self.onSystemEvent?(.interruptionBegan)
case .ended:
let options = AVAudioSession.InterruptionOptions(rawValue: optionsRaw ?? 0)
self.onSystemEvent?(.interruptionEnded(shouldResume: options.contains(.shouldResume)))
@unknown default:
break
}
}
}
)
observers.append(
center.addObserver(
forName: AVAudioSession.routeChangeNotification,
object: AVAudioSession.sharedInstance(),
queue: .main
) { [weak self] notification in
let reasonRaw = notification.userInfo?[AVAudioSessionRouteChangeReasonKey] as? UInt
MainActor.assumeIsolated {
guard let self,
let reasonRaw,
let reason = AVAudioSession.RouteChangeReason(rawValue: reasonRaw) else {
return
}
switch reason {
case .oldDeviceUnavailable:
self.onSystemEvent?(.routeOldDeviceUnavailable)
case .newDeviceAvailable:
self.onSystemEvent?(.routeNewDeviceAvailable)
default:
break
}
}
}
)
observers.append(
center.addObserver(
forName: AVAudioSession.mediaServicesWereResetNotification,
object: AVAudioSession.sharedInstance(),
queue: .main
) { [weak self] _ in
MainActor.assumeIsolated {
guard let self else { return }
self.tapInstalled = false
self.engine = AVAudioEngine()
self.onSystemEvent?(.mediaServicesReset)
}
}
)
}
}