mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
TestFlight build 129 (iPhone 17 Pro, iOS 27.3): Dictate opened Hex every time, and back
in the chat app the Hex key said "Another app is using audio. Try again."
Cause, reproduced on an iPhone 17 Pro on iOS 27.2 with build 128's code (84915a9f,
before #2393) as well as dev: a warm session's dictation fails as soon as Hex is in the
background. beginCapture calls configureAudioSession, which set the category again even
though it was unchanged (the keepalive's .playAndRecord/.measurement/same options). On iOS
27 that asks to interrupt others, which audiomxd refuses for a background app that isn't
Now Playing ("can't stop others because it is in the background", "BeginInterruption
returned an error"), so setCategory throws cannotInterruptOthers ('!int'), which Hex shows
as "Another app is using audio". The keepalive restart after a capture fails the same way
in the background, so the warm session silently ends. #2393 didn't cause this, but its
Off default meant a keyboard that opened Hex for a reason it left no record of got a one
shot each time and no warm session.
What changed
- AudioBackend.configureAudioSession sets the category only when category, mode or
options differ (AudioSessionControlling now exposes category, mode and categoryOptions,
which AVAudioSession already has). A Bluetooth choice, which needs HFP, still sets it.
- Hex records every keyboard launch for a dictation as "the keyboard didn't dictate by
itself" (KeyboardSelfDictation.recordOpenedHex, reason openedHex, keeping a reason the
keyboard wrote), before reading the warm default, so the 15-minute default, the row and
a warm cold start come back from that launch whatever the keyboard's reason was.
- The record is cleared when the keyboard delivers a dictation it made itself (insertResult
with source "keyboard"), not when one merely starts (replaced: the clear in the
onLocalDictateTap closure).
- Removed: "Off where the keyboard dictates by itself: a launch that missed its recording
warms nothing" in activateWarmSessionAndRunHandoff. Every keyboard launch now records,
so it could no longer apply; a late handoff warms as it did before 129.
- README's warm-session paragraph says so.
Verified
- iPhone 17 Pro, iOS 27.2 (UDID 00008150-00163C5C0A47801C), builds signed for the device:
- build 128 code: Keep mic warm 15m, then the Hex key in another app shows "Another app
is using audio. Try again." (the report's screenshot); the next tap dictates in the
keyboard.
- this branch: a keyboard cold start (hex-dictation://keyboard/record) records, Done in
the keyboard keeps the warm session (Hex key orange), the next Hex key records through
Hex in the background with no error, and the warm session stays after it. The App Group
then holds keyboard.handedDictationToHex = openedHex and Settings shows Keep mic warm
at 15m.
- control: with Hex in the back and no warm session, the keyboard dictates by itself,
also with Safari playing audio.
- xcodebuild test on an iOS 27.2 simulator: 878 tests, 3 skipped, 0 failures. New:
AudioSessionSetupTests.testTheCategoryIsSetOnlyWhenItChanges; updated
testKeepMicWarmFollowsWhetherTheKeyboardDictatesItself (a late handoff records and warms)
and KeyboardBridgeTests (recordOpenedHex keeps the keyboard's reason).
- Not checked: iOS 27.3 itself, iOS 26, and a real phone whose keyboard can't dictate by
itself (memory, missing model); a keyboard-made dictation clearing the record was only
unit-tested.
- python3 scripts/code-shape.py passes; git diff --check clean.
357 lines
No EOL
13 KiB
Swift
357 lines
No EOL
13 KiB
Swift
import AVFAudio
|
||
import Foundation
|
||
|
||
nonisolated enum AudioSystemEvent: Equatable, Sendable {
|
||
case interruptionBegan
|
||
case interruptionEnded(shouldResume: Bool)
|
||
case routeOldDeviceUnavailable
|
||
case routeNewDeviceAvailable
|
||
case mediaServicesReset
|
||
}
|
||
|
||
@MainActor
|
||
protocol ActiveAudioCapture: AnyObject {
|
||
var url: URL { get }
|
||
var isRecording: Bool { get }
|
||
var currentTime: TimeInterval { get }
|
||
func normalizedPowerLevel() -> Float
|
||
func averagePowerDB() -> Float
|
||
func stop()
|
||
}
|
||
|
||
extension ActiveAudioCapture {
|
||
func averagePowerDB() -> Float {
|
||
DotScopeSimulation.loudness(fromNormalized: normalizedPowerLevel())
|
||
}
|
||
}
|
||
|
||
@MainActor
|
||
protocol AudioManaging: AnyObject {
|
||
var isKeepaliveRunning: Bool { get }
|
||
var onSystemEvent: ((AudioSystemEvent) -> Void)? { get set }
|
||
var onUnexpectedCaptureEnd: ((URL, Bool) -> Void)? { get set }
|
||
|
||
func requestPermission() async -> Bool
|
||
/// Holds the iPhone's microphone between dictations (`MicrophoneRoute.keepalive`).
|
||
func startKeepalive() throws
|
||
func stopKeepalive()
|
||
func deactivateSession()
|
||
/// Records from `microphone`'s input for this route (`MicrophoneRoute.capture`).
|
||
func beginCapture(at url: URL, maxDuration: TimeInterval, microphone: MicrophoneChoice) throws -> any ActiveAudioCapture
|
||
}
|
||
|
||
/// Keeps AVAudioRecorders that stopped or failed to start alive until their audio queue is
|
||
/// done calling back.
|
||
///
|
||
/// A recorder's AudioQueue input callback reads the recorder (a weak reference inside it) on
|
||
/// the queue's own thread. When the queue fails or stops from inside that callback (no
|
||
/// microphone, a lost route), it is disposed asynchronously and callbacks it already fetched
|
||
/// are still delivered. Freeing the recorder in that window crashed Hex in `objc_loadWeak`
|
||
/// (`AudioRecorderAQInputCallback`) when the microphone failed during the keyboard's switch to
|
||
/// Hex. Nothing reports when the last callback has run, so a retired recorder is held for
|
||
/// `holdTime`: the queue takes about a millisecond to wind down, and a stopped recorder costs
|
||
/// next to nothing to keep.
|
||
@MainActor
|
||
final class RetiredRecorders {
|
||
static let shared = RetiredRecorders()
|
||
static let holdTime: Duration = .seconds(5)
|
||
|
||
private var held: [UInt64: AnyObject] = [:]
|
||
private var nextToken: UInt64 = 0
|
||
|
||
/// Recorders still held.
|
||
var count: Int { held.count }
|
||
|
||
func hold(_ recorder: AnyObject, for duration: Duration = holdTime) {
|
||
nextToken &+= 1
|
||
let token = nextToken
|
||
held[token] = recorder
|
||
Task { [weak self] in
|
||
try? await Task.sleep(for: duration)
|
||
self?.held[token] = nil
|
||
}
|
||
}
|
||
}
|
||
|
||
@MainActor
|
||
private final class RecorderCaptureAdapter: NSObject, ActiveAudioCapture, AVAudioRecorderDelegate {
|
||
let url: URL
|
||
private let recorder: AVAudioRecorder
|
||
private let onFinish: @MainActor (URL, Bool) -> Void
|
||
private var explicitlyStopped = false
|
||
|
||
init(
|
||
url: URL,
|
||
recorder: AVAudioRecorder,
|
||
onFinish: @escaping @MainActor (URL, Bool) -> Void
|
||
) {
|
||
self.url = url
|
||
self.recorder = recorder
|
||
self.onFinish = onFinish
|
||
super.init()
|
||
recorder.delegate = self
|
||
}
|
||
|
||
var isRecording: Bool {
|
||
recorder.isRecording
|
||
}
|
||
|
||
var currentTime: TimeInterval {
|
||
recorder.currentTime
|
||
}
|
||
|
||
func averagePowerDB() -> Float {
|
||
recorder.updateMeters()
|
||
return recorder.averagePower(forChannel: 0)
|
||
}
|
||
|
||
func normalizedPowerLevel() -> Float {
|
||
let decibels = averagePowerDB()
|
||
return max(0, min(1, (decibels + 48) / 48))
|
||
}
|
||
|
||
func stop() {
|
||
guard !explicitlyStopped else { return }
|
||
explicitlyStopped = true
|
||
recorder.delegate = nil
|
||
recorder.stop()
|
||
// The controller drops this capture next; its queue may still be calling back.
|
||
RetiredRecorders.shared.hold(recorder)
|
||
}
|
||
|
||
nonisolated func audioRecorderDidFinishRecording(
|
||
_ recorder: AVAudioRecorder,
|
||
successfully flag: Bool
|
||
) {
|
||
Task { @MainActor [weak self] in
|
||
guard let self, !self.explicitlyStopped else { return }
|
||
self.onFinish(self.url, flag)
|
||
}
|
||
}
|
||
}
|
||
|
||
@MainActor
|
||
final class SystemAudioManager: AudioManaging {
|
||
private let session: any AudioSessionControlling
|
||
private var engine = AVAudioEngine()
|
||
private var tapInstalled = false
|
||
nonisolated(unsafe) private var observers: [NSObjectProtocol] = []
|
||
|
||
var onSystemEvent: ((AudioSystemEvent) -> Void)?
|
||
var onUnexpectedCaptureEnd: ((URL, Bool) -> Void)?
|
||
|
||
var isKeepaliveRunning: Bool {
|
||
engine.isRunning
|
||
}
|
||
|
||
init(session: any AudioSessionControlling = AVAudioSession.sharedInstance()) {
|
||
self.session = session
|
||
installNotifications()
|
||
}
|
||
|
||
deinit {
|
||
for token in observers {
|
||
NotificationCenter.default.removeObserver(token)
|
||
}
|
||
}
|
||
|
||
func requestPermission() async -> Bool {
|
||
await withCheckedContinuation { continuation in
|
||
AVAudioApplication.requestRecordPermission { granted in
|
||
continuation.resume(returning: granted)
|
||
}
|
||
}
|
||
}
|
||
|
||
func startKeepalive() throws {
|
||
stopKeepalive()
|
||
try configureAudioSession(recordingFrom: nil)
|
||
|
||
// Changing the session (a dictation's own input, or a route change) can move the
|
||
// hardware to another sample rate. An engine built before that keeps the old input
|
||
// format, and installTap raises an Objective-C exception Swift can't catch, so build a
|
||
// fresh engine for the current route and tap with the format the hardware reports.
|
||
engine = AVAudioEngine()
|
||
let input = engine.inputNode
|
||
let format = input.inputFormat(forBus: 0)
|
||
guard format.sampleRate > 0, format.channelCount > 0,
|
||
input.outputFormat(forBus: 0).sampleRate == format.sampleRate
|
||
else {
|
||
throw CocoaError(.coderInvalidValue)
|
||
}
|
||
|
||
input.installTap(
|
||
onBus: 0,
|
||
bufferSize: 4_096,
|
||
format: format,
|
||
block: Self.discardSamples
|
||
)
|
||
tapInstalled = true
|
||
engine.prepare()
|
||
do {
|
||
try engine.start()
|
||
} catch {
|
||
// Leave nothing half-installed: no tap on an engine that never ran.
|
||
stopKeepalive()
|
||
throw error
|
||
}
|
||
}
|
||
|
||
func stopKeepalive() {
|
||
if tapInstalled {
|
||
engine.inputNode.removeTap(onBus: 0)
|
||
tapInstalled = false
|
||
}
|
||
if engine.isRunning {
|
||
engine.stop()
|
||
}
|
||
engine.reset()
|
||
}
|
||
|
||
func deactivateSession() {
|
||
stopKeepalive()
|
||
try? session.setActive(false, options: .notifyOthersOnDeactivation)
|
||
}
|
||
|
||
func beginCapture(
|
||
at url: URL,
|
||
maxDuration: TimeInterval,
|
||
microphone: MicrophoneChoice
|
||
) throws -> any ActiveAudioCapture {
|
||
stopKeepalive()
|
||
try configureAudioSession(recordingFrom: microphone)
|
||
|
||
let settings: [String: Any] = [
|
||
AVFormatIDKey: kAudioFormatLinearPCM,
|
||
AVSampleRateKey: 16_000,
|
||
AVNumberOfChannelsKey: 1,
|
||
AVLinearPCMBitDepthKey: 16,
|
||
AVLinearPCMIsBigEndianKey: false,
|
||
AVLinearPCMIsFloatKey: false,
|
||
]
|
||
let recorder = try AVAudioRecorder(url: url, settings: settings)
|
||
recorder.isMeteringEnabled = true
|
||
recorder.prepareToRecord()
|
||
guard recorder.record(forDuration: maxDuration) else {
|
||
// The microphone didn't start (none in the simulator, or one another app holds).
|
||
// The recorder's queue made it this far and is still winding down: freeing the
|
||
// recorder here crashed Hex on the queue's thread.
|
||
recorder.stop()
|
||
RetiredRecorders.shared.hold(recorder)
|
||
throw CocoaError(.fileWriteUnknown)
|
||
}
|
||
|
||
return RecorderCaptureAdapter(url: url, recorder: recorder) { [weak self] finishedURL, ok in
|
||
self?.onUnexpectedCaptureEnd?(finishedURL, ok)
|
||
}
|
||
}
|
||
|
||
/// Captures and the warm keepalive record alongside other audio: `.mixWithOthers` never
|
||
/// interrupts another app, so its video or music keeps playing (over A2DP in a car or
|
||
/// headphones) with nothing to resume, and no `.duckOthers`.
|
||
///
|
||
/// A dictation records from Settings › Microphone's input for this route
|
||
/// (`MicrophoneRoute.capture`): with Automatic (the default) a wired headset's
|
||
/// microphone, the car's in CarPlay, else the iPhone's; Bluetooth hands-free microphones
|
||
/// only with Bluetooth, the one choice that adds `.allowBluetoothHFP`. The keepalive
|
||
/// (`choice` nil) stays on the iPhone's microphone without HFP, so a warm session never
|
||
/// holds a car or headset in call audio between dictations. The input is always set,
|
||
/// never left to an earlier preference or to iOS, which routes HFP and CarPlay first.
|
||
///
|
||
/// No preferred sample rate: `.playAndRecord` runs the microphone and the output at one
|
||
/// hardware rate, so asking for 16 kHz moved everything the iPhone played and recorded (a
|
||
/// screen recording's audio too) to 16 kHz for as long as Hex held the session, and back
|
||
/// at its end. The hardware keeps its own rate: the recorder writes 16 kHz itself, and the
|
||
/// keepalive taps whatever format the hardware reports.
|
||
///
|
||
/// The category is set only when it changes: on iOS 27 setting it again, even unchanged,
|
||
/// while Hex is in the background asks iOS to interrupt others, which it refuses there
|
||
/// ("can't stop others because it is in the background"), so a warm session's dictation
|
||
/// failed with "Another app is using audio" (cannotInterruptOthers, TestFlight build 129).
|
||
func configureAudioSession(recordingFrom choice: MicrophoneChoice?) throws {
|
||
let options = MicrophoneRoute.categoryOptions(allowingBluetooth: choice?.allowsBluetooth ?? false)
|
||
if session.category != .playAndRecord || session.mode != .measurement || session.categoryOptions != options {
|
||
try session.setCategory(.playAndRecord, mode: .measurement, options: options)
|
||
}
|
||
try session.setActive(true, options: [])
|
||
let inputs = session.microphones
|
||
let input = choice.map { MicrophoneRoute.capture($0, from: inputs) } ?? MicrophoneRoute.keepalive(from: inputs)
|
||
try? session.prefer(input)
|
||
}
|
||
|
||
nonisolated private static func discardSamples(
|
||
_ buffer: AVAudioPCMBuffer,
|
||
at time: AVAudioTime
|
||
) {}
|
||
|
||
private func installNotifications() {
|
||
let center = NotificationCenter.default
|
||
|
||
observers.append(
|
||
center.addObserver(
|
||
forName: AVAudioSession.interruptionNotification,
|
||
object: AVAudioSession.sharedInstance(),
|
||
queue: .main
|
||
) { [weak self] notification in
|
||
let typeRaw = notification.userInfo?[AVAudioSessionInterruptionTypeKey] as? UInt
|
||
let optionsRaw = notification.userInfo?[AVAudioSessionInterruptionOptionKey] as? UInt
|
||
MainActor.assumeIsolated {
|
||
guard let self,
|
||
let typeRaw,
|
||
let type = AVAudioSession.InterruptionType(rawValue: typeRaw) else {
|
||
return
|
||
}
|
||
switch type {
|
||
case .began:
|
||
self.onSystemEvent?(.interruptionBegan)
|
||
case .ended:
|
||
let options = AVAudioSession.InterruptionOptions(rawValue: optionsRaw ?? 0)
|
||
self.onSystemEvent?(.interruptionEnded(shouldResume: options.contains(.shouldResume)))
|
||
@unknown default:
|
||
break
|
||
}
|
||
}
|
||
}
|
||
)
|
||
|
||
observers.append(
|
||
center.addObserver(
|
||
forName: AVAudioSession.routeChangeNotification,
|
||
object: AVAudioSession.sharedInstance(),
|
||
queue: .main
|
||
) { [weak self] notification in
|
||
let reasonRaw = notification.userInfo?[AVAudioSessionRouteChangeReasonKey] as? UInt
|
||
MainActor.assumeIsolated {
|
||
guard let self,
|
||
let reasonRaw,
|
||
let reason = AVAudioSession.RouteChangeReason(rawValue: reasonRaw) else {
|
||
return
|
||
}
|
||
switch reason {
|
||
case .oldDeviceUnavailable:
|
||
self.onSystemEvent?(.routeOldDeviceUnavailable)
|
||
case .newDeviceAvailable:
|
||
self.onSystemEvent?(.routeNewDeviceAvailable)
|
||
default:
|
||
break
|
||
}
|
||
}
|
||
}
|
||
)
|
||
|
||
observers.append(
|
||
center.addObserver(
|
||
forName: AVAudioSession.mediaServicesWereResetNotification,
|
||
object: AVAudioSession.sharedInstance(),
|
||
queue: .main
|
||
) { [weak self] _ in
|
||
MainActor.assumeIsolated {
|
||
guard let self else { return }
|
||
self.tapInstalled = false
|
||
self.engine = AVAudioEngine()
|
||
self.onSystemEvent?(.mediaServicesReset)
|
||
}
|
||
}
|
||
)
|
||
}
|
||
} |