hex/ios/HEXMobile/TranslationSessionHost.swift
Ryan Vogel fbcec37464
iOS: Apple's on-device recognizer for older iPhones and languages Parakeet can't do, and iOS 18 translation (#2190)
* iOS: older iPhones and other languages dictate with Apple's on-device recognizer before iOS 26

TestFlight build 56, iPhone XR (3 GB, iOS 18.7.12), sent twice: setup fails every
time while loading the speech model, the app closes, then shows a low-memory
message; restarting and closing apps didn't help. The tester mainly needs Chinese,
which iOS 18 couldn't dictate at all: Parakeet V2 is English only and SpeechAnalyzer
needs iOS 26 (the iPhone XR can't get it).

Measured (iOS 26.2 simulator, iPhone SE 2nd generation profile, Parakeet forced,
phys_footprint sampled every 20 ms plus footprint/vmmap):
- Parakeet's encoder is 425 MB on disk, 6-bit palettized (294 constexpr_lut_to_dense
  ops); Core ML's compiled program for it is 1.19 GB of 16-bit weights
  (e5bundlecache bnns_program.bnnsir, 1,186,681,184 bytes).
- In the simulator that program is mapped clean from disk, so Hex's own dirty memory
  stays small: first run peak 103 MB during the encoder load and compile (before
  #2171 77 MB), 90 MB holding the models, 156 MB during a first transcription; the
  pre-#2171 double load compiled once (1.6 s) and reloaded from cache (60 ms).
- The simulator runs the models on the CPU, so the jetsam number that matters is
  device-only: an iPhone's Neural Engine before the A17 holds the unpacked 1.19 GB,
  which with Hex's own ~100 MB is about a 3 GB iPhone's whole foreground limit
  (1.4-1.8 GB), and twice that while build 56 loaded every model twice.

Now (SpeechEngineChoice):
- iOS 26+: unchanged, Apple Speech or Parakeet, never the recognizer, so iOS 26
  never asks for Speech Recognition (App Review notes). Parakeet beside Apple
  Speech also needs no set-aside load and room to load.
- Before iOS 26: Parakeet only for English on an iPhone over 3 GB with no
  interrupted load and os_proc_available_memory at least 1.4 GB (0, the simulator,
  counts as unknown). Everything else goes to OnDeviceSpeechTranscriber:
  SFSpeechRecognizer with requiresOnDeviceRecognition, punctuation, the custom
  vocabulary as contextual strings, in the language the recognizer has on device
  (zh-Hans-CN -> zh-CN, zh-Hant -> zh-TW, en -> en-US; OnDeviceSpeechLanguages),
  else Automatic moves on to the next language as on iOS 26.
- Speech Recognition is asked for only as that engine gets ready (onboarding's
  first page, Set up on home, or a keyboard handoff), after progress copy that says
  it stays on the iPhone. A launch prepares it by itself only once it's allowed,
  and then marks the model installed so the keyboard's handoff starts recording.
- Parakeet checks os_proc_available_memory right before its load; short of 1.4 GB
  it sets itself aside (ParakeetLoadGuard.setAside, like an interrupted load) and
  throws notEnoughMemory, and prepareModel moves on to Apple's engine with a
  notice instead of letting iOS end Hex. A launch after an interrupted load picks
  the recognizer (before iOS 26) instead of failing with "Hex closed while loading
  the speech model": no jetsam loop.
- Onboarding says Preparing rather than Downloading for the recognizer, and its
  wait page's note says speech recognition rather than the model download.
- Settings > Dictation language shows before iOS 26 too, listing the recognizer's
  on-device languages, with where iOS adds them.
- defaultTranscriber and appleSpeechLanguage moved from DictationController.swift
  (1,499 of 1,500 lines) to DictationController+Engine.swift with the launch logic.
- zh-Hans for every new string.

Verified (with the next commit, on origin/dev acf0f2b7): xcodebuild test on an
iPhone SE (2nd generation) iOS 26.2 simulator, 546 tests, 0 failures, 1 skipped
(AppleSpeechClipTests, as before), including the new DictationControllerTests cases
for the choice by OS, memory and language, the fallback after an interrupted or
memory-short load, the permission asked only as the recognizer prepares, iOS 26's
default engine never being the recognizer, and the locale matching. A probe in that
simulator showed the XR choice building OnDeviceSpeechTranscriber and the recognizer
reporting no on-device language there (supportsOnDeviceRecognition false for all
63), so transcription with it, the permission prompt and the XR's memory are
device-only. The iOS 18 onboarding and Settings copy rendered in zh-Hans in a
simulator build that forced those branches.

* iOS: Translate what I say translates before iOS 26 through a session in Hex's window

Before iOS 26 DictationTranslator skipped every translation, while Settings let
Translate what I say be turned on and its language downloaded, so on iOS 18 every
dictation went in as spoken without a word. The TestFlight tester on an iPhone XR
(iOS 18.7.12, build 56) needs it with Chinese.

Apple's Translation framework is there on iOS 18 (LanguageAvailability,
TranslationSession.Configuration, prepareTranslation), but a TranslationSession
comes only from SwiftUI's translationTask; TranslationSession(installedSource:
target:), which DictationTranslator uses, is iOS 26.

- TranslationSessionHost: Hex's root view hosts a translationTask; each
  translation queues its text, sets or invalidates the configuration for its pair,
  and the task translates the queue with the session it gets. TranslationEngine
  .hosted is set as the view appears, and DictationTranslator uses it before iOS 26
  (the keyboard has none and doesn't translate there).
- iOS may not run the task, or the translation, while Hex is in the background (a
  dictation finished from the keyboard). A translation not answered within 10 s
  goes in as spoken with the new reason needsHexOpen, not retried; the keyboard's
  note says "iOS 18 translates only with Hex open" and Settings says the last
  dictation couldn't be translated while Hex was in the background.
- zh-Hans for both strings.

Verified (on origin/dev acf0f2b7): xcodebuild test on an iPhone SE (2nd
generation) iOS 26.2 simulator, 546 tests, 0 failures, 1 skipped, with
DictationControllerTests.testBeforeIOS26ATranslationWithoutHexsWindowSaysSo for the
reason, its copy and no retry. The simulator says Translation isn't supported on it,
so a hosted session translating, in the foreground and in the background, is
device-only.
2026-11-21 08:49:11 -05:00

82 lines
3.4 KiB
Swift

import SwiftUI
@preconcurrency import Translation
/// Translate what I say before iOS 26. There a `TranslationSession` comes only from SwiftUI's
/// `translationTask` (iOS 26 makes one directly, `TranslationSession(installedSource:target:)`),
/// so Hex's root view hosts one (`TranslationSessionHostView`) and each translation asks it for
/// a session for its pair. iOS may not run the task, or the translation, while Hex is in the
/// background (a dictation finished from the keyboard): one that isn't done within `wait` goes
/// in as spoken, and Settings says why (`TranslationFailure.Reason.needsHexOpen`).
@MainActor
@Observable
final class TranslationSessionHost {
static let shared = TranslationSessionHost()
static let wait: Duration = .seconds(10)
private(set) var configuration: TranslationSession.Configuration?
/// Texts waiting for a session, in order.
@ObservationIgnored private var queue: [(id: UUID, text: String)] = []
/// Every translation not answered yet, queued or in a session.
@ObservationIgnored private var waiting: [UUID: CheckedContinuation<String, Error>] = [:]
var engine: TranslationEngine {
TranslationEngine(
status: { source, target in await LanguageAvailability().status(from: source, to: target) },
translate: { text, source, target in
try await TranslationSessionHost.shared.translate(text, from: source, to: target)
}
)
}
func translate(_ text: String, from source: Locale.Language, to target: Locale.Language) async throws -> String {
let id = UUID()
return try await withCheckedThrowingContinuation { continuation in
waiting[id] = continuation
queue.append((id, text))
if configuration?.source == source, configuration?.target == target {
// Same pair: a new run of the task, with a new session.
configuration?.invalidate()
} else {
configuration = TranslationSession.Configuration(source: source, target: target)
}
Task { @MainActor in
try? await Task.sleep(for: Self.wait)
self.answer(id, with: .failure(TranslationEngine.NoSession()))
}
}
}
/// The view's task: translates whatever waits, then returns, as a session can't be used
/// after its task ends.
func serve(_ session: TranslationSession) async {
while !queue.isEmpty {
let (id, text) = queue.removeFirst()
guard waiting[id] != nil else { continue }
do {
answer(id, with: .success(try await session.translate(text).targetText))
} catch {
answer(id, with: .failure(error))
}
}
}
private func answer(_ id: UUID, with result: Result<String, Error>) {
waiting.removeValue(forKey: id)?.resume(with: result)
}
}
/// Hosts `TranslationSessionHost`'s session in Hex's window, before iOS 26.
struct TranslationSessionHostView: View {
private let host = TranslationSessionHost.shared
var body: some View {
Color.clear
.frame(width: 0, height: 0)
.accessibilityHidden(true)
.translationTask(host.configuration) { session in
await host.serve(session)
}
.onAppear { TranslationEngine.hosted = host.engine }
}
}