hex/ios/HEXShared/SpeechModelStore.swift
Ryan Vogel 90a10a5222
iOS: Experiments › Speech model in keyboard: Parakeet in the keyboard, with the app closed (off by default) (#2381)
What changed
- New experiment, off by default (KeyboardParakeetSettings, App Group): the keyboard's own
  dictation transcribes with Parakeet on the Neural Engine in the keyboard's process
  (HEXKeyboard/KeyboardParakeet.swift): V2 for English, Ultra for a language the Multilingual
  model experiment dictates once its files are there (Dictate and the Hex key then stop opening
  Hex for it, once the keyboard has it loaded). Everything after transcription (styles, cleanup,
  punctuation, insertion, held results) is the same code path; KeyboardSpeech is unchanged and is
  what runs whenever Parakeet can't.
- The keyboard loads the model after its first frame and launch work, off the main thread
  (preprocessor on the CPU, the rest on the CPU and Neural Engine, one chunk at a time, never the
  GPU), keeps it while its process lives and releases it on a memory warning. It falls back to
  Apple's speech recognition, silently, when the model is missing, still loading (the first
  compile), not on the Neural Engine (neural ledger, else Core ML's compute plan), short of
  memory (KeyboardParakeetBudget: limit = footprint + os_proc_available_memory must be 96 MB for
  V2 / 115 MB for Ultra, with headroom to load and to transcribe), busy, empty or failing.
- Parakeet's files move from the app's Application Support into the App Group
  (SpeechModelStore, Library/Application Support/SpeechModels), renamed in place at launch
  (moveLegacyDownloads, partial downloads included) instead of downloaded again; folders are
  marked complete (.hex-complete) and the keyboard loads only a marked one. ParakeetTranscriber
  and ParakeetUltraModel point there; turning the multilingual model off still deletes it (and
  any old copy).
- Turning the switch on downloads Parakeet V2 (464 MB, KeyboardParakeetDownload, FluidAudio's
  resumable download); off stops and deletes it unless Hex itself dictates with V2. The row says
  what is happening in a few words (Downloading the speech model, Downloaded. The keyboard gets
  it ready the next time it opens, Getting ready in the keyboard, Ready in the keyboard, or that
  the keyboard can't use it now), with Simplified Chinese; the keyboard writes its state to the
  App Group, and "getting ready" shows only while fresh (3 min).
- Each keyboard dictation with the switch on logs its engine, the keyboard's limit, footprint
  before and at peak, load and transcription time (keyboard-capture-probe.log / os_log).
- FluidAudio is linked into the keyboard target. README updated.

Why
- Ryan: a better model than Apple's should run from the keyboard with the Hex app not running.
  The probe (~/dev/experiments/2026-11-22-hex-keyboard-asr/native) showed it fits: Neural Engine
  memory is outside the keyboard's footprint, the limit on an iPhone 17 Pro is 177 MB.

Verified
- Studio simulator (iPhone 17 Pro, iOS 27.2, fresh): xcodebuild test, 856 tests, 0 failures,
  3 skipped (as on dev). scripts/code-shape.py passes.
- iPhone 17 Pro, iOS 27.2, Release, Hex app force-quit: switch on, 464 MB downloaded, first
  load in Hex's keyboard 15.4 s (compiling), then a dictation from the Mac's speakers:
  engine=parakeet-v2 limit=177 MB footprint before=56.1 MB peak=72.0 MB transcribe=0.28 s for
  9.4 s of audio, neural 468 MB, text exact. Switch off: the same clip with Apple's speech
  recognition, 0.34 s, footprint as on dev; the model's files deleted.
- Switch off against dev (12+12 cold keyboard launches, Release): first appearance unchanged
  (median 491 vs 479 ms), footprint at launch work +0.2 MB (18.3 -> 18.5 MB median), keyboard
  binary 5.92 -> 12.64 MB. The 0.2 MB is FluidAudio's writable data (164 KB __DATA + 49 KB
  __DATA_CONST) that dyld fixes up at every launch; accepted for the experiment (follow-up: load
  the Parakeet code from a framework only when the switch is on).
2026-11-22 10:44:16 -05:00

50 lines
2.6 KiB
Swift
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import Foundation
/// Where Hex keeps Parakeet's Core ML files: in the App Group, so the keyboard can load them too
/// (Experiments › Speech model in keyboard). Before, they were in the app's own Application
/// Support, which the keyboard can't read; the app moves them here (`moveLegacyDownloads`),
/// never downloading them again. Each folder is marked complete once every file is on disk,
/// and the keyboard loads only a marked folder.
nonisolated enum SpeechModelStore {
/// FluidAudio's `Repo.parakeetV2.folderName`: its loader finds the files by this name.
static let parakeetV2Folder = "parakeet-tdt-0.6b-v2"
/// `ParakeetUltraModel`'s folder, named for its pinned revision.
static let parakeetUltraFolder = "parakeet-ultra-coreml-95eaa59"
static let completeMarker = ".hex-complete"
/// `Library/Application Support/SpeechModels` in the App Group (where devicectl can list it
/// too, which a group container's root isn't); the process's own Application Support where
/// there is no App Group (an unsigned simulator build).
static var root: URL {
let base = FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: KeyboardBridge.appGroup)?
.appendingPathComponent("Library/Application Support", isDirectory: true)
?? FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask)[0]
return base.appendingPathComponent("SpeechModels", isDirectory: true)
}
static var parakeetV2: URL { root.appendingPathComponent(parakeetV2Folder, isDirectory: true) }
static var parakeetUltra: URL { root.appendingPathComponent(parakeetUltraFolder, isDirectory: true) }
/// Every file of the model in `directory` is on disk.
static func isComplete(_ directory: URL) -> Bool {
FileManager.default.fileExists(atPath: directory.appendingPathComponent(completeMarker).path)
}
static func markComplete(_ directory: URL) {
guard FileManager.default.fileExists(atPath: directory.path) else { return }
FileManager.default.createFile(atPath: directory.appendingPathComponent(completeMarker).path, contents: Data())
}
static func clearComplete(_ directory: URL) {
try? FileManager.default.removeItem(at: directory.appendingPathComponent(completeMarker))
}
/// Creates `root`, out of backups (the models download again).
static func prepareRoot() {
var folder = root
try? FileManager.default.createDirectory(at: folder, withIntermediateDirectories: true)
var values = URLResourceValues()
values.isExcludedFromBackup = true
try? folder.setResourceValues(values)
}
}