hex/ios/HEXShared/CleanupReconstruction.swift
Ryan Vogel efb7dcafb5
iOS: clean up what I say with an on-device tagger (experiment) (#2023)
* ios: clean up what I say with Hex's on-device dictation tagger (experiment)

What changed
- Experiments › Clean up what I say (off by default, App Group key
  experiments.cleanupTagger): Hex's trained cleanup tagger removes ums,
  repeats, and the words a spoken correction replaces ("3pm, actually 4pm"),
  on the iPhone, in in-keyboard dictation and in app dictation. It runs after
  transcription and learned fixes (in the app also after custom vocabulary),
  before Apple Intelligence cleanup, translation, and formatting, and never
  for Verbatim. When both cleanups are on, Apple Intelligence now receives the
  tagged text instead of the raw transcript; nothing else changes while the
  experiment is off.
- Model: the BERT L-4/H-512 token classifier (ckpt-1200k-l4h512) converted to
  an int8 Core ML ML Program (coremltools 9, fp16 compute, enumerated lengths
  16/32/64/128 plus a length input that masks the padding; the output is
  logit(DELETE) - logit(KEEP)), 28.5 MB, with its WordPiece vocabulary. Both
  ship once, in HEXKeyboard.appex (HEXKeyboard/CleanupTagger/), and the app
  loads them from its embedded keyboard extension. Core ML codegen is off for
  the keyboard target.
- HEXShared: CleanupTokenizer (Hugging Face BertTokenizer: special tokens,
  BertNormalizer, BertPreTokenizer, WordPiece, byte-exact vocabulary keys,
  HF's Unicode 8.0 character classes), CleanupReconstruction (a port of
  js/reconstruct.js with JavaScript's UTF-16 casing), and CleanupTagger
  (chunks of whole words within 128 tokens, P(DELETE) > 0.98, one serial
  queue, 150 ms budget). A timeout, an error, or a result with no words left
  keeps the transcript. The model loads when a dictation starts recording with
  the experiment on, stays loaded, and is freed on a memory warning. It runs
  on the Neural Engine or the CPU, never the GPU.
- Call sites: one each in KeyboardLocalDictation (after DictationCorrections,
  before translation) and DictationController (after VocabularySpelling,
  before the Apple Intelligence pass). prepareIfEnabled runs at recording
  start, in KeyboardViewController and in DictationController.startRecording.

Why
- In the simulator the first load took 0.15 s on the CPU and 0.8 s on the GPU
  path, both over the 150 ms budget. So loading starts when recording starts,
  and the GPU isn't used (it also added 30 MB of footprint, against 3 MB on the
  CPU, which keeps the int8 weights memory-mapped).
- Threshold 0.98, not the exported 0.95: 1/45 false deletes on hand-written
  clean sentences against 3/45, with the same self-correction accuracy.

Verified
- xcodebuild test on the iPhone 18 Pro simulator: 168 tests, 0 failures,
  including the new CleanupTaggerTests and DictationControllerTests
  testCleanupTaggerRunsAfterLearnedFixesAndSkipsVerbatim. After the fixtures
  were rewritten one case per line (same content), CleanupTaggerTests and
  DictationControllerTests passed again (28 tests).
- Parity on the 137 hand-written sentences at 0.98: labels match the fp32
  checkpoint on all 1,128 words, and 136/137 outputs match the ONNX int8
  pipeline. #120 ("The cat keeps trying to scratch that chair.") differs on
  two words that ONNX keeps (P 0.9725/0.9794) and Core ML deletes (0.984).
  The fp32 checkpoint also deletes them (0.9879/0.9853), so the difference
  comes from ONNX's int8 activations. 128/137 outputs match the hand-written
  targets. The tokenizer matches HF on 1,891 words, and reconstruction matches
  reconstruct.js on 670 cases.
- Simulator, CPU path: load 140-161 ms, first sentence after load about 8 ms,
  then 3.3 ms median and 7.5 ms p95 per sentence; phys_footprint +2.9 MB.
- The Release generic iOS build with CODE_SIGNING_ALLOWED=NO succeeds.
  HEXKeyboard.appex is 30.2 MB (model 27.8 MB, vocabulary 0.2 MB).
- Not verified on a physical iPhone: Neural Engine load time, latency, and the
  keyboard extension's footprint next to Apple Speech.

Note: this adds a 28.4 MB binary (weight.bin) to git; the repo has no LFS.

* Fetch the cleanup tagger's Core ML model at build time instead of committing it

The 27 MB mlpackage would have been ~30x the largest file in the repo, kept in history forever. It is now the ios-model-cleanup-tagger-1 release asset; ios/scripts/fetch-cleanup-model.sh downloads it, checks its SHA-256, and skips when the pinned copy is already in place. Xcode Cloud runs it in ci_post_clone.sh before xcodegen; the folder is gitignored.

Verified the fetched package is byte-identical to the one the tests ran against (diff -r), and a second run is a no-op.
2026-11-20 01:29:41 +00:00

181 lines
7.9 KiB
Swift

import Foundation
/// Rebuilds a transcript from the cleanup tagger's KEEP/DELETE labels: joins the kept words
/// and repairs punctuation and capitals left next to a deletion. It only removes characters
/// (commas, hesitation ellipses) or changes the case of a first letter; it never adds,
/// reorders, or changes words, names, or numbers.
///
/// A line-for-line port of the tagger's `js/reconstruct.js`, with JavaScript's string
/// semantics: a word's "first letter" is its first UTF-16 unit (an emoji or other astral
/// character is never cased), while the rest of a word is checked by code point.
nonisolated enum CleanupReconstruction {
private typealias Word = [Unicode.Scalar]
/// Words that never keep a trailing comma when the filler after them is removed:
/// "Can you, um, send" -> "Can you send".
private static let noComma: Set<String> = [
"a", "an", "the", "to", "of", "for", "with", "at", "in", "on", "from", "by", "about", "into", "onto", "my",
"your", "his", "her", "our", "their", "its", "i", "we", "you", "he", "she", "they", "can",
"could", "would", "will", "should", "shall", "may", "might", "must", "do", "does", "did", "is",
"are", "was", "were", "be", "been", "am", "have", "has", "had", "and", "or", "but", "if", "that", "because",
"than", "just", "gonna", "wanna", "want", "need", "think", "thinking", "guess", "get", "got", "let's", "i'm",
"we're", "you're", "they're", "it's", "there's", "i'll", "we'll", "you'll", "i'd", "we'd", "i've", "we've",
"really", "very", "so", "like", "isn't", "don't", "can't", "won't", "didn't", "this", "these", "those", "some",
"any", "every", "what", "where", "when", "how", "which", "who", "also", "go", "going", "s",
]
/// Function words a correction's repair may start with; lowercased when a deleted
/// sentence end capitalized them: "at 5. Wait, no. At 6." -> "at 6."
private static let lowerable: Set<String> = [
"at", "on", "in", "the", "a", "an", "to", "for", "with", "from", "by", "and", "or", "but", "about", "around",
"before", "after", "until", "into", "my", "your", "our", "their", "his", "her", "this", "that", "these", "those",
]
private static let fillerWords: Set<String> = [
"um", "uh", "er", "ah", "hmm", "umm", "uhh", "erm", "uhm", "mm", "like", "you", "know", "i", "mean",
]
private static let trailing: Set<Unicode.Scalar> = [",", ".", "?", "!", ";", ":", "\u{2026}"]
private static let pauses: Set<Unicode.Scalar> = [",", ";", ":"]
private static let sentenceEnds: Set<Unicode.Scalar> = [".", "?", "!"]
/// `words` are the transcript's whitespace-separated words; `deletes` has one flag per word.
static func reconstruct(_ words: [String], deletes: [Bool]) -> String {
var out: [Word] = []
var pending: [Word] = []
for (index, word) in words.enumerated() {
var scalars = Array(word.unicodeScalars)
if index < deletes.count, deletes[index] {
pending.append(scalars)
continue
}
if !pending.isEmpty {
scalars = afterGap(&out, deleted: pending, next: scalars)
pending.removeAll()
}
out.append(scalars)
}
if !pending.isEmpty, !out.isEmpty {
atEnd(&out, deleted: pending)
}
return out.map(string).joined(separator: " ")
}
private static func afterGap(_ out: inout [Word], deleted: [Word], next: Word) -> Word {
var b = next
guard let last = out.last else { return capitalized(b) } // deletion at the very start
var a = stripEllipsis(last) // hesitation "..." before a removed disfluency
let term = terminal(deleted[deleted.count - 1])
if a.last == ",", contains(noComma, bare(a)) {
a.removeLast()
}
if !term.isEmpty, !endsSentence(a), isUpper(firstUnit(b)) {
if contains(lowerable, bare(b)) {
b = lowercasedFirst(b)
} else if deleted.allSatisfy({ contains(fillerWords, bare($0)) }) {
a = trimming(a, pauses) + term // "Thanks, um. See you" -> "Thanks. See you"
}
}
out[out.count - 1] = a
if a.isEmpty || endsSentence(a) {
b = capitalized(b)
}
return b
}
private static func atEnd(_ out: inout [Word], deleted: [Word]) {
var a = stripEllipsis(out[out.count - 1])
if !endsSentence(a) {
a = trimming(a, pauses) + terminal(deleted[deleted.count - 1])
}
out[out.count - 1] = a
}
// MARK: JavaScript string helpers
/// `t[0]`: the first UTF-16 unit. An astral first character is half a surrogate pair
/// in JavaScript, which is neither upper- nor lowercase, so it reads as nil here.
private static func firstUnit(_ t: Word) -> Unicode.Scalar? {
guard let first = t.first, first.value <= 0xFFFF else { return nil }
return first
}
/// `ch === ch.toUpperCase() && ch !== ch.toLowerCase()`
private static func isUpper(_ ch: Unicode.Scalar?) -> Bool {
guard let ch else { return false }
return same(ch.properties.uppercaseMapping, ch) && !same(ch.properties.lowercaseMapping, ch)
}
/// `ch === ch.toLowerCase() && ch !== ch.toUpperCase()`
private static func isLower(_ ch: Unicode.Scalar?) -> Bool {
guard let ch else { return false }
return same(ch.properties.lowercaseMapping, ch) && !same(ch.properties.uppercaseMapping, ch)
}
private static func same(_ mapping: String, _ scalar: Unicode.Scalar) -> Bool {
let scalars = mapping.unicodeScalars
return scalars.count == 1 && scalars.first == scalar
}
private static func endsSentence(_ t: Word) -> Bool {
guard let last = t.last, sentenceEnds.contains(last) else { return false }
return !hasEllipsis(t)
}
private static func hasEllipsis(_ t: Word) -> Bool {
t.count >= 3 && t[t.count - 3] == "." && t[t.count - 2] == "." && t[t.count - 1] == "."
}
/// The sentence-final mark (. ? !) a deleted word carries, ignoring ellipses.
private static func terminal(_ t: Word) -> Word {
endsSentence(t) ? [t[t.count - 1]] : []
}
/// `t.replace(/[,.?!;:…]+$/, "").toLowerCase()`
private static func bare(_ t: Word) -> String {
var end = t.count
while end > 0, trailing.contains(t[end - 1]) {
end -= 1
}
var result = ""
for scalar in t[..<end] {
result += scalar.properties.lowercaseMapping
}
return result
}
/// Every listed word is ASCII, and an ASCII string is canonically equivalent only to
/// itself, so this is JavaScript's exact `Set.has`.
private static func contains(_ set: Set<String>, _ word: String) -> Bool {
word.utf8.allSatisfy { $0 < 0x80 } && set.contains(word)
}
private static func capitalized(_ t: Word) -> Word {
guard let first = firstUnit(t), isLower(first), !t.dropFirst().contains(where: { isUpper($0) }) else {
return t
}
return Array(first.properties.uppercaseMapping.unicodeScalars) + t.dropFirst()
}
private static func lowercasedFirst(_ t: Word) -> Word {
guard let first = firstUnit(t) else { return t }
return Array(first.properties.lowercaseMapping.unicodeScalars) + t.dropFirst()
}
private static func stripEllipsis(_ t: Word) -> Word {
if hasEllipsis(t) { return Array(t.dropLast(3)) }
if t.last == "\u{2026}" { return Array(t.dropLast()) }
return t
}
private static func trimming(_ t: Word, _ set: Set<Unicode.Scalar>) -> Word {
var end = t.count
while end > 0, set.contains(t[end - 1]) {
end -= 1
}
return Array(t[..<end])
}
private static func string(_ t: Word) -> String {
var view = String.UnicodeScalarView()
view.append(contentsOf: t)
return String(view)
}
}