hex/ios/HEXShared/FrenchSpacing.swift
Ryan Vogel 361978579d
iOS: French spacing in dictation, and a second language no longer hurts English autocorrect (#2186)
* iOS: French dictation gets French spacing before ? ! ; : and inside « »

X (Philippe): "When dictating, in some languages punctuation has a space
before, like question and exclamation marks in French. Right now they're not
inserted."

Black box first:
- Apple Speech (macOS 27.2.1, say -v Thomas and Amélie): fr-FR and fr-CH write
  "ça va ?" with a plain U+0020 space; fr-CA's DictationTranscriber writes
  "ça va?" (its SpeechTranscriber a plain space). None writes U+202F/U+00A0.
  Hex's Polished, Casual and Notes then took every space before a mark out
  (fixSpacingBeforePunctuation), so French came out "ça va?".
- Apple's Français (France) AZERTY keyboard (iOS 27.2 simulator, real taps in
  Messages): "Salut" then ? gives "Salut?", "salut ?" keeps the plain space
  typed, same for ! : ;. The keyboard never adds French spacing.

Change:
- FrenchSpacing, the dictation pipeline's last step (DictationPipeline.format,
  after the style, lists, vocabulary spellings and Lowercase; every style but
  Verbatim, which keeps the model's text). France and every other French: a
  narrow no-break space (U+202F) before ? ! ; and a no-break space (U+00A0)
  before : and inside « ». Canada (OQLF): none before ? ! ;, U+00A0 before :
  and inside « ». Switzerland: U+202F before ? ! ; and inside « », U+00A0
  before :. A translation into French is spaced as French (format already
  takes the target language).
- Whatever space was there (none, several, NBSP, NNBSP, thin) becomes exactly
  one; running it twice changes nothing. Only a mark that ends a word is
  spaced (followed by a space, the end, another mark or a closing quote or
  bracket), so 10:30, URLs and ?q=1, a mark typed against a URL or address,
  emoticons (:) ;-) :D), code in backticks, "::", a mark at a line's start and
  the second mark of ?! stay as they are.
- Full keyboard: unchanged, matching the system keyboard (marks go in as
  tapped in every French layout).

Defaults picked (no tester answer): Swiss guillemets use the narrow space
(Guide du typographe); Verbatim is left alone.

Verified (iPhone 17 simulator, iOS 27.2, Mac Studio): xcodebuild test, 530
tests, 1 skipped (needs HEX_SPEECH_CLIPS), 0 failures, including new
FrenchSpacingTests (pipeline in every style, Canada and Switzerland, no
doubled spaces, times/URLs/emoticons/code untouched, idempotent) and
FullKeyboardEngineTests.testFrenchMarksGoInAsTyped (fr, fr-CA, fr-CH).

* iOS: bilingual autocorrect keeps English as good as alone beside French or German

X (Philippe): "Would also be great if the keyboard supported typing multiple
languages without having to actively choose which one to use at a given
time, like Gboard."

What works today: bilingual autocorrect (#2097, #2143, #2164) already pairs
any two Latin-script languages with an iOS dictionary (English with Français,
Deutsch, Italiano, Português, Polski, Norsk, Română, Español), whatever their
order: the first other chosen Latin language is the partner, its dictionary
is checked per word and the strip shows both languages' fixes and completions.
So no new pairs were needed. But the held-out English benchmark (2,000
Tatoeba sentences, Spatial autocorrect on) got much worse with a partner on:
5.75% WER alone, 7.56% with Français, 7.53% with Deutsch (7.18% with the
German Spatial experiment). Causes, measured:
- iOS's French and German dictionaries take many English typos for words
  ("dont", "im", "ive", "dk", "dl", "ro", "tkm"), and a word either language
  knew was never changed, so "Im" and "dont" stayed (61 and 42 losses).
- They also take common English words ("the", "you", "your", "not",
  "please"), so the words-before lean almost never said English.
- The partner's spelling guesses changed names English kept ("Ziri" ->
  "Zuri").

Changes (AutocorrectEngine):
- languageLean: a word both dictionaries know counts, from three letters, for
  the language whose typing model has it among its commonest (-ln P <= 10).
- In a sentence that leans the keyboard's language, a word the partner knows
  still gets this language's sure fixes (unambiguousFixes, two-letter slips).
- Without a lean, Spatial autocorrect may still read the taps as another word,
  counting the partner's word as a valid one (changed only on its valid-word
  margin), except words with a sure fix, which the next word decides.
- bilingualRevision: at a sentence's start, "Im"/"Dont" become "I'm"/"Don't"
  at the next word's space when that word leans English (German "Im Haus"
  stays; the revision's undo is the existing one).
- A partner word without a vowel is no word (iOS abbreviations), and the
  partner's guess of a name is never a fix.
- Kept: #2143's guard (workingPartner ignores a dictionary that takes every
  word), #2164's layout rule (English and Spanish on the first one's keys;
  other languages keep their own layouts), dictation following the keyboard.

Removed: "a word either language knows as typed is never changed" without
exception; it now has the exceptions above.

Black box (iOS 27.2 simulator): Settings offers "Add to English (US)
Keyboard" for French (France, Canada, Switzerland) ("Type in English and
French on the same keyboard": one "English & French" keyboard on QWERTY, no
switching); Portuguese, Polish, Turkish offer "Add to Existing Keyboard".
It fixed "I dont know" and "Dont do it", kept "est" and "merci" after English
words; its French data is thin in the simulator ("je pense" -> "he pence"),
so its French weighting is unverified.

Verified (iPhone 17 simulator, iOS 27.2, Mac Studio, rebased on #2180):
- Benchmark (ZZDecoderBench, not committed): English alone 5.75% (unchanged
  by this change); with Français 7.56% -> 5.79% (correct words broken 125 ->
  99, 113 alone); with Deutsch 7.53% -> 5.79%; Deutsch with its Spatial
  model 7.18% -> 5.70%. Before rebasing (2a023f91): alone 5.24%, Français
  7.07% -> 5.24%, Deutsch 7.08% -> 5.27%.
- Code-switching probe (centre taps): after "we said that", 1 of 80 French
  words changes (dont -> don't) and 3 of 72 German (auch, ist, habe), none
  before; at a message's start and after French or German words, none; slips
  in French/German words at a message's start fixed as before (26/50, 18/40).
- Letter p95 with a partner 4.1-5.4 ms (3.9-5.1 ms before), 2.2 ms alone.
- xcodebuild test: 530 tests, 1 skipped, 0 failures, including new
  AutocorrectEngineTests.testBesideAnotherLanguageEnglishStillFixesItsOwnTypos
  and the existing bilingual tests (testABilingualKeyboardKeepsBothLanguagesWords,
  testTheFrenchKeyboardFixesEnglishTyposWithEnglishBesideIt,
  testAPartnerDictionaryThatTakesEveryWordIsIgnored).
2026-11-21 08:37:33 -05:00

149 lines
6.4 KiB
Swift
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import Foundation
/// French typography's spaces before `? ! ; :` and inside `« »`, the dictation pipeline's last
/// step (`DictationPipeline.format`). Apple Speech doesn't write them (black box, macOS 27.2.1:
/// fr-FR and fr-CH write a plain space before "?", fr-CA's DictationTranscriber none) and the
/// styles take out every space before a mark, so French came out as "ça va?". Apple's French
/// keyboards don't add one either (iOS 27.2 simulator), so the full keyboard types marks as you
/// tap them.
///
/// - France (and every French but Canada's and Switzerland's): a narrow no-break space
/// (U+202F) before `? ! ;`, a no-break space (U+00A0) before `:` and inside `« »`.
/// - Canada (OQLF): no space before `? ! ;`, a no-break space before `:` and inside `« »`.
/// - Switzerland (Guide du typographe): narrow no-break spaces before `? ! ;` and inside
/// `« »`, a no-break space before `:`.
///
/// Whatever space was there (none, one, several, or another kind) becomes exactly that one. A
/// mark is spaced only where it ends a word: followed by a space, a line break, the end,
/// another mark or a closing quote or bracket. So times (10:30), ratios, URLs ("https://",
/// "?q=1"), a URL or address the mark is typed against, emoticons (":)", ";-)", ":D"), code in
/// backticks, a mark at a line's start, and the second mark of "?!" stay as they are.
nonisolated enum FrenchSpacing {
enum Convention: Equatable {
case france
case canada
case switzerland
}
static let narrowNoBreakSpace: Character = "\u{202F}"
static let noBreakSpace: Character = "\u{00A0}"
/// The convention for `language` (BCP 47), or nil when it isn't French.
static func convention(for language: String?) -> Convention? {
guard let language else { return nil }
let locale = Locale(identifier: language)
guard locale.language.languageCode?.identifier == "fr" else { return nil }
switch locale.region?.identifier {
case "CA": return .canada
case "CH": return .switzerland
default: return .france
}
}
/// `text` with French spacing when `language` is French; otherwise unchanged.
static func apply(_ text: String, language: String?) -> String {
guard let convention = convention(for: language) else { return text }
return apply(text, convention: convention)
}
static func apply(_ text: String, convention: Convention) -> String {
let chars = Array(text)
guard chars.contains(where: { "?!;:«»".contains($0) }) else { return text }
let beforeQuestion: Character? = convention == .canada ? nil : narrowNoBreakSpace
let insideGuillemets = convention == .switzerland ? narrowNoBreakSpace : noBreakSpace
var out: [Character] = []
out.reserveCapacity(chars.count + 8)
// Spaces read but not yet written: a mark after them may replace them.
var pending: [Character] = []
var inCode = false
var afterOpeningGuillemet = false
func flush() {
out.append(contentsOf: pending)
pending.removeAll()
}
for (index, ch) in chars.enumerated() {
let next = index + 1 < chars.count ? chars[index + 1] : nil
if ch == "`" {
inCode.toggle()
afterOpeningGuillemet = false
flush()
out.append(ch)
continue
}
if inCode {
flush()
out.append(ch)
continue
}
if isHorizontalSpace(ch) {
pending.append(ch)
continue
}
if afterOpeningGuillemet {
afterOpeningGuillemet = false
if !ch.isNewline, ch != "»" {
pending = [insideGuillemets]
}
}
switch ch {
case "?", "!", ";", ":":
if endsWord(ch, next: next),
let space = spaceBefore(ch, out: out, pending: pending, question: beforeQuestion) {
pending.removeAll()
if let space { out.append(space) }
} else {
flush()
}
out.append(ch)
case "«":
flush()
out.append(ch)
afterOpeningGuillemet = true
case "»":
if let last = out.last, !last.isNewline, last != "«" {
pending = [insideGuillemets]
}
flush()
out.append(ch)
default:
flush()
out.append(ch)
}
}
if !afterOpeningGuillemet { flush() }
return String(out)
}
private static func isHorizontalSpace(_ ch: Character) -> Bool {
ch == " " || ch == "\t" || ch == noBreakSpace || ch == narrowNoBreakSpace || ch == "\u{2009}"
}
/// Whether `mark` ends a word here, rather than sitting inside a time, a URL or an
/// emoticon: what follows is nothing, a space, a line break, another mark or a closing
/// quote or bracket, and for `:` and `;` not an emoticon's mouth (":)", ";(").
private static func endsWord(_ mark: Character, next: Character?) -> Bool {
guard let next else { return true }
if mark == ":" || mark == ";", ")(][|".contains(next) { return false }
return next.isWhitespace || "?!.…,»”\"’')]}".contains(next)
}
/// The space to write before `mark` (`.some(nil)` for none), or nil to leave what is
/// there: at a line's start, after an opening quote or bracket or another ";" or ":"
/// ("::"), and for a mark typed
/// against a URL or an address. The second mark of "?!" goes against the first.
private static func spaceBefore(
_ mark: Character, out: [Character], pending: [Character], question: Character?
) -> Character?? {
guard let last = out.last, !last.isNewline, !"(«“[{".contains(last) else { return nil }
if "?!".contains(mark), "?!".contains(last) { return .some(nil) }
if ";:".contains(last) { return nil }
if pending.isEmpty {
let word = String(out.reversed().prefix { !$0.isWhitespace }.reversed())
if word.contains("://") || word.contains("@") || word.lowercased().hasPrefix("www.") { return nil }
}
return .some(mark == ":" ? noBreakSpace : question)
}
}