Scribble is arguably the best-known new feature of Apple’s recent iPadOS 14 update, and in a new interview with Popular Mechanics, Apple SVP of software engineering Craig Federighi discusses how it was developed and how it works on the iPad.
Scribble on iPadOS 14 allows iPad users to use an Apple Pencil to handwrite in any text field and have that written text be automatically converted to typed text. The feature allows iPad users to use an Apple Pencil to create sketches and enter text without the need to switch back and forth between the device’s keyboard and Appl Pencil.
Federighi tells Popular Mechanics that Scribble’s early development involved extensive data gathering to allow analyzing of how people from all over the world write things down.
“When it comes to understanding [handwriting] strokes, we do data-gathering. We find people all over the world, and have them write things,” says Craig Federighi, senior vice president of software engineering at Apple. “We give them a Pencil, and we have them write fast, we have them write slow, write at a tilt. All of this variation.”
That methodology is distinct from the comparatively simple approach of scanning and analyzing existing handwriting. Federighi says that for Apple’s tech, static examples weren’t enough. They needed to see the strokes that formed each letter. “If you understand the strokes and how the strokes went down, that can be used to disambiguate what was being written.”
Federighi says the job of integrating stroke-based recognition with character and word prediction is all done on-device, something that machine learning and raw processing power makes possible.
“It’s gotta be happening in real time, right now, on the device that you’re holding,” Federighi says. “Which means that the computational power of the device has to be such that it can do that level of processing locally.”
