Digital inking — we need better hardware and software

Digital inking has been with us for years and there is a lot of research materials from 70's available on that topic. First moves to make it mainstream, however, can be dated on early 2000’s thanks to Bill Gates and his Tablet PC concept. Gates for years has been probably the biggest advocate of pen input for natural interaction with PCs.

Tablet PCs didn’t work and it is not the pen input to blame for that. Or at least not directly. Hardware and software was not ready by then. And in my opinion, it’s still not ready. But let’s start with Tablet PC issues. Poor battery life, bulky devices and mediocre pen accuracy haven’t helped for sure. Windows XP Tablet PC Edition was a desktop operating system designed primarily for keyboard and mouse. Pen enhancements were nice for some tasks, but the OS as a whole was not ready for switching from pen/keyboard to stylus. It just was not natural. Not as natural as pen and paper.

Fast forward to 2016 and neither software nor hardware is there yet. Writing or drawing is still not as natural as on paper. And that is one of the main goals. But is digital inking a real added value or just a fad? What is it for? How is it better than finger touch or keyboard/mouse? Below you’ll find my non-comprehensive and subjective list:

• note taking (especially mathematical formulas, graphs, mind maps),
• annotating and sketching,
• drawing and painting (professional artists),
• OS-wide input/control.

Let’s start with hardware and technology behind modern pens.

Technology

There are 3 main types of pen digitizer technologies: electromagnetic, passive capacitive (those conductive rubber tipped pens that trick the digitizer into looking like a finger), and active capacitive.

• Electromagnetic works by putting a printed circuit board across the entire device, typically situated underneath the display and its backlight. This is a completely separate system from the touch digitizer that usually goes in front of the display. Data is transmitted from the pen to the device (pressure and button data), by modifying the frequency content of the load. The circuit board has a bunch of planar coils that emit an electromagnetic field (like one side of a transformer). The other side of the transformer is located in the pen itself. As the pen approaches these fields and coils it couples the electromagnetic signal and adds a load. This load is picked up across multiple coils and the position of the stylus is then interpolated.
• Passive capacitive stylus, works by simply acting as an extension of your finger, by being a conductor to passively couple with the electrostatic signal from the transmit and receive rows and columns of the digitizer. This technique uses the same transparent conductive lines in front of the display to do both the finger and “fake” finger/stylus.
• Active capacitive — their mode of operation is to use the same transparent conductive lines as passive stylus does above, but rather the pen tip injects an electrostatic signal which is picked up by these touch capacitive lines. We should think of the pen is a mini radio and the sensing lines in front of the display the device are little antennas. The junction (where the rows and columns of the transparent conducting lines cross) which receives the strongest signal is correlated to the position of the pen. To do this, the pen typically needs a battery.

Most Wacom devices use electromagnetic (Surface Pro 1, 2) technology that has its advantages, but also issues with corner accuracy or adding to the device thickness. Newer Surface devices utilize active capacitive technology (from Israeli company N-Trig that Microsoft acquired in 2015) with dedicated battery and Bluetooth connection for features like launching OneNote or Cortana — with pressure sensitiveness. iPad Pro uses passive capacitive Apple Pencil that is purely Bluetooth-based, with surprisingly little lag, great pressure sensitivity and tilt detection.

Inking philosophy or “if you see a stylus, they blew it”

Analyzing current “penabled” device landscape, I can think of two main approaches to digital inking:

• Generic pen input as an alternate way to control operating system (for UI interaction, text input and of course app-specific sketching/drawing/painting) — Windows/Surface,
• Pen input for specialized in-app tasks like painting, drawing, annotating, targeted at designers or artists — iOS/iPad Pro.

Microsoft wants to continue Bill Gates’ dream about natural interfaces that include pens for controlling operating systems and for more specialized tasks. It’s even more clear after recent announcement of Windows Ink, that integrates pen features even deeper into OS. You can annotate on Word or PDF files, screenshots, launch apps using pen, use OS-native handwriting-to-text input or take notes/sketch easily with close OneNote integration. For few years, Windows is not only a desktop operating system, but also a tablet operating system. Desktop part of the OS still isn’t finger-touch friendly — that’s how Microsoft promotes pen input so heavily. It’s more keyboard/mouse replacement, not (only) note-taking/sketching tool. That’s not the only reason, however. MS cites research about daily pen usage and impact of handwriting on learning abilities. It seems that the company truly believes that digital ink is part of the natural OS interactions, next to speech, gestures and finger-touch.

Apple’s approach is much different. A lot of people laugh at Apple and Pencil because of famous Jobs’ quote: “if you see a stylus, they blew it”. It’s easy (and probably right) to make fun of big corporation, but in this case, there’s no contradiction. Apple Pencil is not a stylus to control the OS. Finger-touch is still the main input method and Pencil is just a non-obligatory accessory for narrow group of iPad customers (artists, designers). You don’t need Pencil to interact with OS conveniently. And probably never will. This accessory shouldn’t be treated as an iOS mouse equivalent. It is not the point.

Android doesn’t have native ink support, but companies like Samsung created their own solutions in Note smartphone and tablet lines. Samsung is somewhere between Microsoft and Apple. S-Pen has some OS integration for additional features and cursor, but no nearly as deep as in Windows 10. Korean giant presented also a different approach to store the stylus — you can hide it inside the device in a dedicated silo. Very convenient solution, but it forces the pen to be very thin.

Hardware issues

We covered the history and general approaches, so it’s time to talk about hardware implementations and current issues with pen technology. Every digital pen has its hardware lag and some jitter (especially on diagonal strokes). It’s a matter of signal propagation. The lesser the lag, the more natural it feels. Hardware lag has been reduced greatly recently, and basically it’s no longer an issue. But there is still an undesirable parallax effect — an optical phenomenon caused by light diffraction on a piece of glass. Because of that, there is a slight offset between the pen tip and observed inking position on the digital canvas. The situation gets better and better with modern devices where display layer is bonded with thin glass that let the optical stack to be really thin.

Virtually no parallax and little lag are not enough to ensure pen&paper-like experience. It is also about the glass texture and friction. Even rubber pen tips glide on the glass surface and it doesn’t feel natural. Until it’s not 100% perfect, many people will still choose real pen for note taking as more reliable and convenient than digital version. For Apple Pen, using lightning port to charge it every few hours is also a deal breaker for many. AAAA Battery in Surface Pen lasts almost 2 years in comparison. Due to passive nature of Pencil, there is no hoover/cursor feature — device registers the position of a pen only after touching the glass. That’s not very problematic considering the role of a pen in iOS, but Windows has a clear advantage here bringing much more versatility.

Software issues

With modern software and native ink APIs, things like missed strokes or software lag are not the problem anymore. Most apps and systems that properly implemented the inking engine, are really snappy. Inking engine is not the issue in general. My biggest problem with modern digital inking is still OS and app integration plus palm rejection software (a software that let you place your palm on the screen while sketching, without registering unwanted touch). Again — it has to feel 100% natural to replace real pen and paper. You shouldn’t think how to use it, before you use it. We are slowly getting there, but there still is too much thinking while using digital pens. It’s not 100% reliable in 100% of apps 100% time.

Windows also has a problem with desktop or mobile apps that for some reasons don’t use native ink APIs, but own proprietary ones. Custom ink software often brings performance issues — lag, pressure sensitivity or drivers. We can see it in Photoshop and other pro software. Native approach in UWP apps (in coming Windows 10 Anniversary Update) or iOS ensure deep OS integration for flawless performance and predictable, consistent behavior. We can only hope that big companies like Adobe or Autodesk will utilize these tools in the future. Or… bring their desktop suites to Universal Windows Platform. Which is very inprobable, unfortunately.

What’s next

What one can expect from digital inking in the future? Better performance (reducing HW/SW lag, parallax), deeper OS integration, more convenient ink-to-text, more natural haptic feedback (pen tip friction) — for starters. As technology matures, new features like grip sensing tech, advanced tilt features or ink-to-object engine will arise, bringing completely new experiences. Question of mainstream adoption of digital inking remains open, however. But it definitely shouldn’t and mustn’t be ignored when Apple, Microsoft and Samsung promote pen input in their flagship devices.