The touchscreen revolution — teaching AI to type like humans

There’s no doubt that touchscreens were an incredible innovation in the field of both consumer and commercial computing. Sure, they might be a mainstay these days, but there was a time when they were strictly in the realm of science fiction — at least in the way that they were portrayed in popular media. As ubiquitous as this technology is these days, however, it isn’t without its problems. One of which is the ease and intuitiveness of touchscreen typing.

Interestingly enough, the actual technology of touchscreens was around as far back as the 1970’s. In his 1965 article, “Touch Display — A novel input/output device for computers”, Eric Johnson described the practical implications of such technology, albeit in a somewhat abstract way, as the computing technology hadn’t caught up just yet. Throughout the decades, more and more research was done on the topic, but it wasn’t until the advent of modern computing that it really caught on in a meaningful way.

Getting back to the topic at hand, how do you type more often — on a physical keyboard or a touchscreen one? Though it can be a matter of personal preference, often the choice is made for us. For example, using a touchscreen kiosk to place your order at a fast food restaurant, or purchasing train tickets at a self-service station. Therefore, if the process can be improved, it’s clearly worth improving.

Using touchscreens replaces the physical feedback of typing with a completely visual one (though some digital keyboards do include a “click” sound to indicate that a key has been pressed). Understanding this, the Finnish Center for AI (FCAI), along with Aalto University, have developed a novel AI model that predicts how people move their eyes and fingers while typing. Methods for typing are known to adapt to the task, design, and user. In their recent paper, these researchers have proposed a unifying account of touchscreen typing.

Dr. Jussi Jokinen, who led the work, had this to say: “Previously, touchscreen typing has been understood mainly from the perspective of how our fingers move. AI-based methods have helped shed new light on these movements: What we’ve discovered is the importance of deciding when and where to look. Now, we can make much better predictions on how people type on their phones or tablets.”

One of the interesting things the study presented was the groundwork for developing better, and even personalized, text entry solutions. “Now that we have a realistic simulation of how humans type on touchscreens, it should be a lot easier to optimize keyboard designs for better typing — meaning fewer errors, faster typing, and, most importantly for me, less frustration,” Jokinen explains.

Some of the foundations of the study’s results are based on the classic machine learning method, reinforcement learning, that the researchers extended to simulate people. According to an article on Deepsense.ai, “Reinforcement learning is the training of machine learning models to make a sequence of decisions … The computer employs trial and error to come up with a solution to the problem.” Though traditionally employed in teaching robots to solve tasks, the research team employed this method to simulate the cumulative actions of humans who use touchscreens, including not only their actions, but also errors and corrections.

Jokinen explained that the model was given similar characteristics to that of a human, such as their abilities and shortcomings. The idea was, after all, to accurately simulate the way a person types, not create the ultimate typing machine. These guidelines helped structure the model in a way that would very accurately simulate actual human data — without requiring actual human data.

This approach resulted in accurate predictions, and the team hopes to apply this technology in the future for such fields as programs to improve a person’s typing ability. Keep in mind that as simple as it is to type on a touchscreen keyboard, the programming and intelligence that goes into it makes it a monumentally impressive achievement, even if you’re just typing to your friend to say hello.