Electromagnetic sensing and actuation for pen-based haptic feedback
How to use electromagnetic principles for sensing and actuation across untethered haptic devices.
Humans can feel and process extremely complex tactile sensations. However, most electronic devices predominantly communicate with humans across two senses: visual and auditory. Haptic feedback is concerned with adding the sense of touch to the virtual world. Adding this extra communication channel has proven to increase the immersion when interacting with the digital world, improve a large variety of different user interfaces, and allow for a more dense information stream.
Haptics is an active field of research and many ways to elicit the sense of touch have been proposed; from complex mechanical, robot-arm-like devices anchored in the world to deliver strong forces, to small vibrotactile actuators mounted in wearable gadgets. No matter the design, ideally, a haptic device is designed to react to a user, creating a closed-loop flow of information. In other words, the question each system should answer is two-fold: 1) What is the state of the user and 2) Which feedback should I apply given the state?
In our work for UIST 2020 [1, 2], we aim to answer these questions specifically for electromagnetic haptic feedback systems. Electromagnetism enables systems to be untethered yet grounded, which means it can deliver significant forces over distance. Additionally, as we will see, electromagnetic haptic devices can be used for both actuation as well as sensing; thereby answering both of our questions.
User State Inference
Let’s first see how electromagnetism can be used to infer the state of a user, in this case by reconstructing in real-time the position of a tool they are handling. We elaborate on this in our paper Omni: Volumetric Sensing and Actuation of Passive Magnetic Tools for Dynamic Haptic Feedback. Done at ETH Zurich, by Thomas Langerak, Juan José Zárate*, Velko Vechev, David Lindlbauer, Christian Holz, and Otmar Hilliges.
Omni is a self-contained haptic feedback system that simultaneously integrates 3D tracking and actuation. Omni’s actuator — a spherical electromagnet — applies force on a pen-like tool which has a permanent magnet embedded. Sensing the position and rotation of the tool is achieved with an array of hall sensors. A complete overview of the hardware can be found in Figure 1.
To understand our sensing method, let’s start with the information we get. The hall sensors measure the superposition of three magnetic fields: 1) the field generated by the permanent magnet in the tool, 2) the field generated by the spherical actuator, and 3) the background magnetic field, like Earth’s and noise. The last one can be removed with proper calibration, while the other two can be modeled using, for example, the dipole-dipole approximation.
We use iterative optimization to compare the readings of the array of sensors to what our model predicts. We propose the tool to be at a certain position p (see Fig. 2) and we compute our model. The difference between the modeled and read values approaches to zero as the proposed tool position converges to the real one.
Since we explicitly model the interaction of the electromagnet, our proposed method generalizes to virtually all positions of the pen (within hardware limitations) as well as the actuation of the electromagnet. Simply put, because we understand and can model the system; we can infer its state (i.e the pen position) based on the measurements of the hall sensors.
Haptic Feedback
Now we know how to infer the state of the user, let’s take a look at the second question; which feedback should I apply given the state?
In our paper Optimal Control for Electromagnetic Haptic Guidance Systems, also published at UIST this year, we frame controlling the magnetic actuator as an online optimization problem. This project was done at ETH Zurich and NYU by Thomas Langerak*, Juan José Zárate*, David Lindlbauer, Daniele Panozzo, and Otmar Hilliges. We specifically focused on drawing and writing tasks, where the desired trajectory is available. The main question we try to answer is: How can we find a balance between haptic feedback and user autonomy?
We tested this control approach in a different haptic feedback setup, particularly designed for hand drawing tasks. It consists of a bi-axial linear stage on which an electromagnet is mounted. One can think of this hardware as an upside-down 3D printer with a magnet instead of an extruder for material attached to it. The pen position is determined by a pressure tablet or a touch-sensitive tablet with visual feedback. The pen is augmented with a permanent magnet.
For our control, we make two assumptions. First, that we know in real-time pen position (e.g. from the tablet) and secondly, that there is the desired trajectory (e.g. from a given target drawing). There are three questions that we seek an answer to 1) what is the closest point on the trajectory, 2) what is the desired progress along the trajectory, and 3) what is the desired haptic force to be applied. The former two have been answered in previous work. The latter is the main contribution of our paper.
We frame these questions as cost norms in a model-predictive contour control formulation [3], which has been used before to control drones [4].
This is possible because we are able to model how the inputs of our system relate to its state and the pen-position. As an example, we know how acceleration influences position. Similarly, using the dipole-dipole model, we know how the distance between the electromagnet and pen influences force. Simply put, because we understand and can model how our system works we can approximate an optimal control strategy.
In our user study, we found that users improved their accuracy while they remained to have creative freedom in drawing tasks. For, more technical details and information please refer to the paper.
Conclusion
We think that fundamentally, every haptic device should answer and react to two questions. First, what is the state of the user and, secondly, what feedback to apply given this state? We outlined two papers presented at UIST that answer the questions for electromagnetic systems, where we leverage known dipole-dipole models to understand the world and actions. If we want to use the world around us for sensing and actuating we need to understand and be able to model how it works. In our papers, we specifically used a closed-form solution. In some cases, however, methods such as iterative optimization and deep learning could be explored. Improved models of the system, should increase overall performance drastically.
About the Authors
Omni: Volumetric Sensing and Actuation of Passive Magnetic Tools for Dynamic Haptic Feedback
Thomas Langerak*, ETH Zurich
Juan Zarate*, ETH Zurich
David Lindlbauer, ETH Zurich
Christian Holz, ETH Zurich
Otmar Hilliges, ETH Zurich
*Contributed Equally
Optimal Control for Electromagnetic Haptic Guidance Systems
Thomas Langerak, ETH Zurich
Juan Zarate, ETH Zurich
Velko Vechev, ETH Zurich
David Lindlbauer, ETH Zurich
Daniele Panozzo, NYU
Otmar Hilliges, ETH Zurich
Our papers:
[1] Thomas Langerak, Juan Zarate, David Lindlbauer, Christian Holz, and Otmar Hilliges. 2020. Omni: Volumetric Sensing and Actuation of Passive Magnetic Tools for Dynamic Haptic Feedback. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST ’20). Association for Computing Machinery, New York, NY, USA, 594–606. DOI:https://doi.org/10.1145/3379337.3415589
[2] Thomas Langerak, Juan Zarate, Velko Vechev, David Lindlbauer, Daniele Panozzo, and Otmar Hilliges. 2020. Optimal Control for Electromagnetic Haptic Guidance Systems. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST ’20). Association for Computing Machinery, New York, NY, USA, 951–965. DOI:https://doi.org/10.1145/3379337.3415593
Other References
[3] Denise Lam, Chris Manzie, and Malcolm Good. 2010. Model predictive contouring control. In Decision and Control (CDC), 2010 49th IEEE Conference on. IEEE, 6137–6142. DOI: http://dx.doi.org/10.1109/CDC.2010.5717042
[4] Tobias Nägeli, Lukas Meier, Alexander Domahidi, Javier Alonso-Mora, and Otmar Hilliges. 2017. Real-time Planning for Automated Multi-View Drone Cinematography. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH) (2017).
