Beyond The Force: Using Quadcopters to Appropriate Objects and the Environment for Haptics in Virtual Reality
This article summarizes a paper authored by Parastoo Abtahi, Benoit Landry, Jackie Yang, Marco Pavone, Sean Follmer, and James Landay. This paper will be presented at CHI 2019, on Wednesday, May 8th 2019 at 2:40 PM, in the session “Mapping and 3D”, in room Boisdale 1.
Imagine a future …
where drones are smaller, quieter, safer, and more affordable. Now imagine that you own a few of these drones, with a gripper mechanism attached to them so they can pick up different objects. You wear your Virtual Reality (VR) headset and enter a virtual world. The drones begin to scan the room in search of everyday objects or 3D printed props that resemble the shape of the virtual objects in the scene. When a match is detected, the closest drone picks up that object. As you reach out to touch a spaceship in VR, and before you make contact with it, the system predicts what you’re about to touch. Then the drone carrying the object with the closest match, a dust pan in this case, flies quickly to where you are, just in time for you to feel, grasp, and manipulate the spaceship. In our research, we take the first steps for making this future a reality.
Encountered-Type Haptic Devices
Haptics refers to forms of interaction involving the sense of touch. In virtual reality this sense of touch is often missing, so when you make contact with a virtual object, you don’t feel that object. Haptic devices aim to create this sensation of touch, to not only make virtual experiences more immersive, but also to provide additional information about the object, such as surface texture or weight, and to enable easy manipulation of that object. Encountered-type haptics are a subset of haptic devices that move around and present themselves, such that when you contact a virtual object, your hand touches the device in the real world. Conventionally, encountered-type haptic devices are grounded robotic arms; however, robotic arms are often expensive, require complex motion planning, and have a limited workspace.
Drones as Encountered-Type Haptic Devices
In our work, we highlight the use of drones as hovering encountered-type haptic devices, as they can fly quickly almost anywhere in the room and may be more affordable than robotic arms. We argue that drones can facilitate rich haptic interactions, beyond providing force feedback, by appropriating objects and the environment.
Our System: Hover Haptics
To showcase haptic interactions mediated by drones, we first built a safe-to-touch drone completely enclosed in a meshed cage. We then integrated this autonomous drone with a virtual shopping experience. We placed different fabrics around the drone, allowing users to feel the texture of different pieces of clothing. We also attached a hanger to the drone, so that users could pick up a clothing item by the hanger and place it in their shopping basket. When picking up a shoebox, the drone could land on a table and act as a passive box.
Design Considerations
There are many requirements for having a usable hovering encountered-type haptic system, including high carrying capacity, safety, high speed, minimal delays, accurate position control, and low noise. Due to the inherent limitations of drones, it is challenging to satisfy these requirements individually; however, the bigger challenge is that these requirements are interdependent and cannot all be satisfied simultaneously. For example, by increasing the carrying capacity or the speed of the drone we compromise on safety since in the unlikely event of a collision the impact force will be higher.
In our work, we describe how we prioritized these requirements and the techniques we used to achieve a usable haptic system, despite these interdependencies. For example, to compensate for the drone’s lack of position accuracy, we implement a dynamic retargeting algorithm to create the illusion that the user is touching the virtual object, while touching the drone at a slightly different position in the real world.
User Evaluation
We ran a preliminary user study with 9 participants to better understand the subjective user experience when using our system and interacting with a drone in virtual reality. Most participants enjoyed the haptic experience and felt safe. With the dynamic retargeting illusion, everyone was able to make contact with the drone. Participants most enjoyed picking up the hanger, and we think this is because this task went beyond exploration and enabled active manipulation of a virtual object. Finally, the majority of participants were able to hear the noise and feel the wind generated by the drone, but to our surprise, they appreciated this as an ambient feedback for where the drone was relative to them. For more information, please refer to our paper.
Full Citation
Parastoo Abtahi, Benoit Landry, Jackie (Junrui) Yang, Marco Pavone, Sean Follmer, James Landay. 2019. Beyond The Force: Using Quadcopters to Appropriate Objects and the Environment for Haptics in Virtual Reality. In CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, Glasgow, Scotland UK. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/ 3290605.3300589
