Imagining the future of remote collaboration with robots
on physical tasks. Photo Credit: @wisalumni
đ˘ Check out our paper! We had a presentation and demo at CSCW 2023 đ¤.
Technologies that support remote work have advanced significantly in recent decades, and they were indispensable during the COVID-19 pandemic. However, remote support for jobs that involve physical work, such as nursing or factory work, was rare. This highlights opportunities for the use of robots that can leverage their physical embodiment to extend peopleâs abilities in remote spaces.
Advancements in robotics have enabled collaborative robots (or cobots) that are safe, easy to use, and capable, thus opening up a promising space for designing systems that support remote human collaboration. In this post introducing our research, we present Periscope, a robotic camera system that we developed using a cobot platform to enable new capabilities for remote collaboration on physical tasks.
What motivated the design of Periscope?
Remote collaboration on digital tasks, such as co-editing a document, delivering a presentation, or collaboratively playing a video game, is fundamentally different from physical tasks such as furniture assembly, cooking, or appliance repair. Physical tasks require people to look at physical objects from various angles and in varying levels of detail.
Cobots, equipped with cameras, have a wide range of motion that allows for precise camera control for complex physical tasks. However, this improved capability also increases the complexity of camera control. Thus, despite their potential, cobots with high kinematic capabilities â in contrast to traditional tablet-on-wheels telepresence robots â have rarely been utilized in systems that support remote collaboration. In developing the Periscope system, we tackled the challenge of designing camera control that enables the use of highly capable robotic cameras for remote collaboration.
What is Periscope?
The Periscope system (shown in the figure above) supports a âworkerâ (figure: left) in completing physical tasks with remote guidance from a âhelperâ (figure: right), who views the workspace through a robot-mounted camera. The camera view is displayed on a screen interface for both the worker and the helper, enabling them to share dynamic, task-relevant visual information and develop a mutual understanding during collaboration.
The illustration above shows the different panels in the Periscope user interface, and below is a brief demo of two people remotely collaborating on a circuit assembly task using Periscope.
What is shared camera control?
Previous research suggests that in collaborative scenarios, both the helper and the worker should have the ability to control the camera view at different points (e.g., to provide guidance or ask questions). However, users may find it challenging to operate a robotic camera with advanced kinematic capabilities. To alleviate this, we implemented some level of automation in the camera control to make it more user-friendly. It is important to note, however, that complete automation of camera control is not desirable either, as users will still want the autonomy to choose and view content according to their high-level collaboration goals.
Our challenge is to design an interface that effectively mediates the interaction between two people by achieving a balance between human control and robot assistance in managing the camera view.
Our shared camera control approach empowers both collaborators to independently control the view using mouse inputs and gestures, while also allowing the robot to reduce their effort through autonomous behaviors like hand tracking and collision avoidance. We implement a leader-follower paradigm to arbitrate the inputs from the three entities â the worker, the helper, and the robot. Thus, Periscope offers three modes, where each mode is led by one of the three entities, with the other two exerting less influence over the camera view (see figure â for details).
The results from our user study supported our hypothesis that each mode would be beneficial at different stages of the collaboration process (refer to our paper for an in-depth analysis of user interactions). The figure below exemplifies a workflow and demonstrates the rich and dynamic interactions facilitated by Periscope. The most encouraging result was the frequent transfer of view control among the worker, the helper, and the robot. Each entity often had ownership of the point of view when they had ownership of a task or relevant information. Thus, with our leader-follower paradigm for arbitration, the leader drove the task forward based on information they possessed, and the follower followed suit.
How can we build on this?
With Periscope, we developed an interface on a cobot platform that enabled the use of a highly capable robotic camera for remote collaboration. Some potential extensions to our system that point to future research include:
- Improving robot initiative: The robot may take on a more proactive role, such as monitoring worker actions during periods of helper inattention (caused by factors such as distraction, interruption, or assisting other workers) and providing task summaries.
- Exploring alternative collaboration setups: Although our system was designed with a specific worker-helper configuration in mind, other configurations can be explored. For instance, mutual collaboration (where both collaborators engage in physical tasks) or experts providing remote assistance to multiple workers present interesting scenarios that make up an exciting research space for robotic camera systems.
- Providing physical assistance: The robotâs actions can go beyond supporting shared visual information for collaborators to include providing physical assistance to the worker. Manipulation actions by the robot will introduce questions around safety and arbitration, which present interesting avenues for future research.
- Introducing new modalities: While we currently employ screen interfaces for controlling and displaying shared information, there are exciting possibilities for incorporating head-mounted, natural language, and haptic interactions into this context.
If you are interested in learning more, trying out a demo, or exploring potential collaborations, please contact me (Pragathi Praveena) or my co-authors: Haoming Meng, Yeping Wang, Emmanuel Senft, Michael Gleicher, Bilge Mutlu.
This work was supported by a NASA University Leadership Initiative (ULI) grant awarded to the University of Wisconsin â Madison and The Boeing Company (Cooperative Agreement #80NSSC19M0124), and an NSF award 1830242.