Imagining the future of remote collaboration with robots

Periscope is our prototype robotic camera system that allows two people to collaborate remotely
on physical tasks. Photo Credit: @wisalumni

📢 Check out our paper! We had a presentation and demo at CSCW 2023 🤖.

Technologies that support remote work have advanced significantly in recent decades, and they were indispensable during the COVID-19 pandemic. However, remote support for jobs that involve physical work, such as nursing or factory work, was rare. This highlights opportunities for the use of robots that can leverage their physical embodiment to extend people’s abilities in remote spaces.

Advancements in robotics have enabled collaborative robots (or cobots) that are safe, easy to use, and capable, thus opening up a promising space for designing systems that support remote human collaboration. In this post introducing our research, we present Periscope, a robotic camera system that we developed using a cobot platform to enable new capabilities for remote collaboration on physical tasks.

What motivated the design of Periscope?

My grandma is very tech-savvy, but most of us have probably had this conversation with a co-worker, friend, or family member at some point! Credit: Pinterest

Remote collaboration on digital tasks, such as co-editing a document, delivering a presentation, or collaboratively playing a video game, is fundamentally different from physical tasks such as furniture assembly, cooking, or appliance repair. Physical tasks require people to look at physical objects from various angles and in varying levels of detail.

Physical tasks require dexterous cameras that allow people to look at physical objects from various angles and in varying levels of detail. Alt text: Dynamic view from a robotic camera viewing a workspace with a person and a control box from various angles and in varying levels of detail.

Cobots, equipped with cameras, have a wide range of motion that allows for precise camera control for complex physical tasks. However, this improved capability also increases the complexity of camera control. Thus, despite their potential, cobots with high kinematic capabilities — in contrast to traditional tablet-on-wheels telepresence robots — have rarely been utilized in systems that support remote collaboration. In developing the Periscope system, we tackled the challenge of designing camera control that enables the use of highly capable robotic cameras for remote collaboration.

What is Periscope?

With Periscope, a “worker” can complete an assembly task with guidance from a remote “helper” who views the workspace through a robot-mounted camera. We use a shared camera control approach in which the worker, the helper, and the robot all contribute to camera control.

The Periscope system (shown in the figure above) supports a “worker” (figure: left) in completing physical tasks with remote guidance from a “helper” (figure: right), who views the workspace through a robot-mounted camera. The camera view is displayed on a screen interface for both the worker and the helper, enabling them to share dynamic, task-relevant visual information and develop a mutual understanding during collaboration.

The Periscope user interface consists of a camera feed that displays the live video feed from the robot-mounted camera, accepts mouse input commands, and can be annotated using the annotation toolbox; a 3D view that shows a visualization of the robot and its surroundings; a video conferencing panel for verbal communication between the helper and the worker; and a control panel for mode selection.

The illustration above shows the different panels in the Periscope user interface, and below is a brief demo of two people remotely collaborating on a circuit assembly task using Periscope.

Demo of two people remotely collaborating on a circuit assembly task using the Periscope interface. Please note that there is audio accompanying the video.

What is shared camera control?

Previous research suggests that in collaborative scenarios, both the helper and the worker should have the ability to control the camera view at different points (e.g., to provide guidance or ask questions). However, users may find it challenging to operate a robotic camera with advanced kinematic capabilities. To alleviate this, we implemented some level of automation in the camera control to make it more user-friendly. It is important to note, however, that complete automation of camera control is not desirable either, as users will still want the autonomy to choose and view content according to their high-level collaboration goals.

Our challenge is to design an interface that effectively mediates the interaction between two people by achieving a balance between human control and robot assistance in managing the camera view.

Our shared camera control approach empowers both collaborators to independently control the view using mouse inputs and gestures, while also allowing the robot to reduce their effort through autonomous behaviors like hand tracking and collision avoidance. We implement a leader-follower paradigm to arbitrate the inputs from the three entities — the worker, the helper, and the robot. Thus, Periscope offers three modes, where each mode is led by one of the three entities, with the other two exerting less influence over the camera view (see figure ↓ for details).

We implemented three modes in the Periscope system: helper-led mode, robot-led mode, and worker-led mode. In each mode, the terms ‘low’, ‘medium’, and ‘high’ denote the level of influence that each entity exerts over the camera control. For example, in the helper-led mode, the helper has the most influence on the view, followed by the robot, and least by the worker.

The results from our user study supported our hypothesis that each mode would be beneficial at different stages of the collaboration process (refer to our paper for an in-depth analysis of user interactions). The figure below exemplifies a workflow and demonstrates the rich and dynamic interactions facilitated by Periscope. The most encouraging result was the frequent transfer of view control among the worker, the helper, and the robot. Each entity often had ownership of the point of view when they had ownership of a task or relevant information. Thus, with our leader-follower paradigm for arbitration, the leader drove the task forward based on information they possessed, and the follower followed suit.

An example of the rich interactions enabled by Periscope. Here, Mode 1 is the helper-led mode, Mode 2 is the robot-led mode, and Mode 3 is the worker-led mode.

How can we build on this?

With Periscope, we developed an interface on a cobot platform that enabled the use of a highly capable robotic camera for remote collaboration. Some potential extensions to our system that point to future research include:

• Improving robot initiative: The robot may take on a more proactive role, such as monitoring worker actions during periods of helper inattention (caused by factors such as distraction, interruption, or assisting other workers) and providing task summaries.
• Exploring alternative collaboration setups: Although our system was designed with a specific worker-helper configuration in mind, other configurations can be explored. For instance, mutual collaboration (where both collaborators engage in physical tasks) or experts providing remote assistance to multiple workers present interesting scenarios that make up an exciting research space for robotic camera systems.
• Providing physical assistance: The robot’s actions can go beyond supporting shared visual information for collaborators to include providing physical assistance to the worker. Manipulation actions by the robot will introduce questions around safety and arbitration, which present interesting avenues for future research.
• Introducing new modalities: While we currently employ screen interfaces for controlling and displaying shared information, there are exciting possibilities for incorporating head-mounted, natural language, and haptic interactions into this context.

If you are interested in learning more, trying out a demo, or exploring potential collaborations, please contact me (Pragathi Praveena) or my co-authors: Haoming Meng, Yeping Wang, Emmanuel Senft, Michael Gleicher, Bilge Mutlu.

This work was supported by a NASA University Leadership Initiative (ULI) grant awarded to the University of Wisconsin — Madison and The Boeing Company (Cooperative Agreement #80NSSC19M0124), and an NSF award 1830242.