Building the future of autonomous robotics

By Chong Jia Yi, Distinguished Engineer, Smart Nation Platform Solutions, GovTech.

With an ageing population posing long-term challenges to Singapore’s labour force, our Smart Nation Platform Solutions team is exploring technological solutions like smart sensors and autonomous robotics to transform the nature of work, easing the reliance on human intervention for labour-intensive tasks.

One of our major projects that I am working on now is the Digital Operations Smart Services (DOSS), a technology stack that enables the fast and flexible deployment of smart sensor and robotics solutions through a common operations management platform.

DOSS allows us to autonomously run various operations like transportation, delivery, repair and manipulation operations. This helps alleviate Singapore’s manpower shortages in the upcoming years, and meet our goals of becoming a Smart Nation.

With DOSS, we are developing deep learning techniques to power real-time analytics and flexible sensor and robotics solutions for the real world, regardless of the type of robots or sensor hardware used.

One key focus of our work is large-scale outdoor autonomy for robots, which will be a monumental achievement if we pull it off. This is a great challenge to many talented engineers and tech companies because of the unstructured nature of an open environment. For example, for a robot (or any mobile sensor) to move from point A to point B, it has to traverse obstacles such as staircases, roads, and grass fields.

By focusing on solving this problem with autonomy, we don’t have to build extra infrastructure just to cater to robots. If we don’t do large-scale unstructured autonomy, then the types of operations and the terrains that robots can traverse are extremely limited.

In developing DOSS, our team is creating a way to manage and control a wide range of robots and devices without the need to build special infrastructure to cater to the various sorts of operations, which can be costly or unfeasible because of the location.

We use a hybrid approach to develop autonomy, with both traditional navigation and motion planning with Light Detection and Ranging (LIDAR) and Simultaneous Localisation and Mapping (SLAM), as well as more state-of-the-art computer vision (perception-based) techniques. This is all done with deep learning.

Adopting a hybrid approach with traditional methods alongside newer techniques allows us to be agile in making autonomous systems work while ensuring performance stability in different environmental conditions.

Running in the real world

We have already deployed DOSS in the Boston Dynamics’ robot dog SPOT that was trialled in Bishan-Ang Mo Kio Park for COVID-19.

With the DOSS software, the four-legged robot helped with people-counting at the park, broadcast safe distancing messages and detected park visitors who did not wear masks.

SPOT has also been helpful in scenarios where human contact has to be minimised. The robot dog was tested at the Changi Exhibition Centre community isolation facility, where it helped deliver medicine and other supplies to the patients.

While the robot’s appearance may have captured the attention of park-goers, a key innovation is the analytics that our DOSS platform is able to deliver.

We are currently testing a real-time deep learning-based human tracker, which enables a robot to track a person’s movement from a single camera’s real-time video footage, with no other equipment required (such as LIDAR, radar or GPS).

This tracker will allow robots to follow people, say for delivery, and for all kinds of different logistical or critical operations. For example, an operator may be in a tunnel where there’s no GPS, 4G and Wi-Fi. So, the robot has to track the operator as they navigate to a critical point in the building to perform a certain task. This is something that we are trying to achieve.

We are also developing real-time analytics to detect a person’s breathing or heart rate, simply by scanning an image of their face. Such features can be used for search and rescue operations, to help detect the vital signs of people in need.

Developing real-world situational awareness

We are also working on real-time visual scene understanding by training deep learning models to understand the intricacies of what’s going on in a scene.

With scene understanding, DOSS is able to identify different classes of objects, such as a rubbish bins, dog, cat, or person.

The system is also smart enough to “segment” an image. What this means is it’s more than just a box. We can also tell which region of the image has a cat, for example. Using this information, the DOSS platform can also infer distance and depth of each object or region in the scene.

With both region and depth, the DOSS system can then reconstruct a full 3D representation of an environment from a coloured image. So, given a single RGB image, our team can reconstruct a complete 3D environment of what a robot sees.

This is something that is very useful, because you can use it for so many things: architecture, construction, search and rescue, mission planning and others.

The team behind the tech

When I first returned to Singapore and started work at GovTech in October 2019, I was the sole engineer building the DOSS platform for several months.

With experience working at Pixar Studios in the US, I draw on my expertise in physics simulation, animation and deep learning to build capabilities in the nascent field of autonomous mobile robotics and real-time deep learning analytics.

Today, I have a lean and eager team making advances in our work on DOSS. My DOSS team is a multi-disciplinary one that draws on each person’s specialisations in one of three domains: computer graphics, computer vision and machine learning.

Vincent, my lead graphics engineer, has more than 10 years’ experience in the industry. Before he joined GovTech, he was running his own start-up doing real-time video streaming for video games.

Zhen Ling, the team’s senior machine learning engineer, is back in Singapore after four years with Facebook in Seattle, where she was in the company’s core deep learning group.

Robotics engineer Wenchao has a PhD in robotics motion planning and has worked with A*STAR.

Anbang, our full stack engineer, has experience working on backend cloud services, and was responsible for implementing the smart thermal scanner SPOTON version 1.0 core, written in C++.

Part of a nationwide platform

DOSS helps build up key capabilities crucial to the Smart Nation Sensor Platform (SNSP), a nationwide network of sensors and supporting technologies, which will provide the real-time data that is the lifeblood of any Smart Nation.

The DOSS platform can also be tapped on by other government agencies to deploy the robotics and sensor applications they require in a much shorter time, as shown by the use of SPOT and SPOTON during this COVID crisis.

There are many exciting possibilities that DOSS will bring in future, as our team builds up experience developing applications for the real world.

Keen to find out more about DOSS and see how you can co-create with us? Write to us at sensors@tech.gov.sg