How is Ocado Technology using automated solutions to build robots?

Nikolay Angelov, Engineering Manager — Bot Availability Systems, Ocado Technology

At the heart of Ocado’s automated warehouses lies ‘The Hive’ — a grid storage system home to thousands of mobile robots known as ‘bots’ which collaborate to pick grocery orders.

Such is the complexity of automating grocery fulfilment that these bots need to operate at peak performance, with downtime virtually eliminated. They operate in swarms of thousands, whizzing within 5mm of each other, and at speeds of up to 4m/s.

Each bot covers 60 km per day and can travel the distance from London to New York before having to be taken off the grid for preventative maintenance.

Building thousands of bots that consistently set new benchmarks for efficiency and performance is a challenging task.

Rigorous testing is embedded in the manufacturing process through multiple automated solutions developed in house — from parts testing in the manufacturing process, through to ‘wake up’ inspections and performance examinations.

Testing parts on the production line

Our bots consist of hundreds of different mechanical and electrical parts which require extensive testing before being assembled into the complete machine.

Testing tools vary from test boxes for a control board to specialised test rigs for different bot peripherals. These tools aim to provide as much test coverage of the board’s electrical and communication interfaces as possible, by using bot firmware and emulating the external environment from a specially designed interface board.

Once the manufacturing engineers have tested the parts and have fully assembled the bot, they are ready to ‘wake it up’ for the first time.

How do you ‘wake up’ a robot?

The bot is placed on a micro-grid and powered on for the first time so that the engineers can complete an initial inspection.

This involves flashing and configuring software to each sub-component, a process managed by a designated component called a BotPC (a chip running Linux built into the bot). This Linux system is connected to all sub-systems and communicates through widely used standards such as CAN, USART, and SPI. This means that it can be used with multiple tools without the need to install special software on operators’ laptops.

Manufacturing engineers also run initial checks on the sensors. For these tests, they use both physical tools and a command-line utility available on the BotPC.

The applications provide a simple console interface that allows engineers to execute single commands to the bot as well as predefined automated scenarios which exercise each of the sub-systems and check that the hardware behaves as expected.

Once this is confirmed, the bot is ready for the next stage — performance verification.

Performance verification: from on-bot automation to cloud solutions

After initial wake up verifications, which confirm basic bot functionalities and hardware behaviour, it’s time for the real challenge: performance verification testing. Engineers put the bot on a test grid with all the necessary ecosystem equipment to prove that the bot’s performance is optimum.

To guarantee that our tests are as close to warehouse conditions as possible, we use a cloud application that communicates with the BotPC. The Cloud application makes remote support much more manageable and facilitates deployment all around the world.

We do this performance test through a Java application that interacts with an application located on the bot via WiFi. This test continues for about 45 minutes and executes multiple operations, similar to the ones required for warehouse tasks.

Our on-bot applications not only provide testing but also diagnostic capabilities. If a bot underperforms during any of the tests it can effectively self-troubleshoot. It provides engineers with in-depth diagnostic information about the state of its peripherals, voltages and sensors at the point of failure. It can also send an error message which states the specific issue and highlights which condition deviates from the expectations of the control software. This is essential for the efficient build process of such a complex physical system.

After the test, the BotPC uploads logs to a cloud, where a Python application processes and verifies the data produced during the procedure. This data includes everything from sensor information to the program flow. Once data analysis is complete, the Python application sends a report to the engineer who can confirm that the bot is ready to ship.

Testing the hardware using digital twins

In the physical world, testing software solutions on real hardware is crucial. This is because varying conditions — from temperature and voltage, through to resistance, noisy connections, the tightness of bolts and vibrations — can cause disparities in the system behaviour.

The same is true when you are testing hardware and complete physical systems. It is essential to use up-to-date software in conditions that are as close to the operational environment as possible to ensure that verification results are comparable with real-life bot activity.

To achieve this, we have developed platform software, which includes many tools, mainly written in C. The digital solutions are synthetically verified through unit tests as well as system-wide synthetic testing using in-house simulated digital twins of the bot written in Java.

This additional synthetic layer of testing allows us to predict behaviour before it even changes in our firmware or reaches a physical device.

Why do we use C and Java?

C is the most commonly used programming language for embedded systems. It enables high performance and reliability, which are crucial for real-time embedded applications running on limited hardware resources. Furthermore, there are many compilers for microcontrollers that enable software engineers to quickly develop portable code. C allows us to use the object-oriented programming approach depending on the need and the application.

Java is one of the most widely used and high performing programming languages in the world. It has a mature ecosystem with many libraries and frameworks which significantly speeds up the development process. Last but not least, it allows us to run our digital twin bot in a Java-written simulated warehouse environment.

Of course, Java and C are not the only tools in our arsenal. We also use SQL, bash scripts and Python to automate manual everyday tasks or analyse and understand the data which we collect from our bots. This is critical because the data provides an objective view on performance.

Create the best solutions for a problem — don’t be limited by your tools

Many companies and software engineers restrict themselves to a single programming language or only attempt to find solutions using their existing tool kit.

Sometimes I get asked, “isn’t it too much to expect an engineer to use such a diverse technical stack?” But my colleagues and I see this rich technical stack (programming in C for embedded systems, Java application development, cloud applications, data analysis with SQL and Python along with the need for system-wide electrical understanding) as an opportunity rather than a burden.

We think of ourselves as being enabled to deliver the right solution, using the right tools, with the freedom to experiment, make data-driven decisions, and learn from mistakes in order to provide the best solution for the problems we have to solve. This is a precious advantage which isn’t possible in many organisations.

Originally published at https://www.ocadogroup.com.