How we built an AI-driven eWaste Visual Inspection Demo Machine

8 min readOct 24, 2022

[Although I work for IBM, the mix of letters and images as they appear in this article is a mix of mine and does not represent IBM official views in any way]

For the Telstra Vantage 2022 conference last month, I had the chance to team up with a bunch of great colleagues (see Thanks at the end of this article) to think about, then build one of those crowd-attracting demo systems for IBM’s Let’s create a sustainable future together booth. After some brain storming and a lot of team work, we decided to create an AI-driven, automated eWaste Visual Inspection Machine using a mix of cutting edge technologies including IBM Maximo Visual Inspection (MVI), MVI Edge and IBM Maximo Monitor, all part of the IBM Maximo Application Suite (MAS).

The collage below shows what the end result looked like — with the main demo booth on the left picture, an alternate mini booth in the middle one, some of the key demo screens on the bottom-right pictures, and on the top-right image … that’s me in the white shirt demo’ing the system to an enthusiastic crowd (haha!).

In this article, I will first share the story line behind this demo system, then present a summary of what the system is, how it works, what it does, and why I think it’s smart.

The demo story

The eWaste material that we wanted to visually recognise then inspect included 12 types of old or broken objects like USB sticks, chargers, earphones, modems, iPhones, Arduino boards, cables, batteries and so on. The ‘bad guy’ that we wanted to ultimately track was the lithium contained in many of those objects, especially in the Battery_Rechargeable and Battery_3V ones.

The story behind the whole demo flow was something along the line: “Automatically send an actionable alert when an anomaly is detected in the mass of lithium treated by the conveyor belt”.

This picture highlights the 12 detected objects on the right side, and, on the left side, the totals, anomalies, trends and alerts that we wanted to be informed of regarding the mass of lithium treated during the conference.

What the demo machine is and how it works

The hardware elements that we ordered to build this demo machine included (a) a big variable-speed VEVOR Conveyor Belt and an alternate, small Belt Conveyor, (b) a Topodome RTSP Wifi IP Camera mounted on the conveyor belt and having a stable vertical view on what unrolls on it, (c) a Jetson TX2 NX AI Board with a GPU, (d) a TP-Link AC750 Nano Travel Router, (e) the 12 aforementioned eWaste objects.

On the software side, we used the Visual Inspection and Monitor applications of a Maximo Application Suite running on IBM Cloud, and installed MVI Edge on the Jetson board.

More schematically, this picture shows how the IP camera continuously monitors the conveyor belt and the eWaste objects moving on it, how MVI Edge on the Jetson board connects to and uses the camera as an input source via its RTSP feed, and how MVI Edge interacts with both the MVI Server and MAS Monitor.

NOTE: due to supply chains issues at the time of our orders, we had to restrict our Jetson AI board choice to the TX2 NX version (1 GPU & just 4 GB) vs. the better-fit Jetson Xavier NX (8GB) version— please refer to these official MVI Edge system requirements. To be noted however is that the TX2 NX version behaved like a charm throughout the whole conference, running several TinyYOLOv3 model-based inspections on the slow moving conveyor belts without any issue.

What the demo machine does

This picture summarises the 5 steps that underpinned the demo story, and the subsequent text details what we did at each step:

COLLECT: we initially let the conveyor run for just 5 minutes, period during which we enabled an Inspection in Collecting mode on MVI Edge to trigger one image capture every 0.5 second, and asked our kids to keep randomly putting new eWaste objects on the belt as it moved at various speeds. That sent 600 images to the MVI Server in one go — more than enough to proceed with the next step …
LABEL & TRAIN: based on that initial data set, we started drawing labels on the images to highlight the 12 eWaste objects, quickly trained a first Object Detection model (Faster R-CNN), used it to auto label more pictures, retrained and refined it, then ultimately got to a reasonably accurate and precise Tiny YOLO v3 model, ready to be deployed to the Edge.
DEPLOY: we simply downloaded and deployed the Tiny YOLO v3 model to the MVI Edge running on the Jetson board. Note that as an extra demo step, we also created a simpler Classification model (GoogLeNet) that we deployed to Maximo Visual Inspection Mobile— which allowed us to also take pictures on the fly and display eWaste classification results on our iPhone’s MVI Dashboard.
INSPECT: then we triggered an inspection in Inspect Mode on which we enabled a visual trigger that took a picture out of the movie each time an object appeared. So, each time someone visited the booth for a demo, the objects that we put on the belt were detected, their type inferred, and the results of those inferences sent back to the MVI Server for further model training if required.
MONITOR & ALERT: by using the integration between MVI Edge and Monitor, we were able to push the experiment one step further, i.e. one where we could get actionable insights from the visual inspection data. So, here, we first enabled rules on the MVI Edge Inspection, so to send different MQTT messages to Monitor each time an object (e.g. iPhone, Modem, Battery_Rechargeable or Battery_3V) was detected with a given level of confidence. Then, we were able to analyse our inspection data, to create custom data items (like Mean Daily Count of Batteries Detected or Total Mass of Lithium Treated or Lithium Mass Daily Anomaly Score etc) and functions, or detect anomalies and create dashboards.

This image shows a more logical view of the same system, where the absence of arrows on the connectors highlights how virtuously circular the whole process is, e.g. how all new coming inspection images can be used to further train, fine tune and redeploy the model for better overall inspection accuracy and precision.

Why the demo machine is smart

Although it still is fun to see how an object detection model can run almost continuously and draw boxes around the objects that it detects, what impressed me more with the capabilities that this MAS MVI — MVI Edge — Monitor system unveiled are:

Self sufficient ‘Collection & Inspection Station’: in the past, when I did MVI demos, I always used to ask: “Give me pictures !” of what you want me to detect. Now, such a moving conveyor belt combined with an automated collection inspection allows me to rather ask: “Give me the objects !” that you want the demo to recognise — since the system easily allows one to collect enough pictures for the the labelling and training process. If instead of a conveyor belt, you imagine a drone, it appears that this MVI — MVI Edge combination can also be used ‘from scratch’, i.e. in situations where you may not have any images of what you want to detect yet, but first need to collect them!
Anomaly Detection, Data & AI: because every object detected can send an MQTT message to MAS Monitor, one soon realises that this is a ‘big data’ story, and that beyond the actual object detection itself, what is important is to get actual actionable insights from the massive amount of data generated by the visual inspections. That is where the powerful ‘Data & AI’ capabilities of MAS Monitor come in and enable us via just a few clicks to e.g. “…send an alert when an anomaly is detected in the mass of lithium…”.
Alerts then Service Requests: an extra added value we have here is that the detected anomalies can automatically generate Alerts on which Service Requests can be created in the MAS Manage (aka ex Maximo) part of MAS — that establishes another bridge between the ‘Visual Inspection’ process itself, its continuous ‘Monitoring and Alerting’, and the broader EAM Lifecycle of the given assets being inspected !

Thanks

Although in retrospect, I remember a system that ran smoothly during the whole conference, the reality is that it took quite some time, team effort and brain power to build and run the whole thing.

This was a fun collective work which involved many colleagues — from IBM product development to marketing, from the local client execs and architects to worldwide client engineers, tech sellers and more. While it’s impossible to list and thank all those who helped, I want to send a HUGE THANKS to (at least) these few: Frank Yeh, Poonam Zham, Langley Millard, Michael Dawson, Mike Lamb, Cesar Maciel, Mike Hollinger, Sean Innes, Ross Judd, Dane Gambrill, Alexandra Kelly, Frank Brinner without whom … well, you wouldn’t be reading this article! A special Thanks to Ed Neubecker for having kick started us on the visual edge, both in person in Boston and via his great MVI Guy Youtube channel (to which I encourage you all to subscribe). Another special one to Shuang (Sherry) Yu and Nirandika Wanigasekara (aka the Data Detectives) for having inspired me to try something on Medium.com. And of course, the final one to John MacLeod who, as everyone knows, is the one who actually made every connection and turned every screw between the edge and the non-edge elements here— think we did better with this v2.0 than we did with the pre-Covid Beast v1.0, John!

Thanks for your time and attention, cheers, Christophe