TrackTrack: how the Digital Factory won a hackathon using AI
This article was written by Raissa Garaichina, Baptiste Rondeleux, and Yichang Wang.
A team including three members of TotalEnergies Digital Factory (TDF) and two members from OneTech (OT), Refining & Chemicals (RC) has recently been awarded the first place in the first-ever Hackathon organized by TotalEnergies OneTech branch. Our application, TrackTrack a.k.a. Track², has received financial support from OneTech, which will aid in its development into a practical solution. In this article, we will explain our organization during the hackathon, and the technology behind TrackTrack a.k.a. Track².
What is OneTech Hackathon?
OneTech Hackathon is an internal event at TotalEnergies, with the primary goal of helping, through innovative digital solutions, to better measure, save, produce and distribute energy. The challenge involved solving business problems, driving the development of new product features, and igniting the participants’ professional skills and innovation. OneTech provided excellent logistical support during the Hackathon, including four data science-related training sessions. Moreover, this Hackathon served as an excellent platform for connecting people across TotalEnergies departments.
The Hackathon kicked off on September 12th, 2023, and lasted uninterrupted for 36 hours at the Bougival Campus near Paris (France). Ten teams participated to tackle ten of the most valuable projects selected by OneTech. The participants came from diverse backgrounds, including software developers, data scientists, process control engineers, product managers, and more. During the event, we even had the opportunity to meet TotalEnergies board members, who tirelessly cheered on the teams late at night.
The application: TrackTrack a.k.a. Track²
Our project, TrackTrack a.k.a. Track², is taking shape in the Refining & Chemicals (RC) branch. More specifically, it focuses on transports within refineries, in which freight wagons can circulate to ensure safe deliveries to key points on these industrial sites. TrackTrack is an AI-and-cloud-based digital solution designed to recognize freight wagons from photos, along with the integration of train positioning and other functions. This solution aims to enhance traceability of freight train carriages in refineries to improve safety and delivery quality.
We worked well together: in the first 24 hours of the Hackathon, we communicated constantly with business members and improved our model until we achieved a 100% accuracy rate in wagon ID recognition on the 15 test photos provided. The photos were distortion-free and reflection-free because of a well calibrated camera. The model also did well on printed photos, even on the hardest image with some twist and over 100 characters (see Figure 1). In parallel, we integrated this AI module into an application deployed to the cloud and accessible to users via a mobile app. On the second day, we conducted user interviews and demonstrated the product with printed wagon plate pictures. Three users from different refineries expressed strong interest in our solution.
Our application consists of a front-end and a back-end. As shown in Figure 1, the front-end offers a user-friendly interface that enables users to take a photo of a wagon plate. Subsequently, it allows users to validate related information such as the wagon ID, the wagon’s location within the train station, the product it carries, and the bogie certification date (i.e. carriage underneath a rail vehicle, on which the axles are mounted), among other desired details. As illustrated in Figure 2, the back-end is composed of two parts: a wagon number identification pipeline and database storage.
The artificial intelligence component of our application focuses on wagon number identification, which is an object detection task in computer vision. Our goal is to accurately locate and identify a 12-digit UIC wagon number (for more information, see UIC classification of goods wagons).
During the development of our algorithm, we encountered several challenges, primarily due to the significant variation in wagon number plates. We had considered and tackled all these edge cases:
- Based on the UIC wagon number norm, the position of a wagon ID can appear anywhere on the image, and the wagon ID can be presented in either a single-line or a three-line format,
- On one number plate, we may have at most a hundred characters, among which some telephone numbers can be misidentified as a wagon number,
- The torsion of an image due to the installation of the camera.
Our machine learning pipeline is composed of two parts: optical character recognition (OCR) and post-processing with clustering techniques to obtain the correct bounding boxes. We tested three different OCR packages: TesseractOCR (from Google), PaddleOCR (from Baidu), and Azure OCR services (from Microsoft). We found that, comparing to TesseractOCR and Azure OCR services, PaddleOCR performed better on distorted images. Due to time constraints, we opted for Azure OCR service for quicker integration with Azure cloud.
As shown in different colours in Figure 3 (Left), we created clusters of different font sizes in the image as candidate wagon numbers. Azure OCR service provided results with fine granularity, whereas we needed to consider the entire UIC number as a single entity. To address this issue, we merged the bounding boxes within the same cluster, as shown in Figure 3 (Right). This merging process is based on an elastic threshold that varies between clusters. For each cluster, we apply a post-processing technique to obtain the true UCI wagon ID.
As a product of a 36-hour hackathon, TrackTrack a.k.a. Track² is not perfect. We could apply an elastic threshold to fuse the clusters, because the wagon plate images provided during the hackathon were quite rectangular, with minimal torsion.
We could also enhance the robustness of the ML model in our application to reduce the effort required during the deployment of cameras on the train station. The potential solution includes:
- Applying and integrating PaddleOCR into the backend of the application, allowing the torsion to be addressed automatically,
- Identifying the torsion through linear regression based on the coordinates of the corner points of the bounding boxes, followed by a perspective transformation on the image to correct the torsion.
Conclusion
TrackTrack a.k.a. Track² caught the attention of the jury, who underlined and appreciated our ability to fine tune the model, to deploy it in the cloud and to build a real mobile app using this model, all within a truly short period of time. The product thus won first place in the Hackathon, a great reward for our team!
What’s next? Moving forward, Nicolas, our “business side” teammate, will lead the implementation and scaling of our project to different TotalEnergies refineries.
The success of this Hackathon is not only ours, but also that of TotalEnergies Digital Factory. We would like to express our gratitude to Michel Lutz (TDF Head of Data+AI) and Frederic Gimenez (TDF Managing Director) for encouraging us to participate in such projects, Nadège Brun for organizing this amazing event, and our teammates, Barbara Sansevero Lima and Nicolas Petit, whose business vision was critical for our success. Winning the Hackathon has left us with a sense of accomplishment, and we look forward to participating in more hands-on and business-driven hackathons in the future, contributing further to outstanding innovations and solutions to support TotalEnergies’ transition.
