Building Scalable Edge AI Deployments with ‘FleetTrackr’

SmartCow
6 min readJan 19, 2022

--

The era of AI is changing the future of edge computing, and Edge AI is now at the forefront of this ecosystem due to several key factors like faster decision-making, low-latency processing, and low resource or bandwidth reliability.

Given the wide array of advantages, the AI industry is adopting Edge AI technologies for all large scale projects like Smart Cities, Intelligent Traffic Management Systems (ITMS), industrial automation, safety and security systems, law enforcement, and so on.

While edge AI systems are scalable and versatile, they continue to be a daunting challenge for System Integrators (SIs) and Managed Service Providers (MSPs) to build and manage large networks of Edge AI systems. Since Edge AI systems are spread across multiple locations and run different AI algorithms on them, and given that these algorithms dynamically change as needed, it is imperative for the deployer to be able to manage these devices remotely. The operations management for Edge AI systems can be two-fold:

  1. Initial device registration and installation.
  2. Regular device management and maintenance.

At SmartCow, we have developed a fully functional AIoT device management platform called ‘FleetTrackr’. This can be used to easily manage and maintain large-scale Edge AI deployments as it allows administrators to provision, manage, monitor, and update thousands of devices securely and entirely over-the-air. It offers simplified deployment and centralised management of Edge AI systems though a hybrid-cloud service.

FleetTrackr is designed to help manage and scale deployments of Edge AI systems built using NVIDIA’s Jetson edge devices. This fleet management platform allows IT departments to securely and remotely manage large-scale deployments. Instead of spending weeks planning and executing deployment scripts, administrators can upgrade their AI solutions, add or delete application containers, flash system firmware, streamline operations and administrative tasks, and monitor the health metrics of devices spread over vast distances from a single control panel.

FleetTrackr is also cloud native by allowing device flashing via Firmware-Over-The-Air (FOTA) functionality, adding SSH Keys and managing device passwords remotely, managing device security certificates, and flexible integration with the client’s CI/CD pipeline.

Some of the key advantages of using FleetTrackr are:

  1. Negligible Configuration Requirements — Deploy thousands of AIoT devices on the edge with next to no configuration. Get your systems up and running with a single-click.
  2. End-to-End Security — Never worry about security, right from the edge to the cloud as FleetTrackr offers complete end-to-end security for all AIoT communications.
  3. Seamless AIoT Device Management — Scaling from 10 devices to 10,000 devices is no longer a challenge with complete remote management.
FleetTrackr — Unified Device Management Dashboard

There are three key components in FleetTracker provisioned by the FleetTrackr UI:

  • Device Management — Easily monitor device metrics and device status, get access to crucial device performance KPIs based on historical data. Access device documents, specification sheets, and user manuals for ease of deployment.
  • Container Management — Update or restart your container with a single click and create a group of devices to provision containers easily.
  • Issue Management — Group edge devices and raise tickets when a device goes faulty, create and manage site leaders and teams to resolve tickets, schedule regular device maintenance tasks, and get access to KPIs on ticket history.
FleetTrackr — Feature Set

At the heart of FleetTrackr sits an application called ‘Tegra Producer’. This is a lightweight metrics shipper for the Jetson edge devices. Tegra Producer is installed on thedevice before deployment, and it sends all its observed metrics such as device temperature, CPU, GPU and RAM usage, uptime, board component temperatures, storage details like eMMC and NVMe, process statuses of hardware accelerators, encoding and decoding, management of SSH connections, and certificates and communication to the front-end for issue management.

Tegra Producer is also responsible for managing and maintaining several key functions related to the edge device like:

  1. Monitoring and automating failover for multiple data storage nodes.
  2. Maintaining a web-based repository manager.
  3. Accessing advanced database management systems.
  4. Dynamically adding and removing data/stream pipeline nodes.
  5. Securing artifacts with policies and role-based access control.

The Edge AI devices run multiple AI applications, and these applications require regular updates either to upload an optimised model or to deploy a different use case. To seamlessly provision software updates, FleetTrackr relies on an ‘Update Client’. The update client is an application that is installed on the edge device before deployment and communicates with the container repository to perform scheduled and immediate software updates when triggered from the FleetTrackr UI.

Just like the requirement for regular software updates, the need for re-flashing devices arises with updated firmware or software kernels. A large number of heterogeneous devices forming an IoT network should be capable of running 24/7 unattended. Certain situations may demand that these fleets of IoT devices are reprogrammed with newer versions of device kernels or security updates. Reprogramming a group of devices is a challenging task when this cannot be done remotely. It is a rather complicated procedure when a device needs to be dismantled from its edge location, transferred all the way back to the vendor’s factory to be re-flashed, and then transferred back to be re-installed. This process is almost impossible when thousands of devices need to be re-flashed on the edge.

For ease of maintaining the fleet, FleetTrackr comes with it’s own FOTA functionality, allowing users to program, reprogram, and push updates remotely using the FleetTrackr UI. All the client needs to do is to create a group of devices to be flashed, select the Kernel/OS from its path and click on ‘Flash’ to reprogram all the devices in the desired group with the new software. The back-up firmware is also stored on the device storage or NVMe for quick re-flashing.

FOTA updates through FleetTrackr UI

As we have gone through the key features of the FleetTrackr, here is a look at the end-to-end data flow architecture.

The last part of this blog covers one of the most important aspects of FleetTrackr, i.e. Provisioning. One-click Provisioning is one of the most sought-after features from SIs and MSPs that would cut down their time-to-deployment by over 1/4th. First time registration, configuration, and installation of the hundreds of thousands of devices is very challenging and can take up to several weeks to complete. They face an added challenge when they attempt to scale up, which requires obtaining additional devices and deploying them whilst managing and maintaining existing devices. FleetTrackr provides easy provisioning in five simple steps:

  1. Identify and validate all Jetson devices.
  2. Generate SSH keys and secure certificates for all devices.
  3. Update hostnames/passwords if required.
  4. Deploy the native Tegra Producer app onto all Jetson devices.
  5. Add device startup commands.

With this, the device is registered, validated, configured and ready for deployment, and all of this is controlled from the FleetTrackr UI.

Provisioning Workflow

Conclusion

While it is easy to plug-in, power-on, and deploy tens of devices, managing thousands of Edge AI systems is an ever-growing task. Device OS, AI applications, firmware, and other software need to be continuously updated to keep the edge systems running seamlessly. The end-customer also needs a unified platform that addresses these challenges and makes it easier to manage and maintain these edge devices. FleetTrackr is thus designed to cater to these specific functionalities that meet the end-customer’s needs. With FleetTrackr being used in several large-scale deployments, it is always updated with state-of-the-art features that make Edge AI deployments faster, more scalable, and more reliable.

About the Author: Pooja Venkatesh, the Product Manager manages end-to-end product development at SmartCow. She comes with experience in deep learning and computer vision. Pooja is also a professional Indian classical dancer, and enjoys traveling and reading during her free time.

--

--

SmartCow

SmartCow is an AI engineering company that specializes in advanced video analytics, applied artificial intelligence & electronics manufacturing.