Computer vision industrial and logistics applications
Computer Vision is a technology that allows computer systems to analyze images, including video. This area is becoming really popular, given the increasing amount of data and algorithms available for image processing. Computer vision is a key part of developing autonomous vehicles, industrial robots, and other scenarios that require the same visual analysis ability that humans have.
Mining and natural resources processing are becoming increasingly complicated, involving operating in extremely harsh conditions. Whether it’s a coal or minerals mining at a depth of several kilometres or oil wells drilling at the bottom of the sea or ocean, people performing these jobs are at serious risk. It is much more preferable to replace human labour in hazardous conditions with machines.
By using artificial intelligence (AI), industrial companies can make a leap in that direction. Industry 4.0 means the transition to a fully automated digital production, controlled by intelligent systems in real-time while constantly interacting with the environment.
At the heart of this concept is the creation of “digital twins”, which refers to making digital replicas of actual physical assets. A prerequisite for a successful implementation of the Industry 4.0 concept is the installation of sensors to collect the data about processes for further analysis. In this context, video cameras are among the most expensive sensors. Computer vision systems make it possible to obtain a visual representation of real objects, processes and analyse them late on to solve various applied tasks.
CV has a lot of practical applications in industry, ranging from product quality control on production lines to monitoring of safety policies compliance. Here are a few of the most versatile industrial case studies in the computer vision field.
Production quality control
Results of human visual inspection largely depend on the capabilities, experience and attentiveness of an operator. This pretty laborious and intensive process leads to natural consequences such as omissions or misclassification. It is also important for companies to ensure transparency of the process and ensure that the results of inspections are recorded for subsequent analysis and production upgrades.
In order to mitigate human influence, improve the accuracy and reliability of the quality control process, computer vision systems are used. The use case can be divided into two parts: the control of semi-finished material and the inspection of finished products at the end of the production cycle. With the help of neural networks, it becomes possible to detect 92%-99% of all defects, depending on the task; false positives account for 3–4%. The normal level of defects in different industries varies from 0.5% to hundredths of a per cent. Such performance rates are a good reason to consider replacing a human operator responsible for detecting these defects.
Missing a defect may lead to significant costs, so industrial players have already been working on this case. In addition to visual analysis, there are other methods of non-destructive testing, such as ultrasonic, eddy-current and x-ray control. In addition to traditional cameras, it is possible to use information about surface temperature and geometric data of the subject.
Industrial safety
Modern computer vision systems are capable of identifying potentially hazardous situations at industrial assets. Tracking events according to specified parameters allows to minimize the number of dangerous incidents, ensure the constant equipment operation and reduce the risk of industrial injuries. By using cameras and data from other sensors from a production site, robots and machines are able to work together safely.
Among the main scenarios in which industrial computer vision systems are used there are the control of personal protective equipment wear (helmets, safety cables, overalls, headphones) and personnel presence in hazardous areas. Such a solution automatically detects violations and provides sound feedback to personnel, thereby creating a safe working habit. If a person appears in a hazardous zone, an alarm is triggered.
With the help of video analytics, one can also detect an open fire, pipe bursting, spills, identify breaks in fencing or any attempts to carry items outside the guarded zone and detect abandoned objects. It also becomes possible to track smoking people or mobile phone use in certain areas (such as gas stations, etc).
Another scenario for computer vision use is fatigue detection. The system tracks employee activity and productivity, resulting in a task assignment management improvement. These solutions are relevant to those industries where continuous production is in place and night shifts are established.
Operations control
Computer vision is used to monitor production facilities and infrastructure. The capabilities of video analytics range from detection and localisation of moving objects and vehicles, as well as equipment and persons’ location. Different activities might be recognised and tracked with precision, surpassing human abilities. Based on this monitoring, smart task assignment might improve productivity and increase overall equipment use.
Old equipment digitalisation
The data collection step is often linked with the presence of old equipment. In this satiation, embedding modern sensors is not always economically feasible. Equipment digitalisation and saving on re-equipping becomes possible thanks to the ability to read characters directly from the screens using computer vision techniques.
One of the most popular areas related to image processing tasks is the quality check of finished products.
The mining industry has its own peculiarities and respective scenarios of computer vision use. We are witnessing the emergence of autonomous vehicles dictated by the need to keep humans away from a hazardous environment. But these trucks still operate amongst other machines and personnel. The principles of computer vision are actively used to analyse constantly changing situation around. A surprising side effect of computer vision application is that there is no need for dump trucks to turn around: with 360-degree vision, they are perfectly capable of driving in reverse gear.
Strategically, everything tends toward an orchestration of the entire production cycle, including mining and loading material into haulage trucks. The operator functions become limited to issuing expert instructions at the start of the work shift and backing the machines up in extraordinary situations so that a single operator becomes able to manage 3–5 robotic machines.
Ore mining. We now can classify, count, and estimate the size of moving chunks of ore rock during mining and transportation. Rock fragments analysis makes it possible to adjust blasting operations. Such systems lead to an increase in process productivity by 3–4%. Optical granulometry provides a real-time understanding of the work done. A related task is to monitor rock fragments on conveyor belts in order to detect outlier objects in crushing and sorting plants.
Ore fragments size estimation fed into the equipment for subsequent processing allows to automatically adjust the crushing machine mode and achieve an optimal yield. Systems designed to automatically classify the rock type after drilling process, help to identify the minerals set found more quickly and accurately than humans.
There are also monitoring systems that determine the presence of bucket excavator teeth in real-time mode. The loss of just one bucket tooth decreases excavator productivity by about 1.3%. Moreover, if the tooth reaches the crusher, this may cause damage and downtime equal to $8,000 per hour, excluding the searching and repair costs. An audio signal to the operator in case of loss and breakage minimises these negative effects.
Access to hazardous areas using unmanned aerial vehicles: companies use drones to monitor their assets and operations, quarries and water dams and detect leaks in the piping infrastructure. There is also a solution that allows analysing the structure and condition of abandoned underground mines to control subsidence and water pollution in unsafe conditions. With the aid of this technology, a geotechnical engineer can better restore the mine map and monitor its current state safely.
In metallurgy, computer vision has the potential to control the quality, determine the microstructure and mechanical properties of alloys and search for new materials with the desired characteristics. It has been proven that machine learning and a reasonable involvement of experts complement each other perfectly solving the task of alloy assessment.
The design and development of materials took several decades in the past, from initial discovery to commercialization. With the use of stored laboratory data, computer vision has the potential to discover materials, design and predict their properties. Since grain structure affects steel cracking, visualizing fractures might be used to link macroscopic mechanical and microscopic structural properties in order to predict crack propagation paths.
The art of defect detection in metallurgy has its unique nature, involving the need to use additional analytical tools in addition to traditional cameras. For example, a surface temperature assessment with the colour analysis can reveal the levels of contamination with unwanted minerals, as well as process or reactor conditions. In particular, neural networks have been successfully trained to recognize perlite, ferrite, martensite and cementite.
Robots
Over the past few decades, the capture and processing of different objects by industrial robots almost replaced manual labour. The first models of industrial robots were designed to perform simple tasks. Now robots and cobots are able to relocate objects with grabber or vacuum cups. They spray paint, bend wire, perform spot welding and do other routine operations. Modern robots augmented with computer vision systems can perform tasks characterized by a significant position variability of pieces.
Robots have been used in industry for a long time, but AI just recently started to penetrate this field. GPU computation for industrial applications is a fairly new area as historically, industrial computer vision involves a range of sensors, not just a video stream. Robots with computer vision systems are able to take into account the location of objects by analyzing the video stream from a 3D-cameras as well as laser and sensor data. This approach allows robots to perform tasks with high accuracy in almost any conditions.
Robots with computer vision require less programming. They should be configured only once before being launched. Moreover, robots can also seamlessly switch between tasks with virtually no downtime. Due to their high flexibility and almost no need for programming, computer-vision robots do not need to know the exact parameters of parts to do the work productively. The adaptability of robots allows them to select parts needed, localise them and grad them from any position. Small investment amount in equipment setup is also one of the most obvious benefits.
Dex-Net 4.0 is a robot with an arm, grabbing objects with incredible accuracy. It is able to capture 95% of objects at a speed of up to 300 objects per hour. It assesses several options to capture an object in a matter of seconds. Humans are able to capture from 400 to 600 objects per hour. Computer vision continues to expand the capabilities of industrial robots and find new ways to increase the productivity of routine tasks.
With the development of e-commerce, logistics robots became a promising robotic field. They serve the delivery of goods and cooperate with humans. They can transport goods weighing up to 15 kg, navigate in difficult conditions and effectively manoeuvre among people. Autonomous mobile robots can work without supporting infrastructure, such as markers, wires, magnets or precisely located targets. They become an important part of the production environment as they increase productivity and reduce costs. Warehouse robots allowed Amazon to reduce its operating costs at each warehouse by about 20% (around $22 million in annual savings). According to the McKinsey Global Institute (MGI), overall operational cost savings due to the various operations automation can range from 15% to 90%, depending on the industry.
When planning robot manoeuvring, computer vision is used to avoid collisions. The data source for the algorithms is a lidar mounted on a robot. Objects are first detected and then movement tracking begins. Robots need to understand the environment and predict the movements of others, which is indispensable when moving in a changing environment. Navigating and avoiding obstacles; remembering and taking into account the path travelled and localising oneself in space — all these tasks are solved applying computer vision principles.
Several types of logistics robots already exist, as well as complex solutions to automate warehousing, including industrial manipulators, mobile robotic carts and palletizers. The new Handle robotic loaders were introduced by Boston Dynamics in March. They are faster than forklifts and are able to sort out the contents of pallets and transfer storage units to a conveyor. Each of these two-wheeled balancing robots is equipped with a manipulator and a vacuum grip, as well as a computer vision model, which allows it to navigate across the warehouse and select the desired shelves and boxes. Advancing computer vision and gripper will expand the range of applications for logistics robots.
