AI-based Vehicle Analytics for Smart Cities

Improving cities is a pressing global need as the world’s population grows and our species becomes rapidly more urbanized. In 1900 just 14 percent of people on earth lived in cities but by 2008 half the world’s population lived in urban areas. Today, 55% of the world’s population lives in urban areas and this percentage is expected to rise to 68% by 2050. There were just 83 cities on earth with more than one million residents in 1950, while as of last year there were 512 such cities. By 2030, 1 billion more people are expected to live in cities. Additionally, freight volumes are expected to grow by 40% during this time period.

The use of artificial intelligence in smart cities can be life-changing if implemented in the right spaces. There are multiple zones in cities or in urban development where AI can be used to improve the performance and efficiency of the system. AI has the ability to understand how cities are being used and how they are functioning. It assists city planners in comprehending how the city is responding to various changes and initiatives.

AI in smart cities is going to play a big role in making urbanization smarter, intending to automate and advance municipal activities and operations at large. The Smart Cities of today are powered by advanced technologies that are constantly reshaping urban areas. AI and IoT are becoming increasingly integral to how the world operates.

Actually, while developing cities to make smart, several challenges like administration, sanitation, traffic congestion, security surveillance, parking management, and many more that AI can help to provide a sustainable solution to habitants.

AI with the help of Deep Learning and Computer Vision is changing the way vehicle analytics is done. With these advancements, vehicle analytics is helping in implementing intriguing solutions like Smart Gate Security, Speed Violation Detection, Toll Booth Automation, Smart Parking, Adaptive Traffic Control System, Traffic Congestion Detection, Vehicle Breakdown Detection, etc.

AI/CV based Vehicle Analytics:

Frame Grab/Capture:

This block takes care of getting the frame/image out of the input stream or image or any given input source. If the input is a video stream, this module takes care of decoding the stream and captures/extracts the image/frame out of it. If the input is directly an image but coming through a socket then this module handles extracting the frame through it.

Detector:

For vehicle detection and license plate detection state of the art object detectors such as Yolov3/Yolov4 or Faster RCNN or RetinaNet can be used. Though RetinaNet and faster RCNN can give better accuracy than Yolov3/Yolov4, they’re 4–6 times slower and Yolo models can yield real-time performance which is necessary for some of the vehicle analytics-based applications like Tollbooth automation, Gate security, and smart parking, etc.

We can train the object detector model with classes like a car, two-wheeler, bus, etc. and importantly LP(license plate) as one of the classes. For training datasets such as UA-DETRAC and KITTI datasets can be leveraged.

Vehicle detector output can be used to perform vehicle counting, vehicle classification, vehicle type detection, and vehicle color detection.

Tracking:

Object tracking helps to send the best possible LP (License Plate) Bounding box to the OCR module. Lucas Kanade Feature Tracker and Kalman filter are good options for non deep-learning based tracking. Deep learning-based tracking trackers such as FairMOT can be used.

OCR (Optical Character Recognition):

Implicit Language Model(LM) in multi-layer recurrent neural networks (RNN, LSTM GRU) can be used for OCR.

Conditional independence limitation with Hidden Markov Model (HMM) was addressed by Recurrent Neural Networks (RNNs), which theoretically have no limitations on the length of context they can utilize. LSTMs provide solutions to the vanishing and exploding gradient problems associated with the training of RNNs. With the introduction of Connectionist Temporal Classification (CTC) loss, LSTMs are well suited to
tackling the OCR problem, removing the necessity for frame-level label assignment. Because of the above-mentioned reasons LSTMs are better for OCR tasks than HMM and RNNs.

For Europe, Brazil, and the US LPs(License Plate) use the “OpenALPR” benchmark dataset and there are various datasets available on https://platerecognizer.com/number-plate-datasets.

Controller:

The controller takes care of efficient communication between all models and manages the entire pipeline. It’ll use a message queue library like ZeroMQ, a high-performance asynchronous messaging library.

Cloud Dashboard:

A centralized platform for data management includes the following data components.

• Vehicle ID
• LP details
• Vehicle class
• Speed violation
• Associated data and time
• Camera ID

Applications of Vehicle Analytics:

Speed Violation:

We can use the speed camera or Radar gun for speed detection of the vehicles and based on the threshold speed vehicle image can be given to the ‘Detector’ module.

Alternatively, we can also use the Tracking module for speed estimation of the vehicle. With the Tracking module, the distance traveled by the vehicle is calculated using the movement of the centroid over the frames and the speed of the vehicle is estimated. Ultimately Detector module will send an LP box of speed violated vehicle to the OCR module for LP Recognition.

Smart Parking:

Smart Parking requires parking slot occupancy and LPR based ticketless entrance/exit. The first step in slot occupancy detection is to find all the parking spaces. In a given space that the camera covers, it is crucial to know which of the spaces are designated parking spaces. Once we know which all are parking spaces, we can then proceed to step two which is detecting if a parking space is occupied or is available.

Unlike the traditional parking structures that require a lot of manpower to manage incoming and outgoing cars and tickets, License Plate Recognition technology identifies the license plate number and records the arrival or departure time of the vehicle. We can log the vehicle’s arrival and departure time but also with images for fee calculation and evidence use. A black/white list can also be pre-set for VIPs or unwelcome visitors.

Toll Booth Automation:

The license Plate Recognition (LPR) module will analyze the captured video to recognize the number. The captured frame together with the recognized number and entry record (entry date & time) will be stored for reference. Once this is completed, the entry barrier will open to allow the vehicle to go out from the toll plaza. The LPR module will match the recognized vehicle number with its own database for this particular vehicle. Once the information is collected, the system will calculate the toll fees and this fee can be deducted from the owner’s account. When the transaction is complete, the exit barrier will open and the vehicle will leave.

Smart Gate Security/Virtual Gate Guard:

Virtual Gate Guard with the help of License Plate Recognition expedites entry transactions and easily verifies repeat visitors at gated community entrances.

This gate security system captures an image of the license plate of each guest using the visitor lane to enter. Using LPR, the image is cross-referenced with the database of approved vehicles allowed into the community. If the vehicle has been to the community before and the license plate is recognized as verified and permanent, the gate will automatically open.

Vehicle Breakdown Detection:

With the help of vehicle tracking, we can find out whether a vehicle has been idle at a given place on highways for long periods of time. LPR can be used to get vehicles and it’s owner’s information for further action.

Traffic Rule Violation Detection:

Predetermined virtual zones can be marked across the traffic signal junction. Vehicle detection output can be used to find out whether a vehicle is in the wrong zone when there’s a red light, if yes then LPR can be used to raise violation tickets.

Based on the no parking areas predetermined virtual zones can be drawn. Vehicle detection output can be used to find out whether any vehicle is parked in no parking area. If any vehicle is found, LPR will be used to get vehicle details and raise violation tickets.

Adaptive Traffic Control System:

Virtual zones will be marked across the traffic signal, within these zones vehicle detector output will be used to get vehicle density at a given day and time for a given location.

The system is capable of adjusting the signal timing parameters in real-time according to the seasonal changes and short-term fluctuation of traffic demand, resulting in improvement of the efficiency of traffic operation on urban road networks.

Conclusion:

City planners and engineers are now working in increasingly complex environments and need to solve increasingly complex problems. AI and IoT are helping them tackle these problems. Traffic and transportation management poses a modern challenge that would be tricky to tackle without the help of software and algorithms.

When data is used for decision making and for gaining a better understanding of city travel dynamics, the quality of the management applications also increases. This ensures that traffic control strategies and future infrastructure development projects will accurately match the citizens’ needs.

As AI and machine learning are transforming the way cities operate, deliver, and maintain public amenities, the technologies come with some drawbacks. Thus, there is a need to consider retrofitted solutions that can hold smart city initiatives continuing.

The important thing is that we don’t need to see any new technology developed to see massive gains from cities becoming smarter. We have existing technology proven to be capable of improving parking utilization, safety, and significantly improving traffic; it just hasn’t been widely deployed yet.

And to develop the AI-enabled systems for smart city development, machine learning, and AI companies need a huge amount of training data for smart cities to train the models.

Beyond the industries directly providing these services to local governments, the spending on smart cities could impact a range of businesses. Reduced traffic would mean cheaper shipping and technicians being able to spend more time at job sites and less time moving between them. Fewer accidents could result in lower insurance costs for everyone.