Deep Learning with Location Analysis
Extracting road features from imagery
Road accidents are things commonly happened in Puncak, a highland area near Bogor, Indonesia. Deaths and injury victims sometimes can’t be avoided, but the number could be reduced. Road accidents in Puncak have several causes: sleepy drivers, vehicle, road, and environmental factors. In response, the government takes some actions by building road features: street signs, traffic and street lights in the recorded area where road accidents come up, named the black spot area.
By putting the aforementioned items above, it is believed will reduce the number of road accidents. Hence, maintaining the road features is crucial to prevent other road accidents or to install new road features in the new black spot area. In the past, the data collection was done by capturing geo-referenced photos and videos from Mobile Mapping Systems (MMS) and tagging the road features one by one. This procedure, of course, takes a lot of time to do it manually.
The capability of creating a model that able to detect the new object is one of the fundamentals of Deep Learning. This automation process would provide a low-cost solution to many use cases faced by governments or local entities as doing it manually will consume a lot of time and high cost. In this article, I will show a use case in Puncak on how to automate the tagging road features process with deep learning and to gain insight with the geospatial dashboard.
Outline:
1. Data acquisition
2. Data modeling
3. Model testing
4. Geospatial dashboard
5. Conclusion
Data acquisition
I take the geo-referenced video with GoPro 7 by placing it on the car dashboard while driving to Puncak. The cool thing about GoPro7 is the capability to extract the telemetry data (gyroscope and GPS) for further analysis while recording the video. If you want to know how to extract the telemetry data you could read my previous article: Extracting GoPro GPS and other Telemetry Data.
Data modeling
In my previous article: Road Crack Detection — General Workflow, I wrote the general procedure on how to build a deep learning model to detect road holes. For more detail, you could read the article. In this case, we change the detected object into road features for the train & test data and classify it into three items: street sign, traffic light, and street light.
Model testing
Test the model obtained from the previous step to the test data set. The result of the model is shown in the figure below.
After testing the model, save all the detected images with an accuracy of more than 0.5 plus the appropriated CSV file. This CSV file will be used to get geo-referenced images as when we split the video to images the geo-referenced information will vanish.
First column: Index
Second column: Image name
Third column: Time (second)
Fourth column: Detected object (1 = street light, 2 = street sign, 3 = traffic light)
Geospatial dashboard
After we get the geo-referenced images, we will plot the location along with its images to the geospatial dashboard. You can access the dashboard from this: Demo.
The probability of the blackspot area is demonstrated as a blueish heat map. If the probability is higher, the color will turn red.
The detected objects are visualized as three items, where the red pin describes a street light, blue pin defines street sign and yellow pin represents traffic light. The locations are captured by applying deep learning. The image with object detection could be accessed by clicking the pin.
For further analysis, the gyroscope and speed data are extracted from GoPro 7 telemetry data. We can see the car speed profile while recording the video. To analyze the road condition, we could use the gyroscope data where the red chart shows the speed on the y-axis and the black chart represents the speed on the x-axis.
Next, we will join all the data on the same layer. Surprisingly, the hypothesis of whether the road features will reduce road accidents seem true. In the winding area where the probability of road accidents is high, the car speed is relatively high and there is no slow-down street sign nor the traffic light. Thus, it should be considered to put more street signs and traffic lights there to give more awareness to the driver. Furthermore, we need to check the road condition because the gyroscope shows a spike in that area.
Conclusion
By applying deep learning with GIS technology, we could do a location analysis while saving time and money by automating the data collection of the road features and extracting the telemetry data.
This solution was published as a solution repository of the Esri Indonesia Geo-AI team.
