Machine Learning for Road Condition Analysis Part 4: “Turning Models into Interfaces: the Z-Roads system”
The previous blogs (Part 1, Part 2 and Part 3) in this series illustrated just what can be achieved when different stakeholders get together to use AI in international development — even when huge data issues are faced.
Despite a roller-coaster journey of such challenges, Z-Roads managed to evidence the potential of machine learning models for automated road analysis, generating models that could predict road-conditions with a 73.3% accuracy level — a figure that can but improve in the future. While this outcome delivered the required research output’s demanded by the project, by itself this doesn’t deliver anything of immediate practical value to our stakeholders on the work — the Zanzibar Department of Roads (ZDoR). This blog focuses on how Z-Roads wrapped up in order to deliver some real interfaces to our partners on the ground.
Z-Roads managed to evidence the potential of machine learning models for automated road analysis, generating models that could predict road-conditions with a 73.3% accuracy level — a figure that can but improve in the future.
For a system to be useful it must be able to communicate its outputs in an efficient way, and produce an interface that integrates into the workflow of end-users — in this case engineers at the Zanzibar Department of Roads. Our starting point on this project was to create a shared prototype “interface” with ZDoR, upon which we could base that conversation. To this end, the team developed a first, basic interface which could be explored, taking the output of the machine learning models we’d created (essentially road segment patch ids paired with the AI’s predicted road quality scores) and transforming those data into a visual representation.
The prototype we developed is shown below (Figure 1) and consisted of two key components: (1) an overview map to enable the engineers to see which road sections required attention, (2) a function to further investigate each road segment to manually check the road quality and view the prediction.
The interface not only had the ability to generate a holistic view of the road network but also highlighted the three overarching categories we’d developed during the Z-Roads journey, specifically to assess the unpaved, and potentially dangerous road segments which predominate in Zanzibar. As discussed in the previous blog our AI models were designed to predict the quality of unpaved roads, by classifying drone imagery segments into one of three categories (which formed part of the 7 total categories in the schema developed to cover all road types). To foster discussion for the interface these were nominally colour coded into Green (good), Red(poor) and Orange (indicating where uncertainty between good/poor conditions would benefit from human review).
From Research to Utility in Practice
Our Version 1 interface was then iterated through direct collaboration with the Zanzibar Department of Roads, who provided further feedback on its suitability for integration in their workflow. Whatever we might desire of our AI implementations, coordinating with actual work practices of those that would use the tool on the ground had to be key. As ever, this conversation did not end as expected!
While the interface met the original project specifications it became clear that this ‘final’ system was unlikely to be directly used within their maintenance planning workflow. So changes were called for, all of which boiled down to two key issues:
- The continued cost/challenges of high quality EO data acquisition was prohibitively high for the ZDoR team.
- The fact that road condition was actually only one of many indicators collected via their traditional surveying meant that this interface could only ever be a “single link” in their chain of analysis — and could not be the basis for overhauling their whole process.
The first point indicates an under-appreciated challenge in the practical deployment of such a system — the high cost of collecting high quality Drone Imagery/EO data on a continual basis to feed into the system. While in some countries the central government may choose to fly drones to underpin a number of other uses (e.g. flood modeling), flying drones purely for road-condition analysis was not considered feasible/cost-effective for the Zanzibar Department of Roads, both in terms of cost but also expertise. As was demonstrated (in visceral detail!) in our previous blog, data collection is a crucial and under-considered step in AI systems.
Moreover, we showed just how much attempting collection without the proper expertise brings with it significant risk: Specifically, even after extremely costly exercises, it is entirely possible that the imagery data obtained will be of insufficient quality. Given these challenges, and with ZDoR having taken this journey with us, we fully understood their reluctance to radically change their work procedures, based on an AI interface. Instead they sought to gradually accommodate advances over time — and specifically with less radical changes to their established processes.
The second point, the fact that road condition is only one of many indicators collected via their traditional manual survey, also reduced the utility of an all encompassing imagery-driven system. Examples of other indicators ZDoR would normally look for include information regarding the state of the road shoulder, line markings and signage. Notably the latter is an example where AI based on drone imagery alone is unlikely to help due to its overhead perspective. If the system was unable to provide all the information Road Engineers needed, then they would have to go and re-survey roads manually anyway (to integrate other gazetteer-type data).
Would the amount of time saved in using drone-imagery as the primary source of their decisions be worth it, given the significant changes to their current work flow that would follow? While the answer over time may indeed be yes, given the lack of ability to reliably update the system with new drone imagery… the answer for the moment was a no. Again, this was totally understandable.
A New Interface to Deliver Utility in Practice
So while contractually the project had met its goals, we realised if we were to deliver real capacity improvements to our central project partners more would have to be done! And, as a consequence, we developed a new visualisation system in collaboration with our ZDoR partners — one that could not only be used on a daily basis, but that could provide a route to gradual, future changes in process.
Whilst the collection of this video footage of roads was not a strict project requirement, we had collected it anyhow — to provide greater context, and to enable even more remote review options for road conditions. And while that footage had provided a rich source of information for us as researchers, it also offered an equally rich source of insight for ZDoR’s hard-working engineers. From their point of view, video-footage of roads enabled the post-survey identification of other road quality indicators (such as road shoulder conditions, road markings and signage).
Interestingly, we soon realised the modified “surveying protocol” we’d developed as part of Z-Roads, which included the GoPro footage collection, could be reimplemented at near-zero cost using ZDoR’s current tech. This provided a clear route for accessing updated information during future surveys. All that was required was software to process that information and visualize it, adapting our initial interface prototypes.
The result was a new system that (1) provided an overview based on road condition scores providing information to shortlist areas for further review and then maintenance and (2) enable the secondary analysis of further road quality indicators as part of the further review, obtained by manual review of the video footage. Importantly, shifting this secondary analysis out of the in-field survey phase to video review enables both a potential reduction of analysis and a faster in-field survey — a clear cost/time saving.
Whilst we no longer were just looking at a “fully-automated” road condition system based on drone imagery, we had to accede that ZDoR were still going to go out in their cars. The new setup would, however, offer advantages both when they did have drone imagery, and when they didn’t due to a significantly improved ZDoR workflow. And perhaps this would be more suited to their short-term operational needs.
The final interface developed is shown in Figure 2. Separate interfaces and software were also developed to process and ingest bump integrator measurements and GoPro footage from future surveys. This software allowed additional datasets to be automatically added into the interface for visualization (two such datasets are shown in Figure 2 in the top left corner). As shown on the screenshot, the system provides engineers with the ability to visualize and explore the whole network in a dynamic fashion. It also enables Engineers to explore or check other road condition indicators through GoPro footage analysis.
Whilst not perfect, this was something that would be (1) used!, (2) could be updated with data from the augmented car-based surveys and (3) and has the ability to incorporate new predictions from drone imagery if it becomes available. A win-win.
Software alone is not always the answer
Having gone through the process of fully understanding where we could add value to the Zanzibar Department of Road’s workflow and developing an appropriate interface (backed by a little Python for processing any new survey data), we still had to ensure that after delivery, it could actually be used at the ZDoR physically.
While it is tempting to just “put it online” and provide a URL, this wouldn’t work in practice. A key issue in international development projects is, of course, connectivity. An online system’s use, in practice, would be low due to the lack of robust Internet access at ZDoR facilities. With no internet link, nor existing computational infrastructure, a physical “solution-in-a-box” was therefore developed by the Z-Roads research team. The solution was engineered using a Raspberry Pi, external storage, and a wifi hotspot capacity to allow engineers to connect via any laptop. This system (pictured in Figure 3) was delivered to the DoR on the 31st of January 2020 during a final field visit and training session.
The hardware was delivered fully configured — simply powering the device enables the ZDoR to connect directly via WiFi. In addition, an updated, crucially GPS enabled, GoPro was delivered for use with the system along with a large rugged external hard-drive for production use over future surveys. The total solution was able to be put together from consumer grade parts at minimal cost (to allow rapid replacement when/if necessary).
The end of the Journey… and the beginning of the next?
After many twists and turns the main research question of the Z-Roads project had been answered — AI, when given good drone imagery data, can make significant headway in automating road condition analysis. However, by adapting to on-the-ground needs and capacity, the programme also managed to deliver a system that would actually be used now — even if it might be the fully ‘automated’ all-singing and dancing software that might one day be in action! However it provided new capabilities for the Zanzibar Department of Roads. Success!
After many twists and turns the main research question of the Z-Roads project had been answered — AI, when given good drone imagery data, can make significant headway in automating road condition analysis.
…Well, almost. While we had shown the ability of machine learning to perform automated road condition detection from high quality drone imagery, we had also shown that, in Zanzibar at least, data collection was the bottleneck — and that currently a fully automated approach was unlikely until that’s solved. So, while potential remains high for AI application… you are only as good as the data you put in — and this means, for now, the automation we were able to achieve fell short of our highest ambitions.
But, this of course generated new ideas! The challenges of drone image acquisition is not limited to Zanzibar. So the question is raised — is there a different approach that, in the short term, could be used to automate road condition monitoring but for which the data collection is currently viable? Based on our discussions with the Zanzibar Department of Roads and the ideas of their talented Engineers we believe that the answer is “Yes”.
And, AI-supported motorbikes may indeed hold the key… but more on that in a future blog!
