AI4TB: Ground-truthing machine-learning innovations
The fourth “Sprint” of the AI4TB pilot met the challenges thrust upon all by the COVID-19 pandemic. Nonetheless, several key advancements and learnings emerged.
The main message we took from our work is that there is a limit to what could be done with old technology; but, at least theoretically, CAD can help
We found poor association between CAD designation and human reader diagnoses in cases in which there only existed old analogue chest films that were then scanned and digitalized. Our experiment allowed us to conclude that the scanning process is not the issue. A further experiment could, theoretically, be designed to assess inter-reader variability and concurrence amongst: CAD, panel decision, and one or two independent readers. This could help ascertain whether the CAD can be actually more accurate than panel human readers on these old films — but we would need independent expert readers to serve as the gold standards, and considerable learning by the CAD from these old films, which, I turn, would require development of a training set of analogue films pre-reassessed by independent experts, using all available information. Given today’s challenges, this is not practical at the moment.
We also learned that much greater efficiency gain could be achieved if the requirement for a 4-person adjudicative panel could be set aside in favour, for example, of all claims going through an algorithm based on years of work, age and possibly job title (still being studied), along with CAD triaging and 2-person panel assessment — with only those in which the 2-person panel disagrees internally or is at odds with the CAD designation going to the 4-person panel. At the moment, only claims for “TB wage loss only” can be certified by a 2-person panel.
Also, we learned that there is still a considerable digital divide, but that regular team meetings using teleconferencing works well to keep everyone informed:
While a certain proportion of people around the world who were forced to work remotely already had home-based work set-ups to certain extents, this was not the case for some key service personnel in South Africa whose work is key to moving this technology-driven project forward. Given that high quality equipment is needed to accurately view and discuss chest x-rays, for example, combined with the need for tight security and confidentiality controls, the Medical Bureau of Occupational Disease (MBOD) panel is still struggling to have all their hardware, software, protocols and connectivity issues sorted out. This has delayed the completion of several of the experiments.
A silver lining to the cloud of the pandemic was turning to videoconferencing (Zoom, Skype, MSteam, etc.), with regular weekly meetings now occurring with the extended team — including not only MBOD personnel but also vendors, other researchers, other researchers and technical experts — all interacting to discuss the ongoing challenges and how to move forward.
The spirit of all the various organizations working together is higher than ever before
At the end of the last sprint, the project was working with 2 technology vendors; in this sprint, the 2 companies that had dropped off have re-engaged, such that 4 vendors are now involved in solving the challenges. All vendors and new team members are given the opportunity to present and discuss their ideas.
When the project started, the MBOD and Commissioner of Compensation of Occupational Diseases (CCOD) were working separately from the Tshiamisu Trust, despite overlapping mandates and shared challenges which were being addressed by this project. There are now much closer working relations. Additionally, the learnings from the Q(h)ubeka Trust will soon be able to be brought to bear.
Progress and Plans
Progress is being made despite the challenges and the lessons synthesized in a report to be shared widely, incorporating consideration of new challenges to the assessment process itself due to the COVID-19 risk precluding use of lung function tests.
We synthesized what we learned from the work to date regarding the case for and against the use of AI in this context, identifying concerns that could prompt not going forward with such innovation as well as approaches for addressing them. With lung function testing now prohibited due to COVID 19, we plan to add these new factors into this analysis and soon submit this for stakeholder input and additional peer-review publication to inform world knowledge.
The plan from here is to conclude the unfinished experiments as soon as the data are provided; and submit two new peer-review publications, after obtaining stakeholder input: efficiency gain from use of AI, incorporating a financial assessment of the savings; and the case for and against the use of AI in this context.
