Opportunities for improving systems to provide social benefits to those who incurred occupational lung disease
Background:
In an attempt to redress long-standing injustice to mineworkers from the South African gold mines, a collaboration between the gold mining industry and state agencies was commenced — and has been previously described (links). One component of this collaboration was the development of an extensive database of claims and employment information. This, together with the global interest in computer-assisted detection (CAD) software systems for TB, provided an opportunity to improve resource efficiency and accelerate response times. It also provided an opportunity to advance world understanding of the role technological advancements could play in improving the efficiency of health systems and promote health equity.
An important bottleneck in the South African system is the requirement under the Act that every claim be assessed by a radiologist and a 4-person medical panel (called the full panel hereafter). Based on an exploratory analysis, we ascertained that only 25% of claims in recent years have been found to be compensable under the Act. Further exploratory work undertaken by our research group with the Medical Bureau for Occupational Disease (MBOD), the statutory agency responsible for certification, concluded that a substantial proportion of cases could be successfully handled through an accelerated process based on triage.
A triage process was therefore instituted by the MBOD in 2020. This consisted of routing a proportion of the claims to two newly instituted 2-person panels, called here the reduced panel and TB Panel. While the full panel consists of practitioners with qualifications in occupational health, the 2-person panels members do not necessarily have these qualifications and are trained on the job.
In this process, claims are pre-screened clerically and then assigned to one of three panels, based on whether there is suspected tuberculosis (TB), and whether the miner had worked for more than 10 years. After a claim is evaluated by a panel, the case is either certified (“finalised”) by that panel or escalated to the full panel as required. The rationale for the TB triage is that a large proportion of claims are for wage-loss only due to TB, but without a permanent impairment determination that could be made at the time of the claim. The role of the TB Panel is to separate out non-compensable disease (NCD), certify those with TB wage-loss only, and escalate those in which there might be TB impairment.
The rationale for the reduced panel is that claims with normal chest x-rays (CXRs) cannot be diagnosed with silicosis and that, based on a preliminary analysis by the MBOD, only a small proportion of silicosis cases was expected to occur in miners with fewer than 10 years of service. The smaller panels thus have the potential to relieve the full panel of a substantial adjudication burden by certifying as NCD claims without disease, leaving the full panel to concentrate on claims with permanent impairment for silicosis or TB and other diseases compensable under ODMWA.
The objectives of this project were twofold.
1. To measure the efficiency achieved by the current triage process in terms of person-time saved and reduction in certification delay.
2. To ascertain the potential for further gains in efficiency by identifying (i) claims currently assigned to the full panel which could be assigned initially to a smaller panel, or (ii) claims currently referred to the full panel which could be retained by the smaller panels.
Of particular interest was: i) the potential for pre-identification of normal CXRs via CAD, using different thresholds for sensitivity (ability to minimize “false” negatives) and specificity (ability to minimize “false positives”) ; and ii) the extent to which alternative service thresholds to the current one of 10 years could improve efficiency, based on modeling alternatives using a variety of demographic and exposure variables.
Figure 1 provides an overview of the basis for the triage and our approach to assessing its effectiveness.
Methods
Objective 1: Overall efficiency of current triage
For the overall efficiency assessment, we analysed the routing and outcomes of 9,944 claims submitted to the MBOD between February and October 2020. Person-time saved or expended was defined as follows. A claim routed to and certified in a 2-person panel counted as 2 “expert-reviews”; a claim routed to and certified in a full panel with a radiologist would count as 5 expert-reviews; and a claim that was routed initially to a 2-person panel but subsequently escalated would count as 7 expert-reviews. (Prior to any triage system, MBOD would assess every case using the full panel with a radiologist.) Therefore, we compare person-time with this baseline where each case would count as 5 expert-reviews and calculate the % reduction.
Objective 2: Opportunities for further efficiency gains.
CAD: For projecting the likely impact of CAD on efficiency we drew on our recent research comparing CAD outcomes to the MBOD certifications of 330 CXRs with a preselected distribution of outcomes, which showed that three of the CAD systems tested achieved a high area-under-the-curve (AUC) for distinguishing abnormality (defined as having silicosis, tuberculosis and/or silico-tuberculosis) from normal. Two of the systems achieved a similarly high AUC in distinguishing each condition individually from normal. The value of CAD in this context is thus to rule out disease. To estimate the false negative and false positive rates in applying CAD to CXRs evaluated by the MBOD panels, we used the calculated specificity of 98.2% and sensitivity of 98.2% of the best performing system. These values were based on the CAD score threshold that maximised Youden’s J statistic, an optimisation procedure.
As the CAD analysis of CXRs cases in in the publication mentioned above was based on a purposive sample, a further sample of 501 unselected chest x-rays submitted with consecutive claims from former gold miners from three rural sites (Stilfontein, Alice and Bizana) in South Africa were analysed as part of this study. These “field study” CXRs were read by three CAD systems and, separately, in a consensus reading by an occupational medicine specialist with experience in assessing occupational lung disease.
Identifying factors predictive of silicosis compensation: In order to identify the optimal triage threshold in years of service, or alternatively, an algorithm of multiple demographic and occupational factors, we undertook an analysis of the predictors of silicosis certification in three compensation databases. Logistic regression was performed to analyze the association with the outcome (certification of silicosis) of age, calendar years of service, dust level- risk rankings of job, and years of service.
Main findings
1. We found a substantial increase in efficiency, both in person-time and reduction of delay in certification of claims using the currently piloted triage system alone. This system required only 80.7% of the staffing-time needs of the traditional system. The gain in efficiency was 19.3%.
2. We then modelled the application of CAD to pre-identify CXRs with neither silicosis nor TB, which would enable the routing of such claims to a lower intensity of medical adjudication (i.e. the reduced panel), we estimated a further substantial gain in efficiency. Depending on where the thresholds are set for sensitivity and specificity, the gain in efficiency would be approximately 24–33%.
3. The regression analysis of previous databases (including the MBOD as well as the QT database, as well as the field study data) indicated that years of service on its own was the best predictor of certified silicosis, with no substantial further gains to be had from adding in age, latency or other variables.
4. Without the use of CAD, the change of triage threshold from 10 years to 15 years, for example, would see a slight efficiency gain (1.6% fewer expert-reviews than using the 10-year threshold); moving to 20 years would add an additional 1.3% efficiency gain from the 15-year threshold.
5. A surprising finding was that the largest potential gain would be to route all non-TB cases from living claimants to the reduced 2-person panel (5.7% fewer expert-reviews than using the 10-year threshold). The overall gain in efficiency compared to the previous 4-panel approach would be 25.0%.
Recommendations
(a) Efforts that would improve the efficiency of the system include better triage in the field, i.e. when benefit medical examinations are carried out, such that claims that have no history of TB, or likelihood of having had TB based on symptom review +/- microbiology +/- CAD, and no CXR evidence of silicosis or other occupational lung disease, would not be submitted to MBOD. Also, those claims that are assessed by the referring physician or field site for “TB wage loss only” be clearly marked as such, and routed to the TB panel. The higher the prevalence of disease in claims submitted, the greater the efficiency of the system.
(b) Further efficiency gains may be possible by identifying if improvements could be made in the clerical file review of claims to identify those that should go to the TB panel, to decrease the the large number of ‘TB wage-loss only’ sent directly to the 4-person panel.
c) The poor agreement in this study between CAD systems and between CAD and expert readers (Supplementary Material), particularly in comparison to the earlier results from the curated CXR set (Young et al. 2020) is an important finding. The fact that the gap between what was the number 1 vendor and the next closest is now very narrow — such that there has been a successful closing of the gap with the top two companies, both of which performing a bit worse in this field study than was the case in the “curated study”; the third company, however, did quite a bit more poorly. It would be useful to discuss the results with the CAD vendor in question to understand if the issues relate to the technology, thresholds chosen, or some other explanation. Because of the potential of CAD to support further efficiency gains, research is needed to further improve confidence in CAD, including the ability of CAD to distinguish disease from normal.
(d) A cost analysis can be done to calculate the financial savings in adopting the optimal system described above. Information on costs would be obtained from the MBOD for this analysis.
(e) The results should be presented to all stakeholders with a view to obtaining consensus on whether a change in claims processing should be implemented, whereby, for example, all claims are routed to either the TB panel or the reduced panel initially, with only those determined by those two 2-person panels to needing the full-panel assessment, to be referred upwards. To do this may require, for example, re-assignment of expert occupational medical
specialists currently only employed at the full-panel level to instead participate in the reduced panel. It may also worth using CAD set at sensitivity f 100% to aid the panels in ensuring that no claims that should be referred to the full-panel are missed.
(f) The above results, starting with the two supplemental more detailed analyses, should be submitted for peer-review publication.
Conclusion
While work is underway to improve CAD, we have calculated that sending all the claims directly to the reduced panel, and using CAD of either of the best two vendors set at sensitivity of 100% to minimize the likelihood of any claims being missed, with only those deemed by the reduced panel as having silicosis or TB impairment, or combined disease, to the full panel, would result in an approximate 25% gain in efficiency, with very little possibility of eligible claims being missed.
The current system assigns the more experienced medical adjudicators and radiologist to the full-panel. Implement this finding, to be practicable, requires consideration of assigning some of these practitioners to the reduced panel.
Our initial supposition that analysis of large datasets of occupational and demographic information on miners and claims would yield a prediction set for silicosis that could be used for triage was not supported. On the contrary, dropping years of service from the triage and using a simple assignment based on presence or absence of TB in the claim diagnosis, yielded the greatest efficiency.
