EXTERNAL EVALUATION RESULTS OF SMART PAPER TECHNOLOGY IN AFGHANISTAN
Assessing data quality, operational costs, efficiency gains and transfer of work processes to the existing health system
In 2015, the Shifo Foundation, the Swedish Committee for Afghanistan (SCA), IKEA Foundation, and the Ministry of Public Health (MoPH) in Afghanistan started a joint collaboration to strengthen child health services in Afghanistan. Data and information are fundamental to inform decisions and assist key stakeholders to allocate appropriate resources to continuously improve the quality of health services. Therefore, one of the main objectives of the collaboration was to strengthen the quality of data and its utilisation in the Expanded Programme on Immunisation using an innovation based on Smart Paper Technology called MyChild Solution.
MyChild Solution is an innovation developed by Shifo Foundation based on Smart Paper Technology. The solution was implemented and evaluated to inform evidence-based decisions on the scale up of the programme. MyChild Solution was implemented in 141 health service delivery points including fixed, outreached, and mobile clinics in the Mehterlam District of Laghman Province in Afghanistan. Currently, using MyChild Solution, more than 45,000 children have been registered, more than 9,000 children are fully vaccinated and more than 11,000 children are being followed up with SMS messages which inform parents about vaccination schedules.
From the beginning of the programme, project stakeholders set several programme key success indicators which informed project development and external evaluations. These success indicators measured data quality such as completeness of data, timeliness, internal consistency, and external consistency and analysed if MyChild Solution could be integrated into the existing health system, thus sustained by the government.
In June 2018, two external evaluations were conducted to assess data quality, operational costs, and efficiency gains as well as the transfer of work processes to the existing health system. This article summarises the results of these external evaluation reports.
The data quality and review toolkit developed by World Health Organisation was used to evaluate the quality of data generated by MyChild Solution. The assessment showed high-quality data generated from MyChild Solution in every indicator, including completeness (100%), timeliness (91,66%), internal consistency (100%), and external consistency (99,4%). Moreover, the ratio of data recording error was low in the study and ranged from 0.05% to 1.7% for two selected data recording errors.
The second evaluation investigated time efficiency. This evaluation assessed the time health workers spent on administrative tasks during and after delivery of care with MyChild Solution and compared the results with existing Health Management Information System(HMIS) tools. Results showed that 64% to 96% of time spent on administration could be reduced with MyChild Solution when compared to the current HMIS.
Incremental cost analysis was done considering two scenarios. The first scenario took into account the monetary value of the reduced time for administration whereas the second scenario was conducted without the time reduction values. The evaluation also took into account two versions of the MyChild Solution. The first evaluation assessed MyChild Forms which is an innovation on facility-based data management tools. The second evaluation assessed MyChild Card which is an innovation modelled after the child health card.
When adding the value of the reduced administration time, the total national cost of MyChild Card was 611,974 USD and the total national cost of MyChild Forms was 316,436 USD. Comparatively, the existing HMIS total national cost was 873,253 USD. Over a five-year period, MyChild Forms would save around 2,938,543 USD and MyChild Card would save around 1,378,875 USD compared to the existing HMIS system. When administration time is removed from the analysis, MyChild Card (501,622 USD) and MyChild Forms (206,126 USD) amounted to be more costly than HMIS forms (195,581 USD). It is noteworthy that when administration time is excluded from the analysis, MyChild Forms were 5% more expensive than HMIS.
The second report displayed the results of the transfer of work processes to the existing health system. This is one of the key elements to evaluate the sustainability of the programme as well as how successful management was by the local stakeholders in Afghanistan.
The results indicate that 95% of the processes essential to the management of MyChild Solution had been transferred to the local stakeholders in Mehterlam District. In most cases, these processes had been transferred in a way that is both accurate and sustainable. The remaining 5% of processes are planned to be fully transferred to Mehterlam by the end of 2018 to increase process accuracy.
The external evaluation reports provide information on the effects of MyChild Solution from four different perspectives which give valuable insights to key stakeholders. These perspectives are data quality, costs, efficiency gains and transfer of work processes to the local level.
Based on the findings derived from these external evaluations Shifo, SCA, and MoPH will collaborate to further expand and investigate the intervention on a larger level to inform decision making for the national implementation of the programme. The programme positively addressed all the key success metrics set in 2015 and brings opportunities to empower health and social workers at all levels of healthcare delivery who continuously work to improve quality of child health services across the country based on the reliable and relevant information.
The full reports and additional information about MyChild Solution can be accessed using these links below:
- Questing The MyChild Solution in Afghanistan- An external evaluation of Data Quality, Operational Cost and Efficiency: https://shifo.org/doc/rmnch/MyChildExternalEvaluationAfghanistan2018.pdf
- MyChild Solution in Afghanistan: An External Evaluation — Transfer of Work Processes to Existing Health System: https://shifo.org/doc/rmnch/ShifoExternalEvaluationTransferProcessAfghanistan.pdf
- About MyChild Solution based on Smart Paper Technology: https://shifo.org/en/solution
- Project progress in Afghanistan: https://shifo.org/en/progress/afghanistan