External evaluation results of Smart Paper Technology in Uganda
In 2016, Shifo Foundation (Shifo), the Ministry of Health of Uganda (MoH), District Governments, IKEA Foundation, IKARE, and Plan International united to implement the Smart Paper Technology Solution (aka MyChild Solution) in Uganda. As of today, more than 145,000 children have been registered, more than 42,000 children have been fully vaccinated and more than 73,000 children have been followed-up with SMS messages (which notifies parents about upcoming vaccination visits). To see the current progress of work in Uganda, please visit Shifo’s website.
Since the inception of the programme in Uganda, partners have identified key success criteria to evaluate the effects of the programme and make informed decisions about scaling up the intervention in Uganda. This article summarises the results of two external evaluations conducted by Umeå University and Tallinn University in 2018. The external evaluation from Umeå University assessed three key aspects of the Smart Paper Technology (SPT) namely 1) the data quality 2) the time efficiency for frontline health workers and 3) the operational cost whereas Tallinn University assessed the transfer of work processes from Shifo to the existing health system structure and the sustainability of SPT.
Data Quality
Data quality of the SPT was assessed using The WHO Data Quality Review Toolkit. The assessment showed that high-quality data was generated from SPT in every aspect, including completeness (100%), timeliness (100%). Data was internally consistent within DPT vaccines, data consistency rate between source documents and electronic records was 100% and data recording errors were in the range of 0.9% to 3.8%.
Time Efficiency
Compared with the existing Health Management Information System (HMIS) tools with SPT, health workers spent less time on all stages of data management tasks with the biggest reduction in post-session data administration — 96%, and 49% and 85% less during the service delivery.
Operational Cost
The national annual operational costs were measured by comparing the current printing costs of HMIS Forms with the yearly operational costs of SPT in two different scenarios: excluding and including the monetary value of administrative time reduction as a result of SPT. The evaluation also took into account two versions of the SPT: MyChild Forms (facility-based data management innovation), and MyChild Cards (innovation based on child health cards).
Annual incremental costs for the existing HMIS tools, MyChild Card and MyChild Forms in Uganda were USD 872,199, USD 506,396 and USD 480,605 respectively while considering the monetary value of administrative time reduction as a result of SPT. When the monetary value of reducing the administration time of the SPT was excluded, MyChild Cards (408,902 USD) and MyChild Forms (383,111 USD) were more expensive than the current HMIS ( 349,247 USD). It should be noted, that the monetary value of improved data quality with SPT is not including in the analysis.
Transferred Processes
The assessment estimated that 84% of processes essential for the proper functioning of the SPT had been transferred to the existing health system structure. All remaining processes were planned to be transferred according to the time plan.
Recommendations
The external evaluations provided a number of recommendations and insights. The evaluations suggested that the Smart Paper Technology Solution be scaled-up on a regional level due to the administration time savings and the high data quality that the solution provides. Since a number of studies determined the lack of knowledge and community engagement as a hindrance to immunization coverage and narrated the importance to address this with targeted interventions and counselling, the saved time with SPT can be utilized with targeted interventions to engage the communities and focus on utilising reliable data to improve health outcomes in Uganda.
Conclusion
Based on the findings derived from these external evaluations, Shifo and its partners are working to further expand and evaluate Smart Paper Technology in Uganda.
The full reports and additional information about Smart Paper Technology Solution can be accessed using these links below:
- Evaluating the data quality, efficiency and operational cost of MyChild Solution in Uganda. The assessment report: https://shifo.org/doc/rmnch/MyChildExternalEvaluationUganda2018.pdf
- MyChild Solution in Uganda: An External Evaluation — Transfer of Work Processes to Existing Health System: https://shifo.org/doc/rmnch/ShifoExternalEvaluationTransferProcessUganda.pdf
- About MyChild Solution based on Smart Paper Technology: https://shifo.org/en/solution/
- Project progress in Uganda: https://shifo.org/en/work/uganda/
