How can we measure better data?

Electronic Immunisation Registries promise to bring better data for better decision-making and for better health outcomes. Better data, in this case, means, that you can trust that for every child who comes to receive immunisation, their health records are recorded and available in the Electronic Immunisation Registries. And that every report is generated based on individual child’s health records. In other words, you know that your data is complete.

“When key actors know the data is complete, they will trust that data.“

Let us take this short journey together to understand how to systematically and routinely measure data completeness/quality of data in the Electronic Immunisation Registries.

Usual Approach

One way to measure data completeness is by copy/pasting the method used in monitoring and evaluation (M&E) of existing paper forms used in health centres today. We call it “Usual Approach”, as many organisations, still use this approach for monitoring and evaluation of data completeness/data quality.

It works like this:

1) After travelling to the health centre, and receiving permission, the vaccination history written in the Child Health Cards brought by parents is reviewed, and the ID number of the child and vaccines received is noted down in a separate form

2) The authorised person logs in and provides access to the Electronic Immunisation Registry, which is checked based on each child’s ID, to see if it includes the same vaccination history

3) The deviations between what is written in the Child Health Cards and what is included in the Electronic Immunisation Registry are analysed

4) The deviation percentage reveals whether the data is complete, meaning if the Electronic Immunisation Registry data has captured all the children that came and received different vaccines in that health centre

5) Based on identified deviations, actions can be taken to improve the data completeness

This method can be done per health centre per immunisation session, and based on that, conclusions can be drawn on the data completeness.

This Usual Approach to analyse data completeness is quite similar to how one performs Monitoring and Evaluation on the existing paper forms, used in the health centres.

But it costs a lot of money, time and energy to do this kind of M&E. And based on the high costs involved, it cannot be done for every health centre on a regular/monthly basis.

In practice you end up not doing any analysis of data completeness, and in reality, you won’t know how reliable the data is that you are getting.

What if data quality could be measured systematically and regularly, and done with limited resources?

Shifo Approach

At Shifo we found a better way of doing systematic evaluation of data completeness and accuracy, which we have begun using regularly.

Let’s say we would like to check the data completeness for June 2017 for Health Centres in District 1, where MyChild Solution is up and running. In this as yet unnamed approach, the method includes the following steps:

1) In the Electronic Dashboard, determine if Health Centres delivered the services according to the schedule and how many children received immunisation services per each session

2) The Stock Management Report for each health centre is checked to determine the amount of consumed and administered vaccines, wastage rates and stock at hand

3) Based on the above steps, if indications of problems in data completeness are found, actions can be taken to improve the data completeness

4) The above steps are repeated on a monthly basis to improve data completeness in all health centres

Electronic Dashboard

Electronic Dashboard, generated by MyChild Solution, tells us whether the health centre has delivered services according to the schedule, and how many children visited the immunisation session. If the health centre delivered services according to the schedule, those days are shown in green, and if they did not deliver services, this is shown in red. Sometimes clinics may deliver services on other days, not according to the schedule, in which case the Electronic Dashboard shows those sessions in yellow.

Ideally every health centre that uses MyChild Solution should have a green colour for each immunisation session and total number of children per session. The aim of using the Electronic Dashboard is to understand if the clinic is above or below 95% in delivered sessions. When used in conjunction with the Stock Management Report, this gives an indication on data completeness.

It is important that every immunisation session has a green colour, but it is not enough. We may see green colours everywhere, but we should also evaluate whether every child’s information is registered in the system during immunisation sessions. For that we use the Stock Management Report.

Stock Management Report

MyChild Solution automatically generates a Stock Management Report per health centre per month. It includes a section called “Vaccine and Syringes Monthly Utilisation Report”. This section includes informationon how many vaccines have been consumed/used (Used Doses), how many vaccines were given to children (No. of doses administered), the wastage rate (% wastage) and stock at hand (Remaining/doses) in the health centre.

To better understand about data completeness, the difference between vaccine doses consumed and administered and the wastage rate is considered. If every child’s vaccination information from a particular health centre was entered into the system, then there should be wastage rates that are normal to that health centre. If the wastage rates for vaccines are much higher than normal for that health centre, then it can indicate that not every child’s immunisation information was entered into the Electronic Immunisation Registry.

Checks are made for Pneumococcal (PCV) and Rotavirus (Rota) vaccines (depending on which one is available in the country), as they are one- or two-vial dose vaccines. Hence, ideally for PCV and Rota vaccines, the wastage rate should be zero percent. Allowed wastage rate for PCV and Rota vaccines is maximum 5%, so a good indication that the data is complete is when the PCV and Rota vaccines’ wastage rate is under 5%.

Where there are red colours in the Electronic Dashboard with excessive wastage rates in the Stock Management Report, this indicates that those health centres may have problems with data completeness and data quality. Equipped with this information, we work together with District Health Team and health centres to monitor and improve the situation, and this is evaluated on a monthly basis.

Certainly, this method requires upfront investment in developing the Electronic Dashboard and Stock Management Reports. But once it is there, it helps health workers, District Health Officers, the Ministry of Health and other partners to monitor and evaluate data completeness on a monthly basis and keep improving.

This approach enables systematic and routine monitoring and evaluation of data completeness and helps in improving MyChild Solution and other processes at Shifo.

At Shifo we believe it is the right thing to do, to prove that our solution actually brings better data for better decision-making, by systematically monitoring for data completeness, month after month, for every health centre, where we implement Electronic Immunisation Registries. We focus equally on developing solutions that produce reliable and actionable data to strengthen planning and decision-making within and across levels; and on process innovation to move away from outdated models of health service delivery that are no longer adequate to modern challenges.

When health centres, District Health Teams and the Ministry of Health see the evidence and data completeness indications through the Electronic Dashboard and Stock Management Reports, governments are much more willing to replace existing paper forms and move completely to Electronic Immunisation Registries, hence making systemic change possible.

If you believe that Shifo is using the right method for systematically monitoring and evaluating data completeness and data quality, let us know your suggestions on how we should name this method by writing to changetheworld@shifo.org.

And if you have thoughts on alternative/complementary methods to systematically monitor and evaluate data completeness and data quality, do let us know.