As You Grow: Activate Data Use at all Levels

Photo courtesy of MiracleFeet

“If we can make data an essential tool for frontline workers, rather than a post hoc distraction, we will get more and better data. With that data, we will be able to ask better questions, increasing our efficiency in delivering our products and services.” — Jim Fruchterman in Using Data for Action and for Impact

For organizations that pursue a growth strategy to scale, the number of active users of data will naturally increase as the organization grows. In addition, while more centralized decision-making is often necessary during earlier stages when a team is trying to validate its model, scaling that model often requires more decentralized decision-making. Therefore, as an organization’s work grows, it will need to empower more people at all levels to collect, analyze, and use data to drive to action. But how can organizations ensure that the right data is getting into the right hands at the right time, and that the people using the data are prepared to extract insights and drive action?

This was a challenge that Living Goods faced as it scaled its work. As of 2019, Living Goods had over 10,000 active CHWs, serving more than 7.8 million people in Kenya and Uganda. Managing this growing footprint required layers of staffing, each with its own data needs. For example, each CHW manages 150 households (with an average of five people per household, so 750 individuals served), supervisors manage 35–50 CHWs, branches manage 200–450 CHWs, and so on through the country level and up to global headquarters. Living Goods knew that its legacy paper-and-pen-based system would not be able to keep pace with this level of growth and so, in partnership with Medic Mobile, it created the Smart Health app and data collection process to collect data on patient health and quality of care, manage CHW performance, and provide dashboards. By carefully designing a system and process that identifies data needs for various stakeholders, leverages technology to speed collection and analysis, and utilizes dashboards to communicate relevant data in real time to all levels of the organization, the Smart Health app allows Living Goods to “exponentially increase the scope of data available to improve performance” and use data to drive action.

The lessons learned from Living Goods and other interviewees showcase critical steps to follow to ensure data is used effectively to empower all levels of the organization as it grows:

DESIGN STEPS

Step 1: Identify data needs at each level: Who needs what, and when?
The first step is identifying the specific data that each level of the organization needs to execute its work most effectively. Start by answering the following:

• What programmatic questions does each level need to answer? For example,
  - Front-line workers often need data to drive actions and identify operational improvements;
  - Supervisors require data to drive performance management;
  - Regional leads need data to understand performance across units/teams;
  - Headquarters leadership use data to see overarching trends and drive strategy.
• How often does it need to be able to answer those questions?
• What is the simplest and timeliest data to help answer those questions?

Example: Once MiracleFeet developed and tested a robust model to identify and treat clubfoot through existing facilities in low-resource settings, it recognized the opportunity to decentralize data to drive scale. While MiracleFeet’s central team was already a heavy user of clinic data, the organization made it a priority to push data towards the frontline clinic teams so that it could act quickly to address performance and quality issues. As an initial step, MiracleFeet worked with stakeholders at each level of the organization to understand their key questions and how often they needed the corresponding data. Sample data questions by level are included in the table below:

Sample data questions at each level of data use, for MiracleFeet

Step 2: Consider leveraging data technology solutions to drive down cost and increase quality.
As founder and former CEO of Benetech, Jim Fruchterman, advised wisely in 2013, “Social entrepreneurs also need to embrace technology and data as indispensable parts of our enterprises.” Technology can make empowering all levels of the organization with data realistic even in the most remote areas. Many of our interviewees reported using mobile phones and tablets (which feed into larger software programs) to ease and improve data collection at the front line; in the energy and agriculture sectors, social enterprises are also using sensors to automatically collect data that would have previously been measured by hand and self-reported. See the Data Infrastructure article for additional tips on data systems and technology.

Example: According to Living Goods CTO, Asif Akram, “It was never a question for Living Goods whether we would use digital tools, it was just a question of when.” Living Goods determined when to invest in digital tools by
carefully considering the conditions which would drive sustainability and scalability of the tools. For Living Goods that meant investing in the Smart Health app only once mobile phones were prevalent and network coverage reliable in the regions where Living Goods worked. Using technology has allowed Living Goods to eliminate costly and often inaccurate paper-based systems and has increased the amount of data available in real time, shifting “from bicycle- powered delivery of health data to light-speed dissemination.” But Akram also cautioned, “Don’t create a problem just because you can solve it. Technology people can think they know more than the user. Sometimes we think we have a solution, but the user is always king. If he/she is not satisfied with the product, nobody will use it.”

COLLECT & SHARE STEPS

Step 3: Incentivize staff and partners to collect and use data.
Many of our interviewees reported that their front-line staff and partners were initially hesitant to include or add data collection to their existing responsibilities. To get these staff and partners on board, our interviewees advised that data collected must not become burdensome and must provide value to the data collector. To ensure this, ask yourself:

• Does it save the data collector’s time? MiracleFeet ensures that the data its clinics collect aligns with those clinics’ reporting requirements, therefore dramatically decreasing the clinics’ time spent on reporting.
• Does it immediately give them information that they need? Pratham ensures that the data its already-busy teachers collect for Teaching at the Right Level is seen as creating immediate value. The regular student assessments provide teachers with information they can use immediately to adjust that student’s learning plan; the assessment data is also aggregated at the classroom, school, regional, and national levels for other stakeholders to act upon. “Data needs to be empowering for people to collect and use it,” emphasizes Pratham’s Head of Program Management, Devyani Pershad.
• Does it make their work more efficient? The BOMA Project’s digital platform, Performance Insights, uses the data its front-line workers collect to provide them with immediate feedback so they can take quicker and more informed actions in their work — as opposed to having to go back to supervisors for guidance. For example, BOMA’s front-line business mentors can use the data collected about the performance of each business in the BOMA network to inform the types of ventures they suggest that new BOMA program applicants pursue.

Ensuring data quality as collection expands.
As the responsibility for data collection spreads across many individuals and groups, organizations must double down on data quality measures. For Pratham, whose data is collected by many thousands of classroom teachers, it institutes audit processes and double verification. Teacher mentors regularly conduct sample verifications, retesting small groups of students at a subset of schools and comparing results to the teacher’s results for those same students. These verification checks are used as teaching moments for the teacher to improve quality of future assessments so that children are receiving learning at the correct level.

Step 4: Utilize dashboards to effectively communicate data.
Even with the plethora of available data, it would be overkill — and ineffective — to provide data on all metrics to all team members. Interviewees shared the following advice for creating effective dashboards:

• Tailor for use, but keep it simple. Dashboards tailored for the needs of each user help the user to home in on the metrics that matter most to their workstream and provide a way to quickly identify areas for further interrogation. Akram, Living Goods’ CTO, cautioned, “Organizations often think about fancy dashboards, but don’t appreciate or realize that people don’t always have the ability to interpret or absorb.” He went on to explain that when Living Goods designs dashboards, it keeps them relatively simple and thinks about the perspective of the team using it. At each level, data is displayed, interpreted, and utilized differently and dashboards shift from simple at the front lines (Akram again: “People will say that’s not a dashboard, it’s more just showing information.”) to multiple dashboards of increased complexity at the headquarters level. [For more tips on creating management and program level dashboards, see Wolk et al, Building a Performance Measurement System: Using Data to Accelerate Social Impact.]
• Keep clients central. Marzanne Collins, Harambee’s Chief Information Officer, stated, “What I find valuable in our dashboard is not just what we’re counting, but what it says about what counts.” Importantly, Harambee asks itself if the metrics included in its dashboards keep young people — its key constituent — at the center of decision-making for the organization. Harambee also ensures that it accompanies the dashboard indicators with stories — making the discussion less about the target itself (e.g., number of phone conversations with young people) and more about the service (e.g., including stories to better understand if these are quality engagements, and what the young people are getting out of these calls).
• Do not forget users without internet or mobile access. Keep in mind that not all intended users of the data have access to the dashboards and visualizations. While Last Mile Health has deployed digital tools for reporting and data collection in partnership with the Liberia Ministry of Health, it realized that technology and bandwidth constraints would prevent the data from reaching some key stakeholders. To address the gap, Last Mile Health uses paper reports to connect health workers and supervisors in these areas.

Beware the tendency to interpret data visualizations as fact.
Another tool to communicate insights curated from dashboards is data visualization. Common tools such as Excel, Tableau, Power BI, or others create visual elements (charts, diagrams, etc.) that help various stakeholders understand patterns in data. According to Jake Porway, Founder and Executive Director of DataKind, “The best data visualization helps people investigate a topic further, instead of drawing a conclusion for them or persuading them to believe something new.” Although these tools can be powerful, Porway also shared the following warning, “The public still treats data and data visualization as ‘fact’ and ‘science.’ I believe the public has gained enough visual literacy to question photojournalists or documentary filmmakers’ motives, aware that theirs is an auteur behind the final piece that intends for us to walk away with their chosen understanding. We have yet to bring that same skepticism to data visualization, though we need to. The result of this illiteracy is that we are less critical of graphs and charts than written arguments because the use of data gives the sense that ’fact’ or ‘science’ is at work, even if what we’re doing is little more than visually bloviating.”

TAKE ACTION STEPS

Step 5: Ensure data users are equipped to draw actionable insights.
“You can’t just make a dashboard and expect people to understand how to use the data,” stresses Pratham’s Pershad. Ensuring that data users are prepared to draw actionable insights means that appropriate training processes must be in place to teach users how to understand, interpret, and act on the data provided. Interviewees also advised embedding data discussions and dashboard reviews in regular meetings and allowing for time in those meetings to walk through interpretations and provide additional context behind the data.

Example: A major role of Pratham’s M&E team is to bolster its users’ capacity to draw actionable insights from the data — above and beyond the basics of ensuring that users understand foundational concepts such as median, outlier, etc. During training and review meetings, Pratham shares case studies (based on actual program data) with participants and has participants practice drawing out insights to develop strategies for giving inputs to teachers and supervisors and to create priority plans for mentors. Pratham also works to incorporate data into regular meetings. For example, regional teams conduct monthly review meetings to discuss data trends, insights, and anecdotal experiences. The teams organize the meetings into three parts: sharing field experience and data, synthesizing the data into common themes, and developing an action plan to address any concerns.

Step 6: Build in systems to automate and drive action where possible.
How do you ensure that your data is not a passive asset for an already busy team member, but rather spurs individuals to further inquiry and action? Interviewees talked about building alerts and automated processes into the system to help all levels improve performance.

Examples: BOMA sets thresholds for where it expects to be on each of its key metrics and has programmed its data system to generate an alert if a metric deviates from the threshold. For each alert, BOMA has standard operating procedures for who should first respond and how. MiracleFeet’s CAST app automatically generates a list of appointments at the beginning of the clinic day and a list of no-shows at the end of the clinic day, so the staff can prepare the right medical supplies and can follow up with those who missed appointments. The app also alerts the user if a clinician has missed a treatment step or the treatment is not progressing well, prompting the clinician to reassess or seek help. Living Goods uses patient data to create “Daily Data-Driven Task Lists” that prioritize follow-up actions for CHWs based on the severity of patient diagnoses and daily action lists for supervisors noting issues that need follow-up (e.g., CHWs missing targets or field offices that are low on stock).

Read next: As You Partner: Adapt Your Approach to Data, As You Drive to Systems Change: Shift your Conceptions of Data, or return to see all articles in Data for Scale.

This article was written by Erin Worsham, Kimberly Langsam, and Ellen Martin, and released in June 2020.