Charting change: what we’ve learned from tracking our data tools and approaches
Pulse Lab Jakarta and Saraswati
How best can we inform ourselves and a broader audience about the impact and lessons of our work?
We recently asked Saraswati to tackle this by writing Stories of Change about six different data tools and approaches developed or supported by PLJ since 2014: Haze Gazer, the Vulnerability Analysis Monitoring Platform for Impact of Regional Events (VAMPIRE), and four separate collaborations that used human-centred design in support of research and stakeholder engagement.
Developed for our own internal reflection, the Stories document how these initiatives are being used by our government or development partners, what kind of behaviour or policy change they enabled, as well as the factors involved in helping or hindering progress. The Stories help us identify what worked, what didn’t, and — most importantly — what we should be doing differently. They also help to put some meat on the bones of our framework for assessing impact.
Applying our own terminology for impact, we’ve charted how, through our work and collaborations, PLJ has made key contributions to how partners work, new applications of data science, and support for the broader data innovation ecosystem over the past few years. Here is a quick summary:
- Operational Impact: Haze Gazer and VAMPIRE supported positive change in the way the collaborating partners and the Government of Indonesia work. The Executive Office of the President adopted these platforms as key building blocks in developing the architecture for its Early Warning System, which is now a cyber centerpiece of the President’s Situation Room. We at the Lab have learned from and scaled these prototypes, including in developing multi-disaster platforms and tools for better monitoring of air quality; and through VAMPIRE, PLJ has supported the World Food Programme’s (WFP) ability to integrate and promote data innovation. WFP continues to partner with us in improving the features and accessibility of VAMPIRE while expanding this concept across the Asia Pacific region. In 2017, VAMPIRE won a WFP Innovation Award. For many partners in human-centered design collaboration — including GIZ and the former Australia Indonesia Partnership for Economic Governance (AIPEG) — this was the first time they had applied such an approach. Each acknowledges how this experience has informed ongoing work, too.
- Methodological Impact: Haze Gazer and VAMPIRE most vividly represent tailored and relevant applications of data science — and provide important lessons from the hits and misses in experimentation along the way. Most significantly, they have contributed to richer and more real-time data being used by the Government of Indonesia for emergency monitoring, planning, response and evaluation related to haze and food security.
- Ecosystem Impact: PLJ work reaches a large audience via the Global Pulse network. Those accessing our data and platforms range from international researchers to the UK Climate Change Unit. And PLJ has contributed to how key stakeholders participate around data innovation. Based on collaboration in co-design work, Brawijaya University in Malang, East Java, started a Masters program in Social Entrepreneurship. Collaboration with GIZ and the KOMPAK programme has supported national and local government uptake of human-centered design approaches.
No self-respecting innovation lab skips over the failures — and the process of tracking these changes highlight that results are not always positive. For example, a co-design workshop conducted in South Sulawesi in 2016 was a litany of missteps: the policy environment in a particular district made the acceptance of proposed changes difficult; the workshop took place right after the local government’s budget cycle and so no local funds were available (as hoped) to sustain designs; and specific prototypes championed from the workshop stuttered due to poor conceptualisation.
But these were in the early days of our experimentation with human-centred design and so were chalked up as valuable lessons! Other key lessons and recommendations emerging from these Stories include the following:
- Work politically. PLJ does work politically, in that we have demonstrated that processes are more important than projects and that the inner politics of government counterparts cannot be ignored. For example, Haze Gazer and VAMPIRE were designed as legacy systems to conform to existing government systems. More significant lessons have been learned in engaging around the politics of local government. One of our implementing partners shared the following valuable advice: “Two important pre-conditions for local government uptake include strong commitment from local leaders who believe that existing public services need to be improved and solid and hard-working second liners (head of subnational agencies and her/his subordinates) who have a strong motivation to do something new in better serving the public.”
- Contextualize. We may be an innovation lab, but experience validates that we need not define success in terms of “innovation”. We are more interested in applying approaches that can be contextualized for particular circumstances and conditions. Haze Gazer and VAMPIRE have made significant waves, but they do not represent radically unique approaches to data science. Yet, they fill a direct need from our partners for fit-for-purpose tools.
- Thicken the data. In a world of Big Data, qualitative research can often take a back seat. PLJ has explored quantitative data through, for example, extensive social media analytics related to haze. But in some of our original human-centred design work, PLJ demonstrated capacity in qualitative research, too. In our collaboration with AIPEG, qualitative research was critical in identifying the pain points of business registration. One of our priorities going forward is in further exploring “thick data” — i.e. the qualitative insights derived from user research that ground-truths the intentions behind what people say and do and thereby augments findings from big data and open data.
- Looks matter (for data). Online platforms and applications are only as good as the data they represent. Complex and layered data are not instinctively meaningful to all policy decision makers (or those on whose support future funding rests). As one WFP colleague noted, “Visual communication is essential for decision makers who need dense information in a digestible way.” The journey mapping produced for studies on business registration and improving the import experience is one such example of clear presentation of detailed information and data that was well received.
- Sustainability remains a key challenge. A more significant challenge than generating good ideas is determining who best to implement these and make positive change happen. Development partners remain highly reliant on external consultants for technology and human-centred design work. And while embracing incubation in name, there is rarely the co-investment and longer-term mentoring provided to ensure the substance of incubation. When the consultant involvement stops, the project often stops or these technical elements are lost.
Food for thought for those of us who generate data tools to identify and support solutions, as well as those who implement these solutions.
Pulse Lab Jakarta is grateful for the generous support of the Government of Australia.
