Our Influence and Emerging Impact
Highlights from our 2021 Annual Report: The Long Read
Throughout 2021, we continued to rely on our results framework that defines impact as our contributions to change. This results framework is built on the premise that within the development ecosystem, we play a significant role in working with and strengthening the capacity of our stakeholders and partners to leverage data innovation, AI, and service design to improve public policy making, social intervention designs, and humanitarian responses. In particular, the framework lays down three main categories of impact by which our work brings value to our range of stakeholders and partners: methodological impact, ecosystemic impact and operational impact.
While developing prototypes and producing analytics is a big part of what we do, our intent has always been to ensure that our influence on our partners lasts well beyond the shelf lives of our prototypes and analytics. Such influence mainly manifests in the form of our counterparts’ shift in mindset; adoption of novel or more robust technical and analytical approaches; and lasting institutional changes in the way our counterparts approach solving complex development problems. Notably, 2021 also saw PLJ’s continuing shift towards a smaller but more focused portfolio. This shift has allowed us to put more energy and resources into contributing to these lasting changes, particularly regarding improving our counterparts’ capacity and systems to make full use of big data, AI, and service design. Compared to our previous annual reports, this year’s impact section has a shorter list of partners, but with a renewed focus on our lasting contributions to change.
Our reflection this year also draws from an Independent Strategic Review (ISR) that was commissioned by DFAT in 2021, which thoroughly reviewed DFAT’s “knowledge to policy” investments in Indonesia. The review asserts that PLJ has made notable success in contributing to evidence-based policy making, and in supporting increased gender equality, disability and social inclusion in Indonesia’s knowledge sector.
In the cases examined for the ISR, GoI partners expressed the view that PLJ’s support was significant in increasing their policy responsiveness and impact. While information alone is insufficient to deliver behaviour change, it can have a catalysing effect. It is clear from interviews that at least for those stakeholders, the ability to obtain relevant insights in real-time from non-conventional, secondary data and present the results in highly visual ways is valued. Furthermore, the value placed on access to up-to-date, relevant information not previously available has increased significantly during the pandemic. (Excerpt copied from the review)
Methodological Impact
In 2021, we strengthened our brand as a mixed-methods data innovation facility. With a clearer branding and the implementation of this approach in two of our key projects for the year, we started to gain recognition with an increase in demand to share our knowledge and experience in combining data analytics and service design. Our mixed-methods approach and the skills incorporated also made us well placed to support the United Nation’s Quintet of Change, especially in areas where we have already built our name, such as digital transformation, data analysis, and behavioural science. Reflecting on our methodological influence and impact throughout the year, below we discuss how we explored the use of agent-based modeling to inform COVID-19 response, experimented with behavioral economics principles to complement the design process, and introduced various aspects of our mixed-methods approach for project implementation to our main government counterparts.
As part of our collaboration with the West Java Government on COVID-19 response, we partnered with CSIRO to utilise agent- based modeling (ABM) to model the actions and interactions of individuals during the pandemic. Taking lessons learned from the risk assessment map that PLJ developed with Jabar Digital Service using big data and available administrative data, ABM was used to predict the likely outcomes if a regional government were –in this case the city of Bekasi of 3.5 million people– to introduce different social restriction scenarios. The use of ABM in this context provided insights on crowd dynamics and infection trends based on places of transmission (home, workplace, and school) to help inform decisions and policies to tackle the evolving pandemic.
The UN’s Quintet of Change acknowledges behavioural science as one of its five main knowledge areas to advance the UN agenda. This emerging shift was in line with PLJ’s continuing experiment with the approach, as demonstrated in our seminal research “Beyond Sticky Floors” that uses behavioural economics principles to complement our intervention design process geared at reducing barriers to adopting technologies among women micro business owners. This complementary approach allowed us to better understand individuals’ thought processes and how to nudge certain behaviours through design. Using this report to guide the discussions, we went on to share our learnings from experimenting with behavioural economics amongst colleagues and programmes working in the knowledge-to-policy space. Two of which included delivering a workshop for our UN colleagues in Pulse Lab Kampala in Uganda and leading an informal exchange with JPAL-SEA on behavioural science and its application to data analytics and service design.
The Lab’s efforts to introduce service design as a viable methodology to improve public services is highlighted by our collaboration with two line ministries.
By request from the Indonesian Ministry of Health and UNICEF, the Lab applied service design to better understand the gaps in the Ministry’s community-based nutrition recording and reporting system, known as e-PPBGM. The collaboration, which is still ongoing, allowed us to introduce the principles of service design to a key line ministry in Indonesia to improve data collection and reporting mechanisms related to children nutritional status at the community level. Our collaboration will also identify specific ways to improve data usage by district and provincial governments. Given the importance of e-PPGBM in providing a database to address stunting, the Lab’s work to introduce service design to the Ministry is expected to contribute to the country’s ongoing effort to improve its human capital. Through a collaboration with the Indonesian Ministry of Foreign Affairs, we worked to develop an in-house machine learning-based tool to more effectively analyse the Ministry’s large volumes of diplomatic information. Besides the data science application, part of the capacity building for the Ministry’s staff included leveraging a service blueprint to holistically understand the required systems and processes for the tool to function and to be fully optimised. This was another practical application of our mixed-methods approach to ensure greater value.
Our commitment to employing and disseminating the use of a mixed-methods approach is further demonstrated in our collaboration with the Ministry of Health (MoH) and Bappenas’ Directorate of SMEs and Cooperatives Development. In both cases, quantitative data-driven analytics and data mapping are complemented with service design to improve the quality of interaction between types of services and their users. We also utilised and introduced our Research Dive approach, which helped develop a shared understanding and learning through a hackathon-style, facilitated methodological process of analytics that are tailored to each organization’s context and objectives. For the Ministry of Health, the Research Dive sought to build a shared preliminary understanding of the potential and limitations of COVID-19 related health data and through the process develop concrete recommendations to help inform further research. More importantly, the Research Dive was carried out to inform the implementation of the Ministry of Health’s recently unveiled blueprint on digital health transformation strategy. For Bappenas, the Research Dive was focused on mapping available datasets that can be utilised to examine the spatial dynamics of Indonesia’s MSMEs.
PLJ’s contribution to the use of a mixed-methods approach to address social problems and improve public service is gaining attention not only domestically, but also internationally. Throughout 2021, we received several requests and presented on our mixed-methods approach in a number of international fora, including for the UK-based social innovation agency Nesta (more on this below), the Global Development Institute at the University of Manchester, and as part of a regional policy innovation exchange organised by UNDP Thailand Policy Lab.
Ecosystemic Impact
Strengthening our role as an analytic partnership accelerator, we continue to invest in building the capacity of our counterparts and fostering new partnerships. This is with full understanding that these are the foundations of a lasting data innovation ecosystem, which in turn is key to ensuring that our impact is sustainable. The Lab’s philosophy on collaborative research is centered on ensuring that capacity building is embedded throughout with targeted and contextualised activities and training sessions. We also contribute to the broader ecosystem as a community of practice, by exchanging knowledge through discussions and engagements in various conferences, and sharing methods, tools and knowledge in technical briefs, scientific journals and white papers.
The Lab’s impact on our partners’ capacity is particularly evident in three collaborations with the Ministry of Foreign Affairs (MoFA), Bappenas’ Directorate of Spatial Planning and Disaster Management, and Indonesia’s railway company PT KAI, all of which wrapped up in 2021. Our work with the MoFA, which began in 2019 and focused on utilizing machine learning to transform the Ministry’s analytical process of diplomatic information, served as a focal point for PLJ’s data scientists to strengthen the capacity of a dedicated team within the Ministry. Through the project-embedded capacity building, the MoFA team is now adept in using and maintaining the tool, including data inputting, system maintenance, information extraction, and data analysis. Our multi-year collaboration in leveraging machine learning in the diplomatic sphere has also both lent credibility to and strengthened MoFA’s role in championing digital diplomacy on the international stage, as showcased in the International Conference on Digital Diplomacy (ICDD) that the Ministry hosted in November 2021.
Meanwhile, our work with Bappenas’ Directorate of Spatial Planning and Disaster Management increased awareness and knowledge within the Directorate on both the potential and the challenges of using big data to inform disaster-related responses and policies. A mapping of available datasets and underlying issues was jointly conducted by PLJ and the Directorate, informing the latter’s plan to strengthen its data governance and data infrastructure in working towards “Satu Data Bencana” — an integrated data system to improve disaster management. In 2021, the Directorate hired a dedicated staff with this plan in mind to better manage and oversee disaster-related data to which the Directorate has access. The overall collaboration provided lessons that informed the development of a disaster management data preparedness roadmap for damage and loss assessment. Insights gleaned from this exercise were presented during a regional symposium organized by SIAP SIAGA (a DFAT-funded disaster risk management program), which discussed lessons from COVID-19 on data management and implications for future disaster response.
With respect to PT KAI, our collaboration has led not only to the realization of how the railway company’s passenger data can be utilised to transform public transport services, but also to the adoption by PT KAI of a robust analytical process to produce evidence-based insights. In a follow-up meeting after the project closed, PT KAI Commissioners expressed satisfaction with the way PLJ worked with the company’s analytics team, especially by utilizing a structured analytical process with scientific methodologies to analyse its data. These processes have been adopted by PT KAI’s analytics team and are becoming a part of the company’s research culture. An op-ed co-authored by one of PT KAI’s commissioners with the UN Resident Coordinator in Indonesia and Head of PLJ was published in the Jakarta Post in 2021, which further affirmed the railway company’s commitment to leverage big data to transform Indonesia’s public transport system, with a focus on improving accessibility for vulnerable cohorts. Coming full circle with our contributions to the ecosystem, the research also highlighted possibilities for transport integration and partnerships with other actors operating in the public-private transport sector towards inclusive transport planning.
2021 also saw PLJ’s significant contribution in advocating for gender equality and social inclusion, especially for people with disabilities, in the data innovation ecosystem. Among our most notable impact for the year is in the diplomatic sphere, as an invited speaker at the International Conference on Digital Diplomacy (ICDD), hosted by MoFA. The Ministry also requested the Lab and other partners to review and provide input on the “Bali Message”, a joint statement of all participating states around digital diplomacy, as a key output of the international conference. Our inputs led to the incorporation of a statement on ensuring digital inclusion, particularly of women, children, the elderly, and people with disabilities, as well as promoting the attainment of the Sustainable Development goals. The Lab’s commitment to serve as a champion for gender equality, disability and social inclusion is also reflected in our key role in the implementation of the UN Data Strategy, where we are providing advisory support to the UN Country Team in Indonesia in establishing a use case focussing on people with disabilities data.
Our impact in the international data innovation ecosystem can be seen with our ongoing work with the Global South AI4COVID Program, where we continue to contribute to policy influence, engagement, and advocacy in the use of data science and AI approaches to address COVID-19 response across 18 countries in the Global South.
PLJ has also been a participating UN entity for the development and implementation of the UN-ASEAN Joint Strategic Plan of Action for Disaster Management 2021–2025. We are providing key support in building awareness in ASEAN in the adoption of new data innovations to increase effectiveness of interventions and responses, and strengthening collaboration and sharing of best practices in application of data innovation in humanitarian response in the Southeast Asian region. Related to COVID-19 and public health emergencies, Harvard University’s Radcliffe Institute of Advanced Studies invited PLJ to participate in its “Safe, Responsible, Fair and Equitable use of Human Mobility Data” seminar, and PLJ provided inputs to a white paper on the topic that was subsequently published.
Members of the Lab’s senior management have also been requested to sit in advisory committees, including our Data Policy and Innovation Lead’s position as a member in Data for Development’s Expert Advisory Committee of the new Responsible Artificial Intelligence Index (RAII). Through this, PLJ is providing inputs by reviewing the methodological framework for the index; advising on the inclusivity of the design process of the index; and advising on capacity building initiatives and mechanisms to advance the impact and reach of the index. Our Social Systems Lead was requested by Nesta, a leading innovation agency for social good from the UK, to share how PLJ operationalizes its mixed-methods approach. Delivered in the form of video recording, the presentation was well received by Nesta and our story has become an important reference point for the innovation agency to develop their own mixed-methods approach. Following this exchange, Nesta requested our Social Systems Lead to sit as an Advisory Committee member for their Collective Crisis Intelligence Project with IFRC. In addition, our Head of Lab is a member of the independent panel of commissioners for the Global Commission on Evidence, where he has advocated for artificial intelligence to be responsibly leveraged to meet the demands for “living” and rapidly updated evidence to address complex social challenges, and cautioned against the potential risk of discrimination when using data.
PLJ also continued to establish new partnerships with Indonesia’s line ministries and agencies in the utilization of data innovation and service design. With the endorsement of the Lab’s Steering Committee, in 2021 we entered into collaborations with the Ministry of Health (MoH) through its newly established Digital Transformation Office (DTO) and further collaborated with Bappenas through the Directorate of SMEs and Cooperatives Development. With the former, PLJ worked with MoH’s DTO in improving the Ministry’s understanding of its data that are coming from different systems. With emphasis on responding to the COVID-19 pandemic, a key objective was to explore what insights could be gleaned from combining available datasets. With the latter, the underlying objective was to use evidence to answer key policy questions to improve the growth of MSMEs and their contribution to Indonesia’s economy. This is done by mapping existing datasets that are relevant, identifying data gaps, and potentially leveraging foresight to help shape the future of Indonesia’s MSMEs. It should also be noted that PLJ’s collaboration with the MoH was catalyzed by our work with Jabar Digital Service, further showcasing PLJ’s contribution to a dynamic data innovation ecosystem in Indonesia.
Operational Impact
2021 saw PLJ’s impact on our partners’ processes in policy making, systems strengthening, and public service delivery in multiple sectors, including COVID-19 response, public transportation, diplomacy, women-owned businesses, and health.
In several instances, we discovered that our impact on our partners’ operation come not only from the uptake of our analytics and prototypes, but also from the process and approach that we introduced and our partners adopted.
Our collaboration with West Java’s digital unit Jabar Digital Service on providing data-driven evidence for the provincial government’s COVID-19 response continues to evolve. The collaboration has produced a data analytics interactive map on COVID-19 transmission risk for the province. A user research, subsequently done by PLJ, identified potential paths for uptake by relevant decision makers. By 2020, the data analysis and visualization tool was installed in the provincial Command Center. In 2021, they were integrated into the West Java Province’s Center for Information and Coordination, Pikobar. As part of Pikobar, the interactive map can now be accessed by the public, informing both policy makers and the population alike on the potential of transmission risks in West Java villages. Beyond the use of the interactive map, our collaboration has also influenced the way Jabar Digital Service conducts research and develops products, as highlighted in our follow up meeting after the project closed. In particular, the agency found high value in PLJ’s user research and has begun to adopt the approach to identify paths for uptake for their future analytics, prototypes, and products.
With PT KAI, the Lab demonstrated how big data can help transform Indonesia’s public transport system and make it more inclusive by first understanding the diverse needs of the country’s passengers. The collaboration leveraged the company’s massive passenger data to better understand the behavior and needs of PT KAI’s customer base, and especially the needs of vulnerable populations who use its railway. Using 2019 data, our analysis found that half of all passengers that year were female, with a significant number of them travelling in the evening hours. Furthermore, whilst more than 1.5 million trips were completed by passengers over 60 years old in 2019, the data shows that only 4% of elderly passengers used the Government’s travel subsidy. Finally, the collaboration also highlighted the lack of quality data on passengers with disabilities. Based on that, PT KAI is now committed to catering to the unique needs of people with disabilities and is looking into more systematically capturing data on how this group is using their services. The railway company has also leveraged on their experience working with us to commission a dashboard based on their massive passenger and transported goods data, which is modeled after the data visualisation that PLJ developed.
Our work with MoFA, which began in 2019 and focused on utilizing machine learning to transform the Ministry’s analytical process of diplomatic information, came to a close in 2021. To quickly reiterate earlier descriptions of this collaboration, PLJ and MoFA developed a machine learning tool to analyze large volumes of texts from diplomatic cables and internal communications to extract meaningful information. As we’ve reported in last year’s annual report, in 2020 the tool was embedded onto MoFA’s monitoring and repository system. In 2021, the tool’s potential to inform diplomatic policies and decisions was brought to the attention of the highest echelons. The Minister and Vice Minister are now fully aware of how the tool can significantly improve the way data-driven diplomatic insights can inform decisions. At a formal presentation, the Minister expressed her support for the related agencies to work towards institutionalizing the tool, potentially by focusing on specific directorates to make full use of the tool.
On the analytics front, recommendations from the policy research that PLJ conducted with UN Women and Gojek, launched in December 2020, was presented to Indonesia’s Committee for COVID-19 Response and National Economic Recovery and the webinar on Economic Empowerment of the ASEAN MSMEs Entrepreneurs in the Digital Era during COVID-19 to support women-owned MSMEs to cope with the pandemic. Subsequently, our use of a human-centered design lens on the qualitative data we collected for the research has produced valuable insights on behavioral barriers among women micro business owners to make full use of digitalization. The report “Beyond Sticky Floors” sketched several archetypes and identified specific behavioral challenges that hold back each archetype from using digital tools. This independent research, which was subsequently featured in the 2021 UN Behavioral Science Report, highlights these challenges, which go beyond structural issues, and bring them to the forefront of the discourse to support women business owners improve their livelihoods by capitalizing on digital technologies.
From our collaboration with MoH’s Digital Transformation Office (DTO), several of the findings from the research dive we conducted with them and the approach that we utilised so far have already influenced DTO’s operations. This is particularly true in terms of data governance and data sharing protocols. MoH has always put the utmost importance in the protection of sensitive personal data that they collect. Our collaboration identified specific ways for the DTO to further improve that protection, including a renewed protocol on data masking and improving access controls to the existing health data lake. MoH is already starting to adopt these changes as part of its larger digital transformation agenda.
While our collaborations, services, analytics, and prototypes continue to give direct operational impacts, we are now starting to capture evidence from where our strategy has left behind lasting changes in our partners’ operations. The Financial Access Map that we handed over to the Indonesian National Council for Financial Inclusion in 2018 is still being continuously improved by late 2021, with scaled-up geographic coverage and data points. Our work with PT KAI strengthened their staff’s capacity and provided them with a working experience that after our collaboration ended, the company commissioned and oversaw a passengers and transported goods dashboard modeled after our data visualization. The Regional Conference on Digital Diplomacy that we helped to initiate with MoFA in 2018 has evolved into the International Conference on Digital Diplomacy in 2021, which MoFA continues to host. And our work with Jabar Digital Service left a lasting influence on their operations, particularly through our introduction of the user research methodology to identify pathways for product uptake and influencing policy with evidence.
The DFAT-commissioned Independent Strategic Review highlighted how PLJ’s focus has never been on making changes to “hard” policy (i.e. laws, regulations, formal government statements of priority). Instead, we have always put greater engagement on bottom-up changes in “soft” policy, which is all about changing the operational practice and procedure that existing policies are implemented or “promoting new policies through new ways of working enabled by the technologies.” The review rings true as we continue to capture these long- lasting changes in our partners’ ways of working due to our influence as an analytic partnership accelerator.
Pulse Lab Jakarta is grateful for the generous support from the Government of Australia
