Impact Canada (FR)
Published in

Impact Canada (FR)

Les effets de l’innovation — Mesurer les effets des défis

Conrad Noah, Analyste junior

Que sont les défis? Quels sont les résultats souhaités?

Les défis d’Impact Canada sont des concours axés sur les résultats, organisés par l’Unité de l’impact et de l’innovation (UII), en partenariat avec d’autres ministères, afin de trouver des solutions à certains des problèmes les plus urgents du Canada. Pour mener les défis à bien, on utilise une approche d’innovation ouverte et on mise sur des incitations financières et non financières pour encourager les innovateurs à orienter leurs talents vers la résolution de problèmes sociétaux majeurs. Généralement, les candidats aux défis s’inscrivent à un concours par étapes où les gagnants reçoivent d’importants prix en argent.

Je m’appelle Noah Conrad, et depuis quatre mois, j’aide l’UII à évaluer les défis. Voici un aperçu de la stratégie de mesure des retombées des défis conçue par l’UII et de certaines options envisagées pour l’avenir.

Pourquoi l’UII mesure-t-elle les retombées?

À mesure que les défis avancent, la mesure formelle de leurs retombées directes et indirectes aidera l’UII à déterminer s’ils génèrent les retombées souhaitées et s’ils sont conçus et réalisés de la manière la plus efficace possible. En effet, pour améliorer continuellement ses programmes, l’UII doit comprendre les aspects de ses méthodes qui fonctionnent comme prévu et ceux qui doivent être améliorés. De plus, vu les prix en argent associés à chaque défi, la mesure formelle des retombées des défis aidera à maximiser leur efficacité et leur efficience dans un contexte de politique publique.

Obstacles à surmonter

Il y a relativement peu d’études récentes qui examinent rigoureusement les retombées des défis. De par leur nature même, les défis portent sur des problèmes à multiples facettes qui n’ont pas forcément une seule cause ou solution. Par conséquent, il peut être difficile d’attribuer les changements positifs dans un domaine aux défis eux-mêmes, alors que de nombreux autres variables et facteurs pourraient avoir influencé les résultats observés. En outre, étant donné que chaque défi porte sur un problème distinct et repose sur l’une des trois structures de base (prix des défis, grands défis et accélérateurs concurrentiels), chacun des défis produira vraisemblablement des résultats selon un chemine qui lui est propre, ce qui complique l’élaboration d’une stratégie uniforme de mesure des retombées. Par exemple, un défi peut avoir une incidence sur une question en rendant des renseignements de haute qualité plus facilement disponibles (voir le premier cycle du défi d’offre de logements), tandis que d’autres peuvent changer un domaine en encourageant une percée technologique (voir le défi Visez haut). Il est donc clair que la détermination des retombées causales des défis nécessitera l’utilisation de méthodes et de pistes de recherche multiples. Les sections suivantes peuvent être considérées comme un sous-ensemble limité des stratégies de mesure des retombées que l’UII a utilisées jusqu’à présent et qu’elle pourrait utiliser à l’avenir.

In 2020, the IIU published a document called Logic Model and Narrative — Impact Assessment of Challenges under Impact Canada, which presents a basic theory of what we expect to achieve with our suite of challenges. Each of the methods of inquiry below is discussed in relation to the outcomes from the logic model that it is designed to measure.

Social Network Analysis: Measuring increased awareness, talent mobilization, and the formation of collaborative networks and partnerships

A desired process outcome from the logic model is the formation of partnerships both internal and external to the challenges themselves. Indeed, studies have shown that the formation of knowledge communities and networks through which actors exchange information are an important driver of innovation. As part of their design, challenges have developed various opportunities for participants to form connections with end users, potential financiers, local communities, and other competitors. Ideally, connections with financiers should help attract initial investments into start-ups while networking with fellow participants may assist in the exchange of ideas. Additionally, challenges should raise attention and generate increased awareness about important issues. Open communication from Impact Canada with strong engagement from innovators, end users, and the media contribute to raising the public profile of policy priorities and problems that a challenge has been designed to address or solve.

Another primary motivation behind the creation of challenges is to mobilize actors who do not typically apply to government programs as well as those who lie outside the normal constituency that may develop solutions to the issue in question. Challenges feature a streamlined application process designed to incentivize non-traditional actors to apply to them. It is expected that new actors who apply to challenges will bring new ideas, capacities, and experiences. These unconventional ways of thinking may help uncover new solutions to the difficult problems that challenges are designed to solve.

A line of inquiry which could help the IIU to measure all three of these outcomes is social network analysis. For those unfamiliar, social network analysis is a research method which outlines the exchanges and connections between actors[i]. Undertaking social network analysis would help paint a picture of the kinds of actors that are engaging in Impact Canada challenges as well as the interconnections between them. Beyond illustrating the connections between actors, social network analysis could reveal the relative centrality and importance of different actors within a network.

Surveys: Measuring the Mobilization of Talent

In addition to social network analysis, the IIU is making use of surveys to determine whether challenges are succeeding in attracting a wide variety of innovators. Surveys can help gather important information about applicants to understand who applies to challenges and what some of their motivations are. Some of the variables that can be analyzed from a survey include the age and gender distribution, location, background, and prior activity of applicants. Analysis of these variables will allow the IIU to measure the effectiveness of challenges in attracting new talent.

Quasi-Experimental Approaches: Measuring Economic Value

By de-risking investments into nascent technologies, challenges should help participants to raise new capital and financing. Evidence from a Nesta case study has shown that challenges are capable of helping participants generate investment into their products and solutions[ii]. Moreover, the staged approach that Impact Canada has taken to challenges in which non-winners are provided with various forms of support should help improve economic outcomes even amongst non-grand-prize-winners. Determining whether challenges have succeeded in generating greater public value will require the use of quasi-experimental approaches in which counter-factual outcomes from a control group are compared to the outcomes seen from applicants. By determining whether participants have experienced increases in economic indicators such as R&D spending, productivity, payroll spending, exports, patents, intangible assets, revenue, capital raised, and job creation, the IIU will be able to determine whether challenges have supported sustainable, innovative businesses that have experienced positive economic outcomes. Some initial work in this direction has begun in partnership with Statistics Canada (StatsCan). Moving forwards, the IIU will continue to collaborate with StatsCan to use data available in its Linkable File Environment to determine the most effective ways to measure the economic impacts of challenges while accounting for relevant covariates that could influence outcomes.[iii]

Short-Term Case Studies: Examining the Innovation Process

Many innovation oriented organizations have used case studies to evaluate the impacts of their programs. One reason for launching challenges is to change the incentives and conditions that currently exist around innovation in Canada. For example, the Women in Clean Tech challenge seeks to strengthen the innovation ecosystem in clean tech by creating a space for talented female entrepreneurs to develop and cultivate their ideas. In light of these goals, the IIU will consider using case studies that examine the impact of challenges on the innovation process. A pioneering approach here has been the short-term case study undertaken by Sitra of their challenge Ratkaisu 100[iv] which documents the impact of a challenge on participants as they are going through it. To date, the IIU has produced two case studies and will be publishing more in the future.

Long-Term Impact Studies: Measuring Public Value

As solutions developed throughout challenges come to be deployed in the real world, it will become imperative to examine the concrete impacts that these solutions are having on their respective issue areas. Through a longer term impact study, the IIU could examine the ways in which technologies, products, and services developed during challenges are being utilized. For example, if data eventually becomes available on the number of daily users the winning solution from the Drug Checking Technology Challenge has, the IIU could conduct statistical analysis about how the technology has been deployed to prevent overdoses within specific geographic areas. Each long-term impact study is likely to be very different given divergent pathways to impact that every challenge is likely to have. Nonetheless, long-term impact studies will ideally show how challenges have had an influence on some of Canada’s must pressing issues, responding to the longer-term outcomes of the challenges logic model, focused on societal, environmental and economic outcomes

Conclusion

As the IIU accumulates evidence on the effects of challenges, this knowledge will be used to understand the process and outcomes of challenges and to improve future challenges and initiatives. However, the importance of impact measurement extends beyond the need for institutional learning. Over the coming years, the insights obtained during impact measurement will help tell the story about how challenges have played a role in improving the lives of Canadians.

Notes

[i] Mohammed Saqr and Ahmad Alamro. ”The role of social network analysis as a learning analytics tool in online problem based learning,” BMC Medical Education 19, no.1 (2019): 1–11. https://doi.org/10.1186/s12909-019-1599-6

[ii] Nesta, Attracting Investment with challenge prizes, by Vidal Kumar. London, England: 2021, https://challenges.org/reports/attracting-investment-with-challenge-prizes/

[iii] Susan Athey and Guido Imbems. “The state of applied econometrics: Causality and policy evaluation,” The Journal of Economic Perspectives. 31, no.2 (2017): 3–32. https://doi.org/10.1257/jep.31.2.3

[iv] Sitra, Sparking social innovation through challenge prizes: Evidence on Teams, Ideas, and Incubation from Finland, by Tuuka Toivonen, Emma Nordbäck, and Ville Takala. ISBN: 978–952–347–052–1. Helsinki, Finland: 2018, sparking-social-innovation-through-challenge-prizes.pdf (sitra.fi)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store