Causal inference for discerning the impact of research grants on company success
This article is part of Discovery Hub’s Data Test Kitchen series. We explore how data science and machine learning can help enhance foresight on emerging innovations and their social impact.
The Discovery Hub aims to combine strategic foresight with data-driven approaches, to provide additional rigour to the way we anticipate change in Nesta’s mission areas such as health, early years and decarbonisation.
Our previous blog explored the potential for machine learning approaches to make predictions about the future success of start-ups. These predictions could provide us with clues about the businesses and sectors to watch closely for developments.
Here, we explore the use of causal inference methods to provide data-informed recommendations on policy and funding interventions. This task requires a different set of tools than when making predictions, as we now need to carefully consider the causal relationships between the variables we’re interested in.
As our test case, we considered the funding lever of using government research grants to catalyse growth in target sectors, the underlying hypothesis being that the distribution of such grants will lead to future company success and wider growth in the sector.
We used data on research grants and venture capital investment into start-up companies and applied the method of causal inference to estimate that research grants increase the likelihood of startup future success by more than 20%.
Inferring the effect of research grants on startup success
We were prompted to explore the effect of research grants on company success because of our research on low-carbon heating technologies. We found a gap between strong public research and development funding into these technologies versus relatively weaker private venture capital investment. Given these rates of public funding, we were curious whether we should be anticipating higher growth of venture capital investment to follow in the future.
This is also an area where we can cross-reference the results from our experiments with previous research undertaken by others. For example, a study from 2020 by the Department for Business, Energy and Industrial Strategy found that each £1 of grants provided leads to an increase in private investment of between £0.41 and £0.74 in the first year and between £1.96 and £2.34 in the long run.
The association between grants and future private investment has also been recently looked at for companies founded out of higher education institutions (‘spin-outs’). Spin-outs that received funding from Innovate UK and the British Business Bank went on to receive greater sums of private investment (£7.3m vs. £1.6m in their first eight years) and had more investment rounds (4.6 vs. 2.9 average rounds) when compared to spin-outs without such funding.
Here, we use the same datasets as in our previous exploration into predicting company success. By linking Crunchbase, Gateway to Research and BEIS/Nesta Research & Development data, we created a dataset containing over 30,000 UK companies with information about the company, founders, investments, investors, research grants, and location-based indicators.
We included all UK companies that were founded between the start of 2011 and the end of 2018 and had not already been acquired or had an initial public offering (IPO) during this time period. The information available in the input variables was limited to simulate what would have been available at the end of the year 2018. Future success for these companies was then defined as whether they go on to receive further investment, be acquired or have an IPO between 2019 and 2022.
Predictions vs. intervention effects
Making predictions can be thought of as estimating the value of one variable given that you have observed another variable (or a set of variables). For example, if we have multiple past observations of the wind blowing and a branch shaking, when we observe a branch shaking we can predict that the wind is blowing.
Traditional machine learning methods are good at making these types of predictions. However, these methods do not use any information relating to the causal relationships between the wind and the branch.
Understanding the effect of an intervention can be thought of as calculating the increase in the probability of the outcome, given that another event has occurred. Using the previous example, what would happen to the wind if there was an intervention such as a human shaking the branch?
If we were to look at the past observations, we may wrongly draw the conclusion that when someone is shaking the branch, the wind is blowing. We know this is not the case because we understand the causal relationship between these two events. This is an example of association not being the same as causation.
Causal inference methods use both data and theory about causal relationships and therefore provide a principled approach to estimating the effect of interventions, which can inform decisions about what actions to pursue.
The causal relationships can be captured in a causal graph where each relevant factor is represented as a node and causation is represented by arrows between the nodes. The causal relationship between the wind and the branch nodes is shown in the figure below.
Average treatment effects
Ideally, to calculate the causal effect of an intervention for an individual case, we would measure the outcome both with the intervention and without the intervention. The problem with trying to estimate individual causal effects is that only one of the outcomes occurs in real life.
For example, if you have a headache and take a tablet and the headache goes away, you cannot compare what would have happened if you didn’t take the tablet: you cannot observe the counterfactual. It is therefore impossible to calculate the causal effect at the individual level.
However, it is possible to estimate the average treatment effect (ATE) of a population. Randomised controlled trials (RCTs) are one method to estimate the ATE. In an RCT, the participants are randomly assigned to either receive or not receive the intervention, then the outcomes are measured and the ATE is calculated. Due to the randomisation of the RCT, association can be interpreted as causation in this case.
For some kinds of interventions, it may not be possible or ethical to run an RCT. Causal inference methods can be used to reach similar conclusions to running an RCT but using observational data instead. Observational data captures data on outcomes but where the interventions have not been randomly assigned. Statistical techniques can then be used to estimate the causal effect from the observational data.
Causal inference analysis
To perform the causal analysis we used the Python library DoWhy, which provides the tools for a four-step process of modelling, identification, estimation and refutation. These steps are explained in more detail below.
Step 1: Creating a causal graph
As mentioned above, causal inference methods make use of both data and the causal relationships between the data. These causal relationships are recorded in a graph where each node is a variable in the dataset and each arrow represents causation. Causal graphs are best designed with domain experts. Our graph is shown below and its design was influenced by our consultations with colleagues from Nesta’s Impact Investments and Mission Studio teams.
It is interesting to note our experience in trialling these consultation workshops. We used a Miro board, which had all the variables in the dataset — and through discussions with our colleagues, we aimed to draw arrows between the variables to indicate causal relationships.
This was thought-provoking and made us consider more carefully the system we are working with (here, the world of venture capitalists and start-ups) and also uncovered gaps in our datasets. These gaps related to:
- Missing data such as company financial records and the local innovation ecosystem (the BEIS/Nesta Research & Development data was added to fill this gap);
- More subtle points that are harder to capture with data but are nevertheless critical when venture capitalists are considering investing in a company such as, “whether the company is solving a pain point in the market” or “whether the company is making something 10x better”.
Acknowledging the gaps, we interpreted the insights from the consultations to arrive at the final causal graph (note that in this visualisation we have omitted some nodes relating to industry and location-based factors for clarity).
Step 2: Causal identification
The next step is to use the causal graph to check that our question — what is the average causal effect of receiving a grant on company success — can be answered based on our current state of knowledge. If our question can be answered, a probability formula will be identified that can be statistically estimated using the data in the following step to answer our question.
Step 3: Estimating the average treatment effect of a grant on future company success
Now that we have the recipe for answering the causal question, we can estimate the effect using statistical or machine learning estimation methods. To estimate the treatment effect, we need to be able to separate the effect of the treatment from the effect of other confounding factors.
We use a method called propensity score weighting. This method simulates the treatment assignment mechanism by estimating the probability of a company receiving a grant. This method aids in isolating the treatment effect from other differences between the group that received a grant and the group that did not receive a grant. The groups are therefore comparable and enable us to estimate the difference in success caused by a company receiving a grant.
The differences in success can be calculated and averaged to give us the average treatment effect.
We found that the estimated effect of a company receiving a grant leads to an average of 22.3% (95% confidence interval: 20.4% — 24.3%) increase in the likelihood of future success. Similarly, the estimated effect of a company receiving a UKRI grant leads to an average of a 22.8% (95% confidence interval: 20.0% — 25.0%) increase.
Step 4: Refutation — model robustness checks
A downside to causal inference methods is that the assumptions in the causal graph are difficult to test and the quality of the outputs they produce are hard to assess. There are a few techniques that can be used to refute the estimator such as:
- Simulating a placebo and checking that the estimated causal effect is close to 0%;
- Adding a random variable as a common cause and checking that the estimated causal effect does not change;
- Using a subset of the dataset rather than the full dataset and checking that the estimated causal effect remains similar compared to when using the full dataset.
These tests alter the estimator and calculate the new effect with an associated p-value. If the p-value is below 0.05, we can say the estimator fails the refutation test. Our estimator was found to be robust to these refutations with all p-values being greater than 0.05.
Conclusion
So, what does this mean? It suggests that providing grants to companies (as, for example, Innovate UK is doing) likely represents an effective policy intervention as it boosts their chance of raising more investment, going public or being acquired.
This is not surprising and aligns with intuition and other existing evidence, but the work has laid the foundations for us to continue experimenting with this methodology and leverage it in future Discovery Hub projects.
It is interesting that the effect of receiving a research grant on company success was quite substantial (more than 20%). From our prediction analysis using more traditional machine learning methods, the model did not find the research grants related features very important for making predictions. This shows the differences in these approaches and highlights that feature importances do not represent causal effects.
We could further improve this analysis by expanding the range of data sources to capture more information about the companies such as revenues, patents, media and policy discourse, the novelty of the company’s work, text data from the company descriptions and the diversity of the skills of the staff.
This work could be also taken further by examining the influence research grants have in specific sectors, as it is possible that in some sectors grants have different effects. We are especially keen to look into the sectors most relevant to Nesta’s missions, for example, green tech, food tech and early years education.
Thank you to Karlis Kanders for his guidance throughout the project and his contributions to this blog. We thank Nesta’s Data Analytics Practice for providing the Gateway to Research and Crunchbase business intelligence data. We are grateful to Alex Gilbert and Alex Hook for their insights and advice about startups and investments. We thank Celia Hannon, Federico Andreis and Siobhan Chan for their helpful comments on this article.
If you would like to discuss the project further please reach out in the comments below. The code for this work is available on Github here and here. The Crunchbase dataset unfortunately cannot be shared as it contains proprietary data, whereas Gateway to Research and BEIS/Nesta R&D spatial data are openly accessible.
