Replicating the Justice Data Lab in the USA: Key Considerations
By Tracey Gyateng and Tris Lumley
Since 2011, NPC has researched, supported and advocated for the development of impact-focussed Data Labs in the UK. The goal has been to unlock government administrative data so that organisations (primarily nonprofits) who provide a social service can understand the impact of their services on the people who use them.
So far, one of these Data Labs has been developed to measure re-offending outcomes- the Justice Data Lab-, and others are currently being piloted for employment and education. Given our seven years of work in this area, we at NPC have decided to reflect on the key factors needed to create a Data Lab with our report: How to Create an Impact Data Lab. This blog outlines these factors, examines whether they are present in the USA, and asks what the next steps should be — drawing on the research undertaken with the Governance Lab.
What are Data Labs and why are they needed?
A Data Lab is the popular name among social researchers to describe a team of analysts/researchers working to produce innovative data solutions. It encompasses a variety of models — from services that enable researchers to access de-identified data within a secure setting, to services where bespoke research is conducted. Here we talk about Data Labs as a standardised evaluation service provided by government departments, or government approved agencies. The model enables providers of services to measure their impact on social issues, such as reoffending, through use of government administrative data sets.
Nonprofits need to evaluate their programmes to assess if their services are operating as expected, and to provide evidence to service users, donors, and funders of the efficacy of their work. Good evaluation requires robust data, but many charities struggle to access or acquire such data. When they can access it, few also have the skills needed to analyse these complex datasets.
Meanwhile, governments routinely collect administrative data from services that are offered to the public. This administrative data is increasingly recognised as being of value, especially if it is opened up for other organisations to make use of. It is in government’s (and society’s) interest as key purchasers and deliverers of social interventions to support evaluations of these services. Understanding which programmes are effective in addressing social issues would support effective decision making for allocating scarce resources. The challenge is that administrative data about individuals is personal, highly sensitive, and must be kept secure — which is a core feature of a Data Lab.
Key factors for developing a Data Lab
By advocating for Data Labs to be set up in several UK government departments, we have identified the most important requirements to aid this process. Full discussions of these requirements are provided in the paper How to Create an Impact Data Lab. Below we examine the key factors and to what extent they appear to be present within the USA.
Environment: A broad culture that supports impact measurement. Similar to the UK, nonprofits in the USA are increasingly measuring the impact they have had on the participants of their service and sharing the difficulties of undertaking robust, high quality evaluations¹.
Data: Individual person-level administrative data. A key difference between the two countries is that, in the USA, personal data on social services tends to be held at a local, rather than central level. In the UK social services data such as reoffending, education and employment are collated into a central database. In the USA, the federal government has limited centrally collated personal data, instead this data can be found at state/city level.
Large population sizes of personal identifiable data is needed for organisations to understand the outcomes of their services and to estimate a counterfactual group.² This can be difficult to achieve at a local level if the data set is not large enough. Compounding the problem, the robustness of estimates produced by a Data Lab is likely to be negatively affected if there is a large amount of migration out of the state/city as the outcomes of these individuals will not be tracked. We suspect this migration issue will be more acute at state rather than national level but State sized Data Labs will need to test what effect this would have.
A solution to problems of scale may be more data sharing and there are some early initiatives to this end taking place. The Actionable Intelligence for Social Policy (AISP) unit based at the University of Pennsylvania are leading experts in developing integrated data systems (see our case study here) and are creating a network of Administrative Data Research Facilities (ADRF) to develop standards for accessing and using administrative data. Further in partnership with Third Sector Capital Partners, Inc, AISP are working with five governments to develop integrated data systems which will enable program evaluation.³
A leading advocate: A Data Lab project team, and strong networks. Data Labs do not manifest by themselves. They requires a lead agency to campaign with, and on behalf of, nonprofits to set out a persuasive case for their development. In the USA, we have developed a partnership with the Governance Lab to seek out opportunities where Data Labs can be established but given the size of the country, there is scope for further collaborations/ and or advocates to be identified and supported.
Customers: Identifiable organisations that would use the Data Lab. Initial discussions with several US nonprofits and academia indicate support for a Data Lab in their context. Broad consultation based on an agreed region and outcome(s) will be needed to fully assess the potential customer base.
Data owners: Engaged civil servants. Generating buy-in and persuading various stakeholders including data owners, analysts and politicians is a critical part of setting up a data lab. While the exact profiles of the right people to approach can only be assessed once a region and outcome(s) of interest have been chosen, there are encouraging signs, such as the passing of the Foundations for Evidence-Based Policy Making Act of 2017 in the house of representatives which, among other things, mandates the appointment of “Chief Evaluation Officers” in government departments- suggesting that there is bipartisan support for increased data-driven policy evaluation.
Legal and ethical governance: A legal framework for sharing data. In the UK, all personal data is subject to data protection legislation, which provides standardised governance for how personal data can be processed across the country and within the European Union. A universal data protection framework does not exist within the USA, therefore data sharing agreements between customers and government data-owners will need to be designed for the purposes of Data Labs, unless there are existing agreements that enable data sharing for research purposes. This will need to be investigated at the state/city level of a desired Data Lab.
Funding: Resource and support for driving the set-up of the Data Lab. Most of our policy lab case studies were funded by a mixture of philanthropy and government grants. It is expected that a similar mixed funding model will need to be created to establish Data Labs. One alternative is the model adopted by the Washington State Institute for Public Policy (WSIPP), which was created by the Washington State Legislature and is funded on a project basis, primarily by the state. Additionally funding will be needed to enable advocates of a Data Lab to campaign for the service.
Where next for Data Labs in the USA?
This project has identified positive indications that the environment, advocacy, customers, data owners and funding make the USA a potentially fertile environment for Data Labs. Further progress is reliant on choosing a state to begin initial testing for a Data Lab. The choice of state should depend on all of the factors outlined above, but the availability of personal data and the governance around access will take precedence.
Once a region is chosen, the issues a Data Lab might work on can be evaluated; the barriers which might exist can be identified; a list of the key partners and data sharing initiatives taking place in that area can be drawn up and an assessment of the likely demand will need to be conducted. NPC conducted this investigation when shortlisting the additional Data Labs to be developed following the Justice Data Lab, see here to read the report, which we recommend should be replicated.
If you would like to discuss any of the issues raised within this post do get in touch with the programme director Tris Lumley
¹ See Bach-Mortensen, A. M. and Montgomery, P. (2018). What are the barriers and facilitators for third sector organisations (non-profits) to evaluate their services? A systematic review. Systematic Reviews, 7:13 doi.org/10.1186/s13643–018–0681–1
² Counter-factual- what would have happened to the group if they did not receive an intervention