A better way to share data

Yes, this is sometimes how data exchange happens.

CalData, in partnership with California’s Health and Human Services Agency (CalHHS), is excited to announce the release of our Interagency Data Exchange Agreement (IDEA) guidebook. We hope that by making our resources public, it can help other jurisdictions adopt this approach.

IDEA is a single legal framework for interdepartmental data sharing at the state. IDEA solves the historically time consuming and complex method of sharing data across departments.

Data sharing is critical to improve services for Californians: linking data allows us to develop holistic versus piecemeal approaches. Simplifying data sharing is a priority for CalData and Objective #4 of the Statewide Data Strategy.

Initial results have been impressive. After CalHHS adopted IDEA, completion time for data sharing agreements went from 2–3 years to an average of 79 days.

To understand why IDEA matters, you first need to understand both why we need data sharing and why it’s hard to do in the public sector.

Important challenges require cross-department data

Californians touch many State services every day. Each of these services have separate back end systems that store these touchpoints.

While departments use their data to improve services, few of the challenges and opportunities facing Californians are contained in a single data system. Data related to complex topics like housing, job readiness, and education outcomes is scattered across many departments and their data systems. This is why the state has made significant investments in systems like the Homelessness Data Integrated Systems (HDIS) and the Cradle-to-Career System (C2C).

If we try to improve services using data from a single department’s system we risk solving for the wrong thing. Like the proverbial story of the blind men and the elephant, each department or program only sees part of the problem. It is only through holistic data that we can develop holistic approaches as well as understand the impact of our services across multiple areas such as education, housing, economic stability, health, and more.

Caption: Like the proverbial blind men and the elephant, looking at piecemeal data leads to piecemeal solutions.

Traditional data sharing is overly complex

If we’re going to share data, we need to ensure it is done ethically, securely, and per governing statute and regulations to ensure privacy and appropriate use.

However, unlike the private sector, public sector data use and sharing is highly regulated by a series of overlapping statutes and regulations at both the federal and state level. Ensuring compliance and adherence with this legal complexity is time consuming and daunting.

Moreover, most data sharing agreements are initiated in a point-to-point fashion. So often data sharing use cases would start from scratch to develop the complex legal, privacy, and security terms that are essential for safe data exchange.

Given this, most agreements took 1–3 years to finish. (I’ve even seen data sharing agreements take 10 years!) The point-to-point approach can also lead to inconsistencies in terms and legal interpretations.

Lastly, renewals or changes often result in complex renegotiations given staff turnover.

These challenges are not unique to California — they are rife across all levels and jurisdictions in the United States. Unlike other countries or coalitions, we lack an omnibus data privacy framework.

In California, we sought a more streamlined and centralized approach that also ensured responsible data exchange.

Caption: Data exchange historically involves complex and bespoke agreements that can take years to negotiate. When it comes to renewal, a new and equally complex process can result.

IDEA structured to focus on what matters

The two level structure of IDEA is designed to address these challenges:

1. The top level is an umbrella agreement that commits departments to data sharing consistent with a comprehensive set of legal, privacy, and security terms that has been signed by participating state entities. It also includes provisions for escalation and mediation in the case of disagreement.
2. The second level is a much shorter and scoped Business Use Case Proposal (BUCP). The BUCP documents the terms of individual data exchange agreements. This allows departments to specify what data is exchanged, for how long, and to incorporate additional security and privacy terms if needed. It covers topics such as data fields to be shared, intended use, legal justification, and other use case specific details.

Separating BUCPs from the main agreement frees up legal and staff time to focus on the specific details of the data exchange. It also reduces the effort to manage changes and renewals for each data sharing instance: only the BUCP needs to be updated.

Caption: Moving from point to point agreements to a single umbrella agreement streamlines administrative effort and enforces legal compliance.

For the over 100 state entities that have signed onto IDEA, we now have a single, streamlined process. CalData has drafted a guidebook to help use IDEA. It contains webinars, template emails, and frequently asked questions.

Many smart hands brought this to life

We want to acknowledge the foundational work CalHHS provided and their continual partnership and support in bringing this agreement to life.

Starting with a 2014 workgroup, CalHHS developed an agency-wide data sharing agreement that served as the model for IDEA. In 2020, CalData helped expand the framework to the entire executive branch. We are proud to have over 100 state entities signed onto the agreement.

Going forward, we have set up metrics to monitor how IDEA is performing in terms of simplifying data exchange. As needed, we will make periodic updates to the BUCP to further streamline data sharing negotiations.