How Data Can Help Solve Climate Change, Part 2

My last post made the claim that data is a key lever for reshaping the climate-change-response system and proposed a high-level design of a data platform for climate — a combination of private data and public data platform infrastructure, data applications, data standards, and data sources:

In this post I’m going to dig deeper into how a climate data platform like this can solve key problems in climate change response, and what it would take to build it.

Solving problems for private organizations (producers & solution providers)

Private organizations — particularly the producers in my system taxonomy who generate GHG emissions — are increasingly facing the need for sophisticated “climate management” solutions. Climate management helps producers achieve regulatory compliance (e.g., SEC- or shareholder-mandated ESG reporting), capture economic incentives (like IRA tax credits), reduce energy use & expenses, reduce climate adaptation risk, or achieve green commitments like going carbon-neutral or -negative.

A common thread in climate management is the need for high-quality data. For example, a producer trying to reduce its scope 3 emissions footprint needs data on the emissions of its supply chain in order to identify high-emission suppliers and choose or negotiate better alternatives. A producer trying to achieve net-zero operation needs data to account for and aggregate all its emissions across various scopes, identify optimization opportunities, and navigate the carbon marketplace to find and secure high-quality carbon removal or offsets to balance the books. And on the flip side, organizations creating climate tech solutions (like carbon removal or lower-carbon processes) need high-quality data to determine and demonstrate the effectiveness of their solutions, and reporting to help them and their producer clients secure regulatory credits or sell effectively in a carbon marketplace.

The solution to these data needs is private data platforms. Private data platforms are software infrastructures that provide foundational capabilities for ingesting, cleaning, standardizing, and storing data from relevant data sources — along with aggregation, query, and analysis capabilities (including ML algorithms), and reporting engines.

They handle the complete flow of climate-related data for an organization, starting with data capture — e.g. direct instrumentation for producers’ scope 1 resource use and scope 2 energy use or MRV instrumentation for carbon removal solutions — as well as integrating external data like weather data for climate risk analysis or emissions data from partners and suppliers for scope 3 emissions (who might be using their own private data platforms). Using this acquired data, appropriately aggregated and transformed, they power data applications with rich modeling capabilities to translate ingested data on activity into estimates and reports of carbon/energy/resource impacts.These applications provide analytics, simulation, and other tools to enable what-if analysis and support decision-making and reporting. Importantly, the value of these data applications justifies the often-expensive work to instrument and capture climate data and bring it into the platform.

Private data platforms appear to have momentum in climate tech, with quite a few companies — like CarbonChain, Gravity, Persefoni, Sinai, Watershed — active in the area. Patch is an interesting variant that appears to be focusing more on an API-centric platform than end use case applications. Venture funding in the space is growing and accounts for 8% of recent climate VC investment (per CTVC).

Solving public-scale climate problems with data

As valuable as they are, private data platforms are limited to solving climate problems for single organizations or value chains. Alone, they can’t solve the larger, public-scale problems that we face in climate response, like:

• Global resource management: understanding use of key constrained resources like water, minerals, metals, fossil fuels, biomass, refrigerants, even the capacity of the atmosphere and oceans to absorb carbon, and managing that use through mechanisms like caps and allocations
• Distributed energy management: ensuring that energy production, storage, and demand are in balance, taking advantage of novel distributed generation and storage technologies, activating demand management, and adapting to the limitations of renewable-centric generation
• Climate policy & economic design: understanding in detail the flows of carbon and energy in the world, identifying and modeling improvement opportunities, and designing effective incentives, subsidies, caps, penalties, and even novel economic constructs (like carbon currencies) to shape the behavior of system actors
• Climate financing: providing funding institutions with the data needed to understand the demand for climate-transformation capital, create the right instruments to enable transformation, and prioritize investments accordingly
• Reshaping the cultural narrative: creating compelling storytelling to shift mental models and activate public and political pressure for climate solutions. Data enables storytelling by providing a grounding of facts and accountability that can cut through disinformation and opinions. It also provides visibility to progress, challenge, opportunity, and hope that can be the basis of compelling stories.

Solving any of these problems is predicated on having a global view of the climate-response system, requiring free-flowing, high-quality data from across the system — i.e., a public climate data commons that interconnects the data stored in private data platforms, augments it with existing global public data, and enables an ecosystem of system-wide data analytics and applications. Creating such a data commons also would enable an unprecedented level of transparency, verifiability, and accountability across the system — fully activating data as the system-wide shaping force I claim it can be.

But what would a climate data commons look like, practically? It would need:

• Schema & metric standards — to ensure a common, trusted lingua franca for climate data and ensure data quality/trustworthiness. Discussed more below.
• Registries & repositories — canonical services that ingest, register, and store data in the commons and provide a single source of truth. These could be either centralized, like today’s carbon offset registries or the government & NGO data aggregations on energy, carbon, and resource use; or decentralized, e.g. using blockchain technology to create a distributed repository. Regardless, it’s critical that these repositories be interconnected and uniformly discoverable and queryable.
• Verifier ecosystems — trusted organizations that audit data in the commons and certify it with their seal of authority. Again there are precursors to this in carbon offset registries, but I’m not aware of anything more global.
• Aggregate query, analysis, and reporting APIs — to provide unified technical access to the data in the commons and abstract away the details of how it is stored in various registries and repositories.
• Data applications — built on top of the APIs, fulfilling the cross-system use cases described above by translating data to actionable insight and providing the surface where system-shaping accountability, transparency, policy-creation, storytelling, and automation can all be realized.

While creating all this seems daunting, there are starting points in the system-wide datasets being manually aggregated by NGOs and agencies, groups like WRI, IPCC, WMO, and datacommons.org; and in some initial data applications like dataset explorers, climate simulators, and grid management; though there is a lot to do to join and integrate these datasets together into a meaningful commons. A particularly interesting effort is the Hyperledger Foundation’s Carbon Accounting workgroup, which is exploring blockchain as a way to enable distributed carbon accounting and certification without heavyweight central registries.

Another set of starting points might be industry- or vertical-specific versions of the data commons. For example it’s exciting to see data-commons-like precursors forming in the rapidly-growing space of carbon offset marketplaces, e.g. in registries and verifiers like the American Carbon Registry and Verra among others. Despite having issues and limitations, carbon offset marketplaces are like a “slice” of a future climate-system-wide data platform, and could represent a seed that could grow into the full data commons vision in the future. There may be similar opportunities in data platforms for energy, specific industries, agriculture, and so on. Whether a data commons emerges from non-commercial or commercial activity, it will ultimately need both as active, integrated participants, so it’s good to see early activity in both sectors.

The importance of data standards

Data standards are the essential foundation of a data platform, public or private. Data standards provide well-defined, clear, and widely-accepted ways of instrumenting, measuring, and representing system activity, thereby creating a data lingua franca that ensures interoperability and sets a common bar for data quality.

Standards most typically take the form of schemas — formats for data representation along with expectations of data behavior and quality — and metrics — measures that represent key quantities accompanied by assumptions for interpretation. For example, data on annual CO2 emissions might have a schema specifying 12 monthly quantities, each using the metric of tons of CO2 equivalent emissions per month, with definition of how CO2-equivalency is calculated for all common emissions scenarios.

Creating data standards is a hard problem. In striving for clarity, standards force debate and resolution on hard questions, like carbon accounting methodology (e.g. how uncertainty is represented; how modeled and measured data are combined; how timescales, like for sequestration, come into play). They require resolving competing interests, especially when data is used for consequential decision-making. And standards are never perfect, so they require both a willingness to compromise and comfort with evolving the standards over time.

Even a quantity as seemingly simple as “sequestered carbon” faces these challenges; consider how to represent this quantity for an imprecise, potentially-temporary intervention like applying minerals to a farm-field for advanced weathering, or the politics involved in defining an offset based on protection of an existing forest area that’s not under imminent threat.

But solving these hard problems is essential for a viable data platform, public or private (the private case tends to be easier since there are fewer actors to align). Without viable data standards, data sources can’t integrate into platforms, and private platforms can’t exchange data with each other (e.g., to support scope 3 emissions calculations) or integrate into a data commons. Conversely, when data standards are established the barriers to measurement and data-based accountability are lowered, since standards implicitly offer roadmaps for instrumentation, delineate best practices for dealing with uncertainty, and enable ecosystems of common instrumentation technology to develop.

So in many ways the grand challenge of activating climate data as a system force starts with establishing good data standards. These need to ultimately cover all the key resource transactions in the climate system, i.e. where:

• resources are extracted and, using energy, are transformed into products (material, industrial, agricultural, services, etc) which are then used (or wasted or recycled, in whole or part);
• energy is produced, consumed, and (sometimes) stored; and
• carbon is emitted or sequestered (at each stage of both the production and energy transactions above)

While there’s a tremendous amount of work to do here, the good news is that a lot of these transactions are already being measured ad-hoc in private climate data platforms, as part of MRV (measurement, reporting, verification) efforts, and/or have standards at the aggregate level used in government and industry reporting. The platform challenge is in bringing those together, standardizing technical representations, and agreeing on schema and representations of uncertainty and data quality — still far from easy, and unclear who will ultimately emerge as the champion of this work, but at least conceptually feasible…

Achieving a climate data platform

As should be obvious at this point, creating a climate data platform is an enormous and complex activity — a systems problem in its own right, especially as one digs into the interplay of incentives and obstacles necessary to align on public data standards or contribute data to a public commons. But the benefit could be enormous, unlocking the ability to reshape the climate system through a web of transparency, accountability, policy, and storytelling that would touch all parts of the climate system.

The path forward likely builds on the starting points we already have: private data platforms that are already getting traction by solving real business needs; registries and a nascent data commons for carbon offsets being driven by regulatory and commercial pressures; plus data standards, repositories, and metrics driven by non-commercial, NGO and government organizations. And increasing pressure for accountability from the regulatory side — for example in California’s proposed corporate emissions disclosure regulation (SB 260) and in incentives (like in ICA) and penalties — may be the magic accelerant that, together with technological advancement, can catalyze these starting points to come together into the full vision of a climate data platform.

What do you think about the promise of a climate data platform? I’d love to hear your thoughts and discuss further — comment here or find me at abbrown at gmail.com.