Preliminary system architecture analysis of California’s water data systems
Tony Castalletto has worked over a decade in library science and IT management leadership roles. He is helping our big California water data project as a “System Architecture” research fellow looking at the institutional and information technology landscape governing California current California water data.
As part of that he is part of the team including Seed Consulting Group working pro-bono in partnership with the California Data Collaborative to conduct an agile development sprint to ingest the data sources identified by the Dodd AB 1775 as the first step in a feasibility study to implement the proposed data integration. See below for how this project integrates with the already completed version 1.0 of the Seed Consulting WaterLog Project:
He has also written the below analysis of California state level water data collection efforts at my direction. I do not agree with all of the analysis and reminded Tony after review that it is much easier to point out an institutional problem than to propose a solution, much less implement one.
Yet pointing out a problem and understanding its root causes is critical in developing and executing a plan to improve the situation, as GK Chesterton eloquently articulated in his parable of a fence. Please consider this a preliminary analysis of an experience IT professional working to get up to speed on water.
Any shortcomings in understanding the nuances of water management in my professional opinion illustrates the challenges of navigating our complex water institutional administrative and technological architecture. See below for the memo and as always your insights, questions and challenges are appreciated in the comments. Thanks much.
Cheers,
Patrick
Memorandum
To: Patrick Atwater, Project Manager of the California Data Collaborative
From: Anthony Castelletto, System Architecture Research Fellow, California Data Collaborative
Subject: Diagnostic Memo of California Department Water Resource’s Data Practices
Date: April 20, 2016
Introduction
The ongoing drought and the forecast of more frequent and persistent droughts throughout the 21st Century has forced the State of California to review its water management and allocation practices. While relatively accurate information on aggregate water use can be found, detailed, granular water usage data essential to planning and evaluation remains fragmented among a myriad of agencies, commissions, and corporations. If California is to remain a populous and economically vibrant state, it must transition to adaptive management to carefully husband the limited resource of water. Urban areas must continue to improve their efficiency. To do this, California governments, municipal water agencies and regional water boards will need a centralized data warehouse containing raw water-use data down to the household level. Currently no agency collects and disseminates detailed water usage data, hobbling efforts at collaborative water governance and frustrating the development of a statewide water transfer market. The state agency best able to collect and centralize such data would be the Department of Water Resources. However, this agency not done so. This leaves independent water management districts to act on their own. Some of these districts have taken action through collaboration, creating the California Data Collaborative to coordinate their work.
The California Data Collaborative is a project run by a coalition of water agencies operating under a Memorandum of Understanding with Moulton-Niguel Water Management District acting as the lead administrator. The Collaborative project seeks to develop data tools to enable smarter water conservation and promote greater efficiency in water use. The limits of utility wide average use statistics stymie the development of these advanced management tools. Adaptive and responsive management requires fine grained data which accurately captures the behavior of individual consumers. This data not only makes possible the examination of usage, but also allows economic effects such as price elasticity to be determined. In addition, since 2013, water utilities have been subject to more stringent reporting requirements. The California Data Collaborative works to support water managers in achieving their reliability objectives by developing a common data infrastructure.
Description and Scope of the Problem
Two developments in state policy drive the need for better data collection and coordination. First, the ongoing shortages of water due to drought and the increased demands on shrinking supplies caused by an expanding population force water agencies to coordinate their plans. In fact, such collaborations already exist. The largest one is the CALFED Bay-Delta Program which manages the Sacramento River and San Joaquin River Deltas. Under CALFED, state agencies collaborate with with the Federal government to regulated water resources in this watershed. Such collaborations in allocating water supplies and managing waste require usable and current data. Secondly, California’s current water transfer markets suffer many inefficiencies due to a lack of timely information about supply and demand. At present, no agency provides the data needed to achieve meaningful collaboration and efficient market transfers.
While the State of California collects a great deal of information on water use, supply, and rights, these data sets have never been collected into a single data warehouse nor are they made fully available to water managers and state agencies. Although legislation such as the Sustainable Groundwater Management Act have led to the collection of such data, a number of problems remain. The data available is disjointed and lacking in context, much of it consists of aggregated information which amount to little more than executive summaries or reports. Policy and academic researchers cannot generally use this sort of data for their work. This constrains efforts to evaluate and develop policy and severely impedes innovation from academia or the private sector. The data format itself presents problems. The Department of Water Resources and the State Water Resources Control Board deliver data as Adobe PDF documents. Although humans find such reports easy to read, extracting data for substantive analysis consumes time and resources. Ultimately, the effort to extract data is wasteful as it duplicates work already done by the boards and the department. The lack of open data protocols and common data structures severely hampers data sharing. This situation harms the public and impairs the functioning of many public agencies. Standardized and open data formats are now commonplace in many disciplines and industries. The question then is why the State of California has not adopted modern data practices in its water management system. Furthermore, the Department of Water Resources, seemingly, has the authority to undertake the construction of a modern water accounting system. Why has it failed to act?
Causes
California’s water governance is fragmented, conflicted and contentious. This fragmentation has only increased over the years. The Public Policy Institute describes the system as “highly decentralized, with many hundreds of local and regional agencies responsible for water supply, wastewater treatment, flood control, and related land use decisions.” Currently, the Federal Government, the Department of Water Resources, the State Water Resources Control Board and nine regional control boards administer California’s water resources. Each organization has distinct priorities and different constituencies. California’s lead water management agency, however, is the Department of Water Resources.
Created in 1956 by legislation, the California Department of Water Resources along with the State Water Rights Control Board (SWRCB) administer the state’s water resources. The Department of Water Resources (DWR) controls the water distribution infrastructure of the state while the Board, originally the State Water Rights Board, allocates supply and manages water rights. The Board also reviews planning decisions made by the director of the DWR. The DWR itself holds water rights of its own amounting to 31 million acre-feet. Thus, authority over water rights was separated out into the State Water Rights Board, now the SWRCB, to prevent conflicts of interest. At its formation, the DWR was granted sweeping powers and authority over the state’s water system and was the sole administrator of the State Water Project, the system of dams and canals that collects and distribute water to agricultural users and population centers. The State Water Rights Board initially focused solely on the management of water rights, while almost every other function fell under the DWR. This all changed in 1969 with the passage of the Porter-Cologne Act. This act, an early effort to deal with pollution made water quality a primary concern and served as a prototype for the Federal Clean Water Act. Under the legislation, water agencies and other users were forced to collect and report data on th quality of their water and regulate discharges. The State Water Rights Board was merged with the State Water Pollution Control Board to form the State Water Resources Control Board. The Porter-Cologne act also split up responsibility for water quality among nine regional boards reporting to the SWRCB. The SWRCB must also determine allocations for all water usersMeanwhile, the DWR controls almost all planning functions, setting the stage for conflict over water management. Complicating matters, the California EPA also has jurisdiction in many areas.
Both agencies find themselves constrained by the state’s complex and multi-tiered water rights laws. These rights range from the Pueblo Era to the new rights in the present one. Riparian Rights secured prior to 1914 remain unregulated to this day meaning those with land along rivers may divert as much water they like. This inconsistent and diverse portfolio of rights makes regulation and measurement of inflows difficult so the state lacks any clear notion of how water is tapped. The Port-Cologne Act constrains the board’s ability to track consumption by prioritizing its mission as enforcement of water quality and by situating it within the California Environmental Protection Agency. Meanwhile, the board lacks any ability initiate investigations of its own initiative; it can only respond to complaints made about appropriation of water in the context of rights violations.
The DWR, on the other hand has substantial powers of investigation. The DWR has the ability to collect data and has made some efforts to do so in recent years, though historically, this has not been a priority. The 2009 Water Conservation Act mandated increased tracking of use from holders of water rights. This, however leaves a large gap in the understanding of end point water use and appears to omit a great deal of information. The DWR’s Integrated Water Resource Information System began to track this data, but lacks deep and detailed information about urban usage. Meanwhile, the SWRCB’s Electronic Water Rights Management Information System only compiled self reported usage figures from rights holders, bringing its accuracy into question. Without any central coordination, California Water Management Districts rely on their own data for planning and coordinate activities on an ad hoc basis. While the evident fragmentation of governance along regional lines and function certainly impedes collection and coordination of usage data, the DWR has not used its statutory powers of investigation to collect data. Under its authorizing legislation (Section 153 and 158), the department has the ability to conduct any investigation into water issues that deems necessary. This power is reinforced by the California Water Code which results from the changes of subsequent legislation. Sections 225 through 238 of the code provide the department with ample authority to collect usage data.
The problem then results from the powers of discretion given to the directory of the DWR. While Director of the DWR may initiate data collection at any time, the department’s authorizing legislation provides neither statutory trigger for routine data collection nor powers of enforcement to compel the delivery of data. Thus any director initiating such a data collection effort takes a risk over the statutory scope of the department’s authority especially where it impinges on local water management districts. Without direction from the governor or a clear legislative directive, the agency risks considerable backlash in attempting to create a comprehensive water data management system. The development of a modern water accounting system may have to wait for new legislation from the state assembly. As of this writing, Assemblyman Bill Dodd has introduced a bill, AB 1755, which would direct the creation of a new data management agency under the DWR to accomplish this task.
