How to Measure Digital Governance through Open Data

What if we told you there was an easier way?

Open Government Partnership
OGP Horizons
13 min readMay 7, 2024

--

By Amy Johnson and Joseph Foti

This article proposes a means of measuring the strength of digital governance through open data. It explores recent interventions that produce centralized repositories of salient information — and highlights the potential usefulness of the administrative data of these repositories and their linked interoperable documents. Because the concept of “digital governance” is emergent and encompasses many competing values, this approach would be helpful in at least two ways. First, it is applicable across a range of governance approaches (top-down oversight, self-regulation, and hybrid approaches), across different policy domains and values (e.g., competition, privacy, human rights), and across contexts with differing state capacity. Second, it is cheap, scaleable, explainable, and fairly accurate and builds upon successful deployment of similar measures in right to information, lobbying, and other governance matters.

Digital governance refers to how power is distributed and used to govern digital technologies and online spaces. It can touch on a variety of topics from data privacy to cybersecurity, democratic freedoms to accessibility, ethics to competition, and innovation to information integrity. Digital governance may focus on the online sphere, but it affects the analog world we still occupy.

To know whether we are governing these platforms, technologies, and infrastructure in a way that balances these sometimes competing demands, we benefit from some amount of measurement.

Yet, some measures are better suited for particular uses than others. We need information on policies and actual implementation. In OGP, where learning and adapting policies occur across borders, comparability is helpful, especially where it enables meaningful learning. The most common questions asked to the OGP Support Unit are, “Who does this well?” and “How is my neighbor doing on this?” To answer that, we need good, legible data.

Of course, measuring actual implementation of policies that promote human rights protections in the digital age or prevent online censorship is notoriously difficult or rare. There are numerous excellent measures, but they tend not to produce comprehensive, readable, and explainable information. A few examples of these measures:

  • “Ambient” measures: Freedom House’s Freedom on the Net focuses on democratic freedoms. These tools measure governance at the national level, identifying general practices and consequences, such as events and actions of particular concern. This is good for understanding the general situation country-by-country, but more difficult in defining the positive actions governments should take. Similar measures that fall into this category are the Worldwide Governance Indicators, Varieties of Democracy, or the World Justice Project, to name a few.
  • De jure measures: These measures look at the quality of law with a secondary eye to implementation. Two examples of where this is done well are the Centre for Law and Democracy’s long-used RTI rating and the Environmental Democracy Index.
  • Implementation measures: There are numerous tools that dive deep into practice. A notable example is the Carter Center’s Right to Information (RTI) Implementation Assessment Tool. However, others tend to rely on case studies or action learning, which make assessment prone to issues of reproducibility, representativeness, and intercoder reliability. The richness of these approaches is also their limitation — they are often expensive. Reaching scale and representativeness, then, pose a particular challenge when assessing implementation.

To be unequivocal, each of these approaches contributes a ton to our understanding of good governance and has helped fuel reform worldwide. But is there a cost-effective approach that combines good elements from each? We would argue that there is.

All models are bad, but….

Our basic proposition is that looking at administrative data and the underlying systems can be an effective proxy for actual implementation — a sweet spot between law and ambient measures that operates at scale. Like all proxies, it follows Box’s insight: “All models are wrong, but some are useful.”

This is the approach OGP took in collaboration with Global Data Barometer and Transparency International when the first Global Data Barometer (GDB) looked into right to information and lobbying, among other topics. We are highlighting these in particular because, to our knowledge, the GDB was the first ever global survey of RTI implementation and lobbying laws.

This was a huge step forward in understanding some basic good governance structures around the world. By looking at whether administrative data was publicly available, we were able to quickly answer questions that were difficult to know otherwise:

  • Whether anyone was tracking administration: Simply knowing whether data exists gives a good insight into whether anyone is in charge of oversight and compliance and whether they have the people, budgets, and capacity to follow their mandate.
  • Whether tracking was systematic: Knowing whether an agency is publishing structured, downloadable, machine-readable data and reports can help understand whether there are standard approaches across branches of government or agencies, rather than ad hoc reporting.
  • Whether comparison was possible: The existence of comparable data does not mean that the law is followed, especially for RTI. But it does mean that we can assess ways that learning and adaptation could take place.

This is a very good proxy for whether something is being managed. Of course, there are many cases where the data is bad, people cannot use it, or they misuse that data. But it revealed fascinating patterns and a number of countries with very good practices.

Figure: Screenshot of the OGP Broken Links Data Explorer

The data governance conundrum

As defined in the beginning of this piece, digital governance is a big bundle of policies and practices, with some regulatory in approach and others falling outside of regulation (such as partnerships and industry standards). We are in a moment of foment, before “best practices,” common arrangements, or international law have set in. Instead, there are:

This is exactly why measuring “best practices” for formal institutions or ambient measures will be inadequate. Rather than ask “Do you have agency X? Do you have law Y?” it may actually be better to understand “Can we address this particular problem effectively?”

Looking at whether there are any approaches to address an issue is helpful in this regard. When we looked at RTI administration through the GDB, we did not (and could not) know whether the number of requests for information that were granted were the right number. However, we knew that no agency should deny all requests. We also knew that not all requests should be granted. (Some requests will trigger exemptions, for example, around privacy or national security.) In this case, we could not say that there was a correct performance metric. But we can say that they should be monitoring compliance with the law. That allows people and institutions to know how the law is performing, for individual requestors, each agency, and the country overall.

The takeaway? At this moment, open data can really shine as a proxy. Of course, it will not solve problems by itself. But it allows the public, oversight agencies, and the private sector to roll up their sleeves and understand where problems lie and to begin crafting solutions.

The essentials of measurement

Administrative data can be used as a proxy to understand whether the digital space is being governed, by whom, and how it is being governed. Measuring this consists of two parts: understanding who is in charge of certain functions (see below for a partial list) and whether they share that information. As with the example from RTI above, this allows us to understand if there is any administration at all, and if tracking and learning occurs across agencies, sectors, and jurisdictions. Knowing if there is public sharing of administrative data can help us understand the degree to which oversight is democratic as well.

Responsibilities, responsibilities…

To know whether something is managed, we need to know if anyone is in charge and what they are in charge of. This includes data collection and publication.

For digital governance, we need to know that there are a few competent functions in place. In defining some key functions, care has been taken to understand that each country might face a different mix of institutions (public, private, and non-profit sectors) that deal with these essential issues.

Ensuring fair and affordable access to the internet. This function is often split between public service providers, private sector providers, and regulators. Regulators may include dedicated digital agencies or this may be a devolved power that regional or local governments deal with.

  • Why it’s important: Making sure that internet access is universal and that the service is provided in a non-discriminatory fashion is foundational to making sure that people can express themselves, assemble, and inform decision-making.

Preventing elite control of digital infrastructure. In some cases, private monopolies of digital infrastructure can create room for abuse for private or political gain. But in other cases, such as rural delivery, there are natural monopolies that need democratic oversight like other utilities. Competent authorities, such as anti-trust regulators or competition authorities, need to ensure competition or democratic oversight where the market or other forms of organization cannot.

  • Why it’s important: Elite control can limit the strength of the public square, especially where regulators do not prioritize human rights, information integrity, or free speech. Unregulated monopolies lessen public welfare and often wield undue political influence.

Ensuring data protection and privacy. This includes the responsible use of sensitive data, including collection, re-use, and sharing. Increasingly, this function is being carried out by “Data Protection Authorities,” who have a responsibility to ensure privacy online. These authorities may be housed in combination with RTI authorities (as in South Africa or Mexico) or AI regulators.

  • Why it’s important: Privacy is an important right in and of itself. It also limits abuse of authority, helps maintain social boundaries (between work and family, for example), prevents discrimination, and promotes property rights and safety.

Ensuring safeguards are in place. Increasingly, there is a general acceptance that some digital activities are high-risk while others are anodyne. Sorting them is one job. A second one is ensuring that protections are in place that will mitigate risks. Different jurisdictions are taking different measures to address this. The EU Digital Services Act has assigned this to Digital Services Coordinators, to give one example. At one extreme, there are outright moratoria (facial recognition processing being the most salient), while at others, there are encouragements of “industry best practices.”

  • Why it’s important: Encouraging innovation while minimizing irreversible human rights abuses or environmental damage is a tricky balance and one that’s fraught with problems. Yet, it is essential nonetheless to try to strike that balance and avoid the worst excesses seen, for example, in social media platforms.

Right to information. The other side of privacy is RTI. Citizens have a right to know what public officials and entities are doing.

  • Why it’s important: RTI contributes to reason-giving and systematic record-keeping. In turn, this reduces abuse of power or arbitrary decision-making by letting decision makers know that the public may scrutinize their actions and that they may be held accountable for decisions.

There are a number of other functions that can also be measured — consumer safety and children’s rights come to mind — and we are sure readers would have more to add. Of course, as mentioned above, nothing is perfect. If we are missing something, point it out in the comments, please. We do read them.

What did you know and when did you know it?

This is where we really get to the essentials of data around digital governance. There are probably four questions that we would want to know about data on how well a particular technology or associated practices is being governed:

  • Risk assessment: Is there a robust, public classification of risk for certain online activities?
  • Disclosure thresholds: Does the system require public disclosure of those practices that meet certain risk thresholds?
  • Structured information: Are such disclosures, such as impact assessments or case law, available in structured data, or at least indexed using regular identifiers that allow them to be referenced, re-used, or combined with other data?
  • Scope: Does it apply to public sector applications? Does it apply to private sector applications?

The European Union’s AI Act presents a typology of risk and a corresponding set of policy responses:

  1. Unacceptable: Applications used by public authorities that comprise subliminal techniques, exploitative systems, or social scoring systems are strictly prohibited. Also prohibited are any real-time remote biometric identification systems used by law enforcement in publicly accessible spaces.
  2. High Risk: These include applications related to transport, education, employment, and welfare, among others. Before putting a high-risk AI system on the market or in service in the EU, companies must conduct a prior “conformity assessment” and meet a long list of requirements to ensure the system is safe. As a pragmatic measure, the regulation also calls for the European Commission to create and sustain a publicly accessible database where providers will be obligated to provide information about their high-risk AI systems, ensuring transparency for all stakeholders.
  3. Limited Risk: These refer to AI systems that meet specific transparency obligations. For instance, an individual interacting with a chatbot must be informed that they are engaging with a machine so they can decide whether to proceed or to request to speak with a human instead.
  4. Minimal Risk: These applications are already widely deployed and make up most of the AI systems we interact with today. Examples include spam filters, AI-enabled video games, and inventory-management systems.

Presumably, even in self-regulatory or other settings, risk classification and management may also be standardized or reported in a semi-standard or structured manner. Importantly, however, each of these levels of risk usually may require increasing levels of disclosure or other actions. Ideally, where regulatory rules or standard-setting processes are in place, there are clear and transparent requirements for disclosure of activities at each level of risk threshold.

Competing models of disclosure

Because there simply is not consensus on the correct approach to disclosure or transparency, any measure must take into account the diversity of approaches currently undertaken. In this case, one can simply carry out a survey of what practices are in place without bias to which model is the best.

They fall into two categories: registers of basic information about how algorithms process data and specific disclosures about that processing.

Transparency registers

As part of their data processing rules, some countries require disclosure of processing (collection, use, storage, and transfer) of basic information about algorithms in registers. Algorithmic transparency registers provide a structured means of disclosing the existence of a system of data processing.

Specific disclosures

Again, there are competing models and no leading candidate for a consensus model. Even within a particular country or province, multiple, overlapping, or complementary models may exist. For example:

  • Human rights impact assessment. A human rights impact assessment (HRIA) is a process for systematically identifying, predicting and responding to the potential human rights impacts of a business operation, capital project, government policy, or trade agreement. Some governments have adopted the publication of this process, similar to an environmental impact assessment process or social impact assessment. While some countries have adopted such practices, HRIA remains typically relevant for governments and multilateral institutions. See this report by the World Bank and Nordic Trust Fund for a comparison of approaches. Guidance for companies can be found in this tool by The Danish Institute for Human Rights, as one example.
  • Privacy impact assessment. Privacy impact statements or privacy impact assessments may be used when personal data is being collected, most often as the result of a planned government action. They require the promulgator of the technology to identify what personal data will be collected, what the purpose of such data would be, and how that data will be collected, used, accessed, shared, safeguarded, and stored.
  • Model cards or data cards. Model cards are brief, consistently structured descriptions of an algorithm that explain the basic functions of an automated decision-making tool. These cards usually contain the following information, seeking to help decision makers understand a particular tool and its application:
  • Model description
  • Intended use
  • Features used in the model
  • Metrics
  • Data used for training and evaluation
  • Limitations
  • Ethical considerations

Similarly, data cards may include information on data being used and can include the following elements:

  • Dataset overview and example of data points
  • Data source collection, validation, transformations, and sampling methods
  • Sensitive attributes
  • Training and evaluation methods
  • Intended and extended use of data
  • Algorithmic impact assessment. This assessment is a similar information-gathering and decision-making tool to other forms of impact assessment. It aims to understand the potential consequences of the deployment of a technology, as well as alternatives and measures to mitigate and limit any negative consequences. In addition, it may include a process for discussion with the public.
  • Publication of relevant cases. Relevant cases (to regulators or courts) may be published and linked to particular algorithms. This is most important for Data Protection Authorities and AI regulators who may hear precedent-setting or instructive cases that help public and private sector officers interpret the law. This information may be listed on the website of a DPA (or its equivalent), although ideally, it will also have common indicators to interlink with other algorithm registers. The European Court of Human Rights posts relevant case law on data protection publicly, although it does not yet offer it as structured data.

A way forward

Measurement of governance in a way that is meaningful for policy makers and reformers is notoriously difficult. That being said, merely measuring the existence of regulatory measures, disclosure practices, and their implementation — with a minimum of bias toward what should be — can be a step in the right direction.

Once more data is available, this opens up the possibility for structured reflection. What is working? (And what does “working” even mean given the many different policy objectives of digital governance?) Civil society and journalists can use the data to identify where things are working or not. Oversight agencies can ask for additional accounting and explanation from platforms or agencies. And platforms or other private sector actors can identify where they should dedicate resources and prioritize compliance with existing standards and regulation.

As part of its 2023–2028 Strategy and its newly launched Open Gov Challenge, OGP is supporting its members to take more ambitious action in digital governance. The OGP Support Unit will continue its collaboration with various partners, especially with Data for Development and the Global Data Barometer, to measure whether progress is being made. Though we do not know right now what the perfect model of regulation and oversight are, we know something must be done. This means of measurement is an attempt in that direction.

Let us know in the comments if we missed anything, or if you want to give a massive (or even medium-sized) grant to help make this measurement a reality.

--

--

Open Government Partnership
OGP Horizons

75 national & 104 local governments, plus thousands of civil society groups, working to deliver the promise of democracy beyond the ballot box through #OpenGov.