Open Government Data - key ingredient instead of side dish
Open data has the potential to be one of the keys to successful government digitalisation. Yet still too often, government agencies treat open data as a side dish. Separate from their principal contents and services. A paradigm shift is overdue.
Disclaimer : this article builds upon the work of all the people working at the Statistical Office of the Canton of Zurich and its partners in the field of Open Government Data. It is based upon a presentation for a guest lecture given at the University of Bern on Open Data Management in January 2022.
At the core of digital services and products of government agencies there lies data. However — still — many services are built in a monolithic way — with data processes that aim just at serving isolated applications. This is inefficient and comes with several downsides. Whenever no sensitive data is involved, there’s a much better way to go.
To unleash the true potential of open data, it has to be treated as the backbone of government apps, digital services and products or at least be built in from the very beginning. Whenever non-sensitive data is involved, applications and services should be built on top of open data themselves or alternatively provide options to easily extract the underlying data.
But what are the advantages of “open data first” for digital government services? Let’s have a look at some real life examples which are based on open data ‘in the wild’.
Example 1: Visualizations & Infographics
The Statistical Office of the Canton of Zurich provides the population, companies, politicians, public administration and organisations with statistics, data and analyses on the Canton of Zurich. A tool we use regularly to display important developments in society and the economy are interactive data visualizations. We also produce data visualizations for other units of the cantonal administration as a service.
Developing data visualizations requires the preparation of structured machine-readable data. In the past, data was often being prepared and shared within the public administration for the single use purpose of the data visualization.
Some years ago, we asked ourselves, what if instead of just serving the visualizations, we would turn the shared data underneath our visualizations into interesting data sets that could be be published as open data and therefore easily be reused?
What started as an experiment quickly turned into a good practice, as the benefits of the approach quickly became evident.
A paradigm shift as a driver for open data: open instead of shared data
In the past, we often had a hard time in making other government units grasp why it would be worth it to publish their data as open government data. There were concerns that it would only lead to extra work and thus constitute a burden with unclear benefits. Only when we started to treat open data in an integrated manner and to show how it can be used as a basis for data driven information products on the web, things changed dramatically.
In recent years we have collaborated with several departments of the Canton of Zurich to modernize existing information products. The initial situation has been often similar. Usually, there was the aspiration to find a new form for PDF-reports with lots of static graphs and tables or outdated web applications, which didn’t provide options to extract the underlying information in an easy way. Usually the only possibility of extraction was to set up complex, time consuming and error prone web scraping or document parsing processes.
As a first step in those projects, we defined an open data “package”, consisting of multiple datasets, which could be used as a basis for visualizations and applications on the web. Often, the goal has been to create a comprehensive thematic website on a given topic, which portraits the relevant numbers and developments. The visualizations and the text aim at providing the general public with the most important facts at a glance. The underlying raw data are made available for easy reusage as open data for the public — be it data journalists, researchers, interested citizens or other government units.
Thus, we actively enhance reusage by third parties (1). Information published as open data becomes fluid and can serve multiple user groups and ultimately reaches a wider audience (2).
One of the early examples in which we implemented the open data first approach in collaboration with another unit of the Canton of Zurich is the webpage on:
Juvenile Justice: Zahlen & Fakten zum Jugendstrafrecht
Open data — a game changer in the case of high frequency data
The “open data as main ingredient” philosophy has helped us become more efficient in our data publication processes. Once new data points became available, we could focus on updating the open data — and only that. Any visualization or application based on them would automatically be refreshed. Even for datasets which get an update once a year this has proved to be convenient.
But the approach was even more powerful in the case of high frequency data. It’s not a coincidence that in one of the most severe crises of our times open data has been at the basis of situation dashboards worldwide (such as the Italian Corona Dashboard, which was one of our models). The same applies to our webpage covering the situation related to Covid-19 in the Canton of Zurich, which is fully fed by open data published on GitHub by the Data Analysis Division of the Office of Health of the Canton of Zurich.
During the critical phases of the pandemic, the data were updated on a daily basis, which led to an automatic refresh of the overview with countless visualizations. This proved to be very resource-saving and efficient at the same time. And as the data is freely available, everyone can develop applications based on it which retrieve the data automatically.
As we consumed our own open data, we quickly recognized where we could make improvements or where corrections were necessary. But as the data is being consumed by the many, a “Feedback Loop” comes into effect, as more eyes contribute to detecting errors and report them instantly. This happened regularly in the case of Covid-19 related data. The resulting corrections at the source had a direct effect on all the external applications which were build upon it, improving the quality of the shared information (3). So it helped us at safeguarding that — given the current knowledge — the most accurate official figures were in circulation.
Over time, as the number of use cases grew, the benefits of open data became evident to more and more data publishers. What started as an ad hoc solution in a single case became a systematic approach that is now implemented on most of the thematic data driven webpages of the cantonal website.
As this happens, we are moving away from a situation where isolated government units publish data in very different ways to one where all data publishers work together and form a community, collaborating on standards and sharing good practices.
A not complete (and not updated) list of thematic webpages of the Canton of Zurich which are either built upon open data sets or provide an easy way to export the underlying data:
Air Temperature and Humidity: Lokalklima-Monitoring
Bathing Water Quality: Badewasserqualität in Flüssen und Seen
Covid-19 Pandemic: Zahlen & Fakten zu COVID-19
Crime: Kriminalität in Zahlen
Cultural Funding: Corona-Hilfen im Kulturbereich
Education: Zahlen & Fakten in der Bildung
Elections and Voting: Wahlen & Abstimmungen
Empty Apartment Census: Leerwohnungsziffern auf Gemeideebene
Gravel and Excavation: Kiesstatistik
Juvenile Justice: Zahlen & Fakten zum Jugendstrafrecht
Municipal Tax Rates: Gemeindesteuerfüsse
Open Government Data: Offene Daten finden
Population: Bestand & Struktur der Bevölkerung
Taxes: Steuerstatistiken
Traffic Accidents: Verkehrsunfallstatistik
Example 2: Real Time Vote Results Webservice
In Switzerland we have the privilege to decide on political matters in a direct democratic process. Up to four times a year the electorate can take part in votes on national, cantonal, regional and even municipal matters. On Voting Sundays, a large part of citizens keeps themselves informed in real time on the counting process and the first results. Especially if a vote result is expected to get close and no clear yes or no trend manifests, the attention by the media and the electorate is huge.
What possibilities do citizens have to follow the counting in real time? Many.
One option is to download the mobile App “VoteInfo” (for Android & iOS). The app, which was launched in 2019, allows citizens to check information on the upcoming votes as well as the real time results on Voting Sundays. What makes the app special? Besides its broad functional features, the fact that its data foundation is made available as open data and thus can be reused.
In the case of the Canton of Zurich, the very same data is used to nurture the results overview on the cantonal website (zh.ch/abstimmungen). The application provides a web service for municipalities (in JSON), allowing them to integrate the results of municipal votes on their websites. Additionally, media outlets use the very same data for their data driven result overviews.
So the beautiful thing about this is, that it really doesn’t matter over which channel you follow the counting and check the results. The fact that there’s a single source for all of them, enlarges the reach of the very same data, makes the counting process transparent, and thus potentially increases the trust in the democratic process.
Besides that, if in future some of our applications will need to be redesigned or migrated to newer technologies, we will not have to bother that much about the data processes (4).
Let’s wrap up
In a nutshell, what are the benefits of putting open data first?
- Enhance Reusage in other Products: An open data foundation can be easily reused. Even by government agencies themselves. The information stored in the data becomes fluid and can take different forms.
- Higher Reach of Official Information: As the data can be reused in different products (1) and be re-published on multiple digital channels, its reach is amplified. The information can reach its target groups and customers over their desired channels. This is crucial in the era of the ‘attention economy’, where official government information is increasingly competing with ‘alternative’ information.
- Higher Quality: As government reuses its ‘own’ data and thus becomes a ‘user’ by itself, it quickly recognizes what could be improved. Additionally, a “Feedback Loop” comes into effect: As the data can be accessed and reused by everyone, and it actually gets used by the many, the possibility increases that errors are detected and reported, leading to a higher quality of the dataset for everyone.
- Easier Redesign of the Services and Migration to Newer Technologies: redesigns and migrations to newer technologies become much easier, as the basis — the data — is not locked in an app or a service. The opportunity cost to ‘redress’ is much lower than having to rebuild the entire thing. Usually ‘data’ lives longer than the apps and services it nurtures, and is migrated (see e.g. André Golliez’ slides “Swiss Data Space”, p. 13).
Last but not least, practically speaking, how can you and your organisation embark the ‘Open Data as key ingredient’-path?
- When building small applications — such as data visualizations or dynamic lists — consider building them on top of open data. Think of the information and data structure you need and prepare an open data set accordingly.
- When planing more complex applications or services — define an API which is well documented (see e.g. Verkehrszähldaten motorisierter Individualverkehr MIV im Kanton Zürich, using Swagger) and can be used independently from the GUI of the application to retrieve the underlying data. In short: Decouple the GUI from the data to increase your flexibility in the long term.
- In the case of older legacy applications, which will not be renewed in the short term, provide a bulk download for the underlying data, or at least the relevant part of it, in an open format.
This applies not only to applications nurtured by data of statistical character, but to all the cases where structured information is involved. Be it in the form of text with legal or procedural character, addresses (locators of agencies/governments) and so on for which there is a demand. In any case it makes sense to do some research on standards that exist in the domain, which the use case belongs to.
The road ahead for Open Government Data
By the examples and explanations above, I hope that my point is clear. Government agencies should be concerned about the ease of reuse and seamless flow of structured information. As citizens, we might be limited in the choice of where to submit our tax records. But much less in the choice of the channels to keep ourselves informed —at least in democratic societies.
In some areas, official information is increasingly competing with third party information — some times even of ‘alternative’ character. Thus, as we live in the era of the ‘attention economy’, it is essential that high quality official information can reach citizens over their preferred channels by flowing to wherever it is needed. By putting data first, path dependency in application- and service lifecycles can be reduced. Recycling the data to update technologically outdated information products becomes much easier.
Encouragingly, in the Swiss context, we’re heading into the right direction. In some domains, such as the field of geoinformation, open data has been an important component of the available services for quite considerable time. Still, in many other domains, that’s not the case yet.
But times are changing. In many Cantons open data is finally receiving the attention it deserves. Open Data strategies are being crafted at all federal levels, which in some cases even postulate the ‘open by default’ mechanism.
At the legal level, the foundations are being laid with the new Bundesgesetz über den Einsatz elektronischer Mittel zur Erfüllung von Behördenaufgaben (EMBaG). It will include the principle ‘open by default’ when it comes to data.
And even at the political level, technical topics such as APIs, have finally received the attention they deserve. This has lately culminated into the new federal API strategy, which points into just the right direction, as it aims at promoting digital access to government services in the machine-to-machine context for businesses, the administration and private individuals.
