The Era of Big Data for Official Statistics

Pulse Lab Jakarta recently had the privilege of representing the Pulse Lab network in showcasing practical applications of big data to national statistical offices from around the globe. Don’t miss out on the presentation slides at the bottom of the blog.

In preparation for the UN World Data Forum in January 2017, a global preparatory seminar hosted by the National Bureau of Statistics (NBS) of China and the Statistics Division of the United Nations Department of Economic and Social Affairs was recently held in Guilin, China, to provide countries with the possibility to discuss data innovation for official statistics.

With the objective of achieving the Sustainable Development Goals in 2030, the seminar was an excellent way to take stock of the opportunities and challenges that exist in the production of the myriad of data that will be required for monitoring and measuring progress towards the SDGs. It was also an opportunity for countries to showcase the work they have been doing in using new data sources to enhance national statistics.

What was encouraging for me was the change in the discourse that has happened in recent years. The adoption of the SDGs has really propelled the advent of new data sources into the minds of officials within national statistical offices.

Wu Hongbo, United Nations Under Secretary-General for Economic and Social Affairs, spoke in his keynote address about how the SDGs will guide the development agenda and how there will be emphasis on more accurate, timely, reliable, and disaggregated data. It is now imperative to strengthen the capacities of national statistical bodies to ensure that enough quality data is being collected and generated to monitor progress towards the SDGs.

Inspiration from national statistical offices

Zheng Jingping, from the National Bureau of Statistics of China, gave a compelling speech where he highlighted how statistics is undergoing a digital revolution and how his country is adapting to this new paradigm. Making sure no one is left behind — a key principle behind the adoption of the SDGs — will mean that a combination of new technologies, innovative methods and traditional data sources is the vision for official statistics.

In his keynote, Zheng Jingping went on to highlight that there are two types of challenges with big data. The impact of big data and the huge data gap from an unbalanced supply and demand for official statistics. This is the era of big data and it is no longer sufficient to rely on traditional data sources. More interesting was the realisation that no longer do national statistical bureaux hold a monopoly over data collection.

The challenges with the use of big data for official statistics are plentiful and the unknowns are still preventing large scale adoption. What needs to be defined better, however, is the objective of using big data. Also the coordination between the producers, suppliers and users of big data should be strengthened with strict privacy frameworks in place to protect personally identifiable information.

NBS China has developed a very interesting and innovative partnership with Alibaba and Tencent. The aim is to create a big data platform for sharing data in a confidential manner. The statistics community is cooperating with different partners to facilitate the data collection from non-governmental organisations and combining with traditional data to ensure its validity.

Collaboration is key

I was really pleased to listen and contribute to the evolving discussion with national statistical offices and debate how we all might meet these challenges moving forward. I was also very pleased to be able to learn about concrete big data projects in such countries as Italy, Denmark, Tanzania and South Africa (amongst others) where innovative methodologies are being adopted to ensure greater efficiency, resource management and accuracy.

I was also happy to be able to contribute to the discussion on the ethics of using new forms of data collection and the general consensus at the seminar was that there is a role for the UN to play in guiding this debate and ensuring ethical principles are addressed in the global use, access and equity challenges that technology, finance, data and statistics may bring about.

My presentation slides are embedded below, which covers projects drawn from the Pulse Lab network and how they might be used for the SDGs. I’m a big fan of collaboration and I very much enjoyed the discussion with the participants after the presentation on ways in which NSOs can develop new partnerships in their respective countries.

It is now hoped that as the discussion moves to Capetown, in January 2017, that the UN World Data Forum should come out very strongly in favour of this new scope of work of NSOs in this new age of multiple data sources and the 2030 Agenda, the use of big data and the ethics and accountability of data.

Pulse Lab Jakarta is grateful for the generous support of the Department of Foreign Affairs and Trade of the Government of Australia.