An Updated Methodology for the Open Science Monitor

After an intense online debate and an excellent expert workshop, we — the Lisbon Council, ESADE Business School and the Centre for Science and Technology Studies (CWTS) at Leiden University — are happy to publish a revised methodology (soon to be posted on the official European Commission website). First and foremost, this is not the final methodology. It has become clear that open science is too dynamic and too difficult to measure for a “definitive” methodology. It is rather a second interim release — and it will require continuous collaboration with the open science community in the future, also in connection with other existing efforts by OpenAire, FAIR Metrics and the European Open Science Cloud.

Of course, the main objection raised in comment period is the use of proprietary tools, such as the Scopus (or Web of Science, for that matter) database. We remain convinced that, given the current level and status of data availability, using proprietary data is a necessity in order to provide timely, relevant indicators. Hopefully, once the European Open Science Cloud is fully deployed, indicators will be automatically produced from open data sources — but it is not the case today.

More precisely, as CWTS clearly presented in the workshop (see slide below), Scopus has the well-curated metadata that we need, which are not available through open sources such as e.g. CrossRef — still an important source used by the consortium to add information to the metadata.

The need for additional metadata — slide presented at the workshop

The approach remains to use and integrate the best data sources we can find, and to critically assess them purely on quality criteria. Based on the insight gained through the comments and the discussion in the workshop, we introduced important novelties in the methodology:

  • We will use Unpaywall data alongside Scopus data. Unpaywall has a very large footprint and will increase the coverage of the Open Science Monitor. And we will continue to look at and collaborate with Unpaywall in its new initiatives.
  • We use multiple sources when possible for double-checking purposes. We will do a comparison of results obtained when using Scopus or Web of Science, and we’re exploring the possibility to do the same with Plum and
  • We clarify that data from Scopus can be made available to individual academic researchers to assess or replicate the OSM methodology, under the standing policy of Elsevier to permit academic research access to Scopus data.

We have also created new, structured ways to collaborate with the community. Firstly, the project now has an excellent Advisory Board (composed by workshop participants and chaired by Professor Žiga Turk) to help us identifying indicators going forward.

Secondly, we will make collaboration with the community not a one-off, but permanent. The commentable document worked well; we received more than 300 comments. Most of these comments are initial ideas that need clarification and refinement. We need more permanent and interactive ways to discuss. Hence, we have opened an Open Science Monitor Linkedin Group to work on new indicators — anyone is able to join.

We thank everyone for the interest and the constructive criticisms. There is a lot of work in front of us. We look forward to joining together with you to monitor and advance open science in Europe.