Research Repositories: A ResearchOps Community Program of Work

A knowledge management system can be a shelf, a spreadsheet, a folder, a physical or digital library. What they have in common, is our desire to document, and share knowledge, our stories.

Top level findings

This article has been written by Brigette Metzler, Bri Norton, Dana Chrisfield and Mark McElhaw on behalf of the entire Research Repositories Program Team.

This is the first of a series of articles to be published on the program of work that commenced in July 2019. This article will detail the top-level findings, and subsequent articles will outline the four separate types of solutions we discovered as we researched the state of user research knowledge management systems. Once this is complete, we will create a website to host everything as a single resource. Given the list of people who contributed in small and big ways (that became a program) is almost a 20 minute article in itself, we will save that for the website — but for expediency’s sake, here is a map of the people of the project. We look forward to publishing every single name individually soon. For now, if you are reading this, and you worked on this project, you have the hearts of the whole community with our debt of gratitude for your resilience, persistent in the middle of a pandemic, and sheer passion. Thank you ❤

Project background

Since the very beginning, Research Operations (ReOps) people have wanted to talk about research repositories, insights registers/hubs/repos or libraries. They’ve all expressed difficulties with what they’ve made, bought or used. That’s not a slight on any of the tools available on the market — the process is hard, getting buy in is hard. Knowledge management and library sciences are professions in and of themselves for good reason. Knowing exactly what’s required for each person’s context makes it very hard for research teams to get it right, at least at first. We’re pleased to have noticed, even over the course of the project, that more and more people are reporting that things are progressing.

Some things we noticed as we watched the chat in the Research Repositories channel:

• Getting researchers to want to add their research is hard. Getting people to access and use the research is much less difficult!
• There seems to be consistency in what researchers need, what they worry about, and what consumers (users, readers) of research seem to need.

Knowing this, the overarching research question for this project has been:

What is the nature of the problem space, and what are the patterns of reasoning and guiding principles among the people involved?

The problems were summarised as:

• Knowing the research exists, but being unable to find it
• Research Groundhog Day — having to cover the same ground regularly without the opportunity to learn from what is already known.
• Losing control of the research by emailing it or printing copies (finding others using the research out of context and without your knowledge)
• Wondering if the research done could tell a different story if looked at over a long period
• Needing to be able to measure or show the impact of research
• Wanting to be able to think of research as an asset, rather than something to be looked at once, in one situation only
• Struggling with the scale of research going on in an organisation and not being able to keep track of it all.

These problems helped guide our questions, created clarity about the return on investment for the effort of putting a library/repository/register together and pulled our findings into four principles and calls to action.

Project streams and activities

The approach the project took was, in the end, rather similar to the approach taken for the Research Skills Framework, and similar to any large research project you might take in your normal life.

After just over a year as a community, we thought we could see some patterns in the problems, and rather than sitting back and letting those problems continue, we did what we always do — we got together to decide what we thought the problems were, and also what we as a community could do about it. We were clear right from the start that we wanted nothing to do with building a solution. The problem seemed not to be in the platforms, but in the lack of clarity about what was being done with the platform. Certainly, that was in the end, our primary finding — that lack of clarity about what people were doing, why, and whom for is what causes a lot of heartache, effort, frustration, and rework across the entire discipline.

The biggest challenge was managing it across several continents using a global force of volunteers as the Covid19 pandemic hit. We managed to facilitate one in-person workshop in Melbourne before moving the project activities online where we coordinated interviews, research analysis and other activities for the three streams of the project: Research, Data and Governance. The project became, in reality, a program of work, which we’ve released in stages over the course of the past two years.

Objectives and scope:

• Review the way knowledge management systems (KMS) are used by various roles, in different organisations spanning diverse fields
• Put knowledge management systems on the map for researchers and encourage a culture shift to the point that curation in repositories/libraries is a given
• Agree on best practices, balanced governance and global standards
• Understand the ways in which different stages of the research pipeline affect the solution required

Research stream:

• Document common tasks and processes
• Identify pain points and delights in these processes
• Create value propositions
• Generate clarity about when, who for, how, and why

Data stream:

• Develop a shared glossary/vocabulary
• Agree on a universal domain taxonomy
• Build a suggested content model

Governance stream:

• Programme of Governance
• Prescribe balanced governance
• Guidelines on governance
• Checklist

Identifying best practises that work for other researchers

The project initially involved a series of community calls while the problem space was fleshed out and the scope of the project defined, then a series of interviews with people in the field who were thinking about starting a repository or were in the process of, or in some cases, had an existing repository for managing some, or all aspects of doing research.

The interviews allowed us to find out more about who was trying to implement a knowledge management system (what roles they were in and who and where were the influences coming from), how and why researchers and organisations were using repositories, and how that might speak to different levels of maturity.

We asked about the team location and size in our interviews to better understand who implements and influences the setup of a knowledge management system. This relates again to Principle 3 (below) about knowing user needs.

Overall findings from the research

We found that the most common pain points were that:

• research was not easy to find
• that it has to date often been siloed or seen as only useful once
• that there was no capacity to see if previous research could be of value to research being done now
• If the research is already in a knowledge management system, tagging and finding the relevant people to talk to was difficult
• That there were several different things people were trying to do, for different users, different purposes, and with different user needs and it was uncommon to find that user research had been done to untangle this
• That the size and complexity of an organisation and maturity of user research made a difference to the motivations and complexities of designing something fit for purpose

Different types of solutions under the same name

As we did the research, we discovered that there are four different types of things that people mean when they say they have a research repository or library. Sometimes folks had combinations of the four. Occasionally, they had all four types. In this section, we’ll describe what those four types are, and what is driving the differences. Those come from two different drivers — the users, and the research itself.

Firstly, we found that the four different models of knowledge management systems were:

• A research register — often the place researchers start — effectively a spreadsheet cataloguing research that has been done, is in progress, or is planned for the future. In combination with the research lifecycle, members of the project also found this an opportunity for streamlining and automating workload management between researchers and research operations staff
• A research data repository — another common place to start when the driver of the knowledge management work is a researcher in a small team, or a research team of one. A data repository can include a repo that allows for analysis and synthesis, using tagging, metadata, highlighting, nuggestising, and atomic research, although the last two often combine this with features relevant to other models as well.
• An ‘insights ‘hub’ — generally prepared for a wider audience, including only de-identified data. These are used as a way of organising and prioritising user needs, pain points, insights, opportunities and other forms of synthesis from the research. They are often combined with links to the raw data, or links to research outputs. In some platforms these combine with user stories, videos, and other forms of media to create a research output that is dynamic and shareable. Somewhere the research can be connected to a repository/hub of insights, facts, and/or findings that can be surfaced as a part of a system for connecting the research to ‘tickets’ for developers and designers. A notable public example of an insights hub is Uber’s ‘Kaleidoscope’.
• A research library — a front end, usually for the whole organisation to search through, that contains research reports, outputs, journeys, and sometimes also templates. Notable examples of this are the Hackney Council User Research Library and Microsoft HITS (though HITS also contains combinations of the other types as well).

Guiding principles for each use case

The following four principles consider the research lifecycle (see below), the users and their roles, their needs, as well as the structure, method, governance, and shared language required to manage research data responsibly.

Those are:

1. Know your user
2. Know what you are trying to do
3. Take time to understand their needs
4. Govern your research responsibly

We also found there are several factors to consider that were often quite unclear. These were:

• Metrics — How you measure success (most respondents had under-developed metrics about the efficacy of their research, and few had even considered how they might measure the success of the KMS itself.
• Technical aspects of implementing a KMS
• The roles that you may need to develop and maintain a KMS

Guiding principles

There were so many insights from an enormous set of interviews completed over 2019 and 2020. Here, we have organised the overarching themes from the insights into a set of principles that can be applied to provide you with guidance — what to consider as you move forward through your knowledge management system journey.

Guiding Principle 1: Know your user

It seems obvious to say, but like any other product, the knowledge management system should be designed for your users. The first and most obvious type of user is a researcher. But there are so many others. There are the people who lead researchers, people who support researchers (yes, even ReOps people are users), and people who use research. Their needs, stories, behaviours and motivations are all different. You know this. Do your user research!

It is worth noting there is a solid reason this part is often skipped, is under-developed, or changes and evolves over time — as with the rest of the profession of research operations, the knowledge management and knowledge transfer parts are often hidden work done by the researcher, off the side of their desks. This is changing, but under these conditions, these projects often start small (with a few notable exceptions), on limited resources, time, and budget. That they have emerged at all is a testament to the passion and dedication of many to scaling the impact of the craft of user research.

Guiding Principle 2: Know what you are trying to do

There was little clarity about what people were trying to do when they said they were building or buying a repository platform. Not many people had turned the principles of user centred design onto their own projects. We found that the users of the product were ill defined, as was the scope, the purpose, or the approach. Usually, a need was identified by a single person, who followed through trying to learn about knowledge management systems, trying to develop incremental change, and to seek buy-in for the project. As was found with the What is ResearchOps project, we found researchers trying hard to push their knowledge management forward, whilst being under-resourced, and over-stretched. Being the creator of the knowledge that is being managed does qualify one to manage that knowledge, but perhaps not at scale, perhaps not to design and build the container in which that knowledge is gathered, stored, maintained and shared.

There is a broad gulf between organising one’s research to make it easier to discover, maintain, and reuse, and the library sciences, product management, data governance, and engineering knowledge required to achieve this at scale.

This meant that while there were notable successes, these were either found on the smaller end of the spectrum, or where a team had been funded ongoing, to design, build, and maintain a knowledge management solution.

Developing a knowledge management system:

Understanding tagging systems and research governance is a specialised skill. User research libraries are often an ‘off the side of the desk project’, and they are generally created by researchers who are already low on time. We found from our research that where a dedicated Operations team is doing the work, they still tended not to have library science qualifications.

In some cases, the decision makers guided the repository work to gather more evidence but in others, smaller teams would start with a spreadsheet (a research register) to manage research being carried out and include tagging and taxonomy to reference and build connections between the research.

Guiding Principle 3: Take time to understand your users needs

As noted above, the four types of knowledge management system types (or combinations thereof) you are trying to build will depend very much on whom you are building a solution for, and why. A complication is also understanding just how big your ‘side of the desk project’ is going to get. It forces a very agile (in the best sense of the word) approach to scaling your solution.

Getting started (pain points):

• Asking researchers to add their data to a centralised repository is hard — each has their own ways of working, and they dislike the extra administrative burden, especially as it tends to have little to no bearing on the progression of the research they are doing at that moment. Getting to success means understanding and working to deliver on their needs, as well as others’ needs (the people using the research). Being able to articulate this dissonance is important.
• Asking researchers to add their final research outputs to a library is hard because they don’t want to go through the bother of dealing with governance of the research (who should be able to open and reuse the research), and their tagging (when they are allowed to generate new tags) tends to result in an overabundance of tags. This becomes a scaling issue, which in turn makes the research hard to find.

Guiding Principle 4: Govern your research responsibly:

Managing access in larger organisations is hard — silos become very obvious in the larger organisations, and it is possible for there to be several ‘libraries’ generated across an organisation, removing some of the strategic value of the library.

• Where a large organisation has successfully built a library for the whole organisation, financing becomes an issue as one area tends to be funding the library, with the whole organisation reaping the benefits. One such organisation existed in our interview set, and several approaches had been tried — with a user pays system, a shared funding arrangement, and a single area (the design area) funding the library for the organisation’s benefit. None of these options were perfect, and each had implications for the effectiveness and content of the library.
• The larger the organisation, the more difficult the governance of research data and research outputs became.
• Research governance refers to international, national, and organisational legislation, policies, and processes. Adhering to each of these circles of control makes governance a specialised skill.
• Research data needs governance that considers the agency of the participant (and what they have agreed to, what was stated on the consent form), the researcher (the ethics of sharing research data), and the internal stakeholders involved in the research. Even if de-identified, these three types of people need to be considered.
• However, we did hear of an organisation that literally staved off closure by pooling its various research knowledge bases and discovering something that changed their entire strategy.

Surfacing externalities that determine what is made/bought, how, why, and what it needs to deliver

Some insights that surfaced in the research were not insights one would get looking at a single organisation, and therefore may be of interest. Early on, we had a hypothesis that the drivers and needs of users may be influenced by the type of research they most commonly did. This is not because the type changes people’s needs, but because the dominant type of research can act as a flag for how research is valued, and its purpose understood within a company. The value ascribed to research, and why organisations feel they need to do research then flows through to how they want to use, or not use, their research data. From the interviews we found a tendency in organisations toward one of three research foci.

• Orgs that focus on evaluative research like usability testing (research based around the design of a product or service)
• Orgs that focus on generative research: understanding people better so they can keep a step ahead of their competitors (research in service of the overall strategy and purpose of the company)
• Orgs that combine mixed methods research from different areas across their organisation (research as a measure or evaluation of experience, those with a ‘Voice of Customer’ program)

Of course, it is possible to find organisations that do many combinations of the above, but it is worth understanding the influence this has on what you build/buy, and how.

• Organisations focused on evaluative research tend to value research as evidence. Such organisations don’t tend to have a strong interest in reusing past research to make decisions in the future so much as to show “this is why we did this” and “this is how we are making change over time”. In such organisations, create a simple taxonomy and consider building your insights hub in tools that the team is already using.
• Organisations focused on generative research tend to see research data as a reusable asset for making decisions moving forward. In such organisations it is helpful to enlist the aid of a digital librarian and co-create an extensible research data repository that allows for an organisation focused taxonomy and team-focused folksonomy. With this in place, a valuable addition would be, on completing a given research project, to create an insights hub as part of finishing the research to easily call up “insights as evidence.” These will eventually, or even at the beginning, also create a desire for a research library.

The findings highlighted a variety of situations that called for using a research repository, insights register, or library, but mainly that it takes time and effort to set up, and it seemed to be more important if the ResearchOps teams were growing or were isolated.

A key to the growth and value add of the knowledge management system was the importance of evangelising it and having advocates in leadership and management who support and promote the use and governance of a repository.

“Knowledge is inherently a part of the human experience, knowledge is communicated from person to person, and every company is different culturally.”

Research lifecycle

If we think of the process of research as a visualisation of a process or a pipeline (from generating to sharing knowledge), the many platforms that are on offer tend to meet the needs of just one part of the process — the gathering of the research, or the sharing of research. Many attempt to do both. Without being explicit about the differences in approaches required to meet the needs of users at both ends of the pipeline or process, we create confusion — that lack of clarity of purpose and context is a problem in and of itself. It influences every aspect of research repositories — what you put in them, how, when, what you need to be in place to be successful at that and what you can get out of it.

• Understanding the research lifecycle helps us know the “where, who, and how” of creating information for reuse to fit into a system contextualised for your organisation’s needs.
• The lifecycle is helpful in itself, outside of the context of repositories as it helps to identify processes involved in helping research happen (it is a process map for ResearchOps). It helps identify duplication of effort and uncover the hidden work of making research happen. Governance structures become clearer as well as the various forces at play in the development of a research repository or library.

(See also: Researching the Research Cycle article.)

If you are already using a repository and experiencing problems, consider why it’s not working, as that can tell you if it doesn’t fit into the Research Lifecycle so well.

Determining the Return on Investment (ROI)

We found there were three main motivations researchers and organisations have when they propose implementing a knowledge management solution:

1. Insights as evidence for decision making,
2. Research data as an asset for tapping into when needed
3. Scaling the impact of research by making it easy to find, reuse, and share research across an organisation

All three are relevant to the principle of what provides value to your users. The purpose of the research informs the research method, and this provides a useful flag for determining what kind of solution is required (register, repository, hub. or library). Often all are required, but this project helped us find clarity about this link between the purpose of the research, the research method used, and the motivation for developing a knowledge management system, whether that is a research register, research data repository, an insights hub, a library, or a combination of all four.

“The problem of course was having to repeat the research all the time because the insights were not really seen as evergreen.”

Overall:

We found that there don’t seem to be many people 100% happy with their KMS’s, there is confusion about what is needed and when and that rules and regulations can be an inhibitor in using the existing platforms. Some of the reasons for this may be:

• Not considering the knowledge management system a product like any other, and therefore not applying human centred design principles to doing the work,
• Lack of understanding of the complexity of going from organising a small amount of data/insights/outputs for oneself, or one’s team, to creating a scalable product.

As with many aspects of research operations, researchers are trying to implement solutions in their spare time, off the side of their desk in service of operationalising their research. They tend to do this in a way that does not adequately provide for the time, funding, skills or knowledge that allows them to easily succeed.

Research registers, research data repositories, insights hubs, and libraries that were successful were either small for a small group of users or had dedicated funding and appropriately skilled staff to work with researchers to properly develop the solutions that worked for the organisation at scale.

Coming up: Our next instalment from the program of work on research repositories will be on Research Registers — the place where most people start, and according to one of our participants with a successful larger scale library, one of the most impactful things a team can do. We’ll add the link here, too, once we’ve published in a week or so.

The ResearchOps Community, and people within the Community have put together resources for you to use. Everything the Community does is licenced under the Creative Commons Attribution-Sharealike International Licence. Please expand on it, share it, and modify it, but always give credit to the ResearchOps Community as its source.

Where possible, please share what you learn back to the Community so we can keep growing and learning collectively.

The ResearchOps research repositories project aims to develop a set of requirements, guidelines, and frameworks to build and evaluate a research repository (repo)/research library (knowledge management system).