Evidence-based policymaking: a story emerges from audience research
Our audience research for Knowledge4Policy is signed, sealed and delivered. What does it tell us, and what do we build next?
After our first two focus groups (with researchers and policymakers) at the beginning of the year, a third focusing on policy assistants was held last Spring, finishing the data collection phase of our audience research project.
Since then, in parallel to building the Platform’s ‘Beta version’, we’ve been studying and comparing findings from the three different strands of the audience research: the online survey’s almost 300 responses; twelve one-on-one ethnographic interviews; and all three focus groups.
The findings (available in full on our ‘beta platform’) will underpin our development workprogramme, which we’ll discuss and adopt, in outline form, over the next few weeks. There’s just one problem:
the three different audience research methods… didn’t agree with each other
Bar charts or interviews?
Somewhat annoyingly, the findings from the different audience research methods didn’t agree.
Specifically, the survey contradicted what the focus groups and interviews told us concerning how policymakers would engage with the platform.
For example:
- in the survey, policymakers rated “detailed studies” as the most valuable content type, while admitting in interviews and focus groups that they’d never find time to read them
- according to the survey, policymakers were more likely to interact online than researchers(!), while maintaining in interviews and focus groups that this was the last thing they’d do, unless the community was both closed and trusted.
The audience research specialists explained this as probably a combination of several factors:
Ambiguous wording, multilingual audience?
There may have been some ambiguity in the wording of some of the questions. The ability of five people to interpret the same question in six ways is well understood among surveymakers. This effect is easily accentuated when a diverse group of people from 20+ countries answer a survey in their non-native language.
More than one sort of policymaker
The survey treated policymakers as a single audience group, rather than distinguishing between senior policymakers and their policy advisors.
This was probably an important factor — when we split the responses from senior policymakers and policy advisors and analysed them separately, we saw major differences emerge (left).
Hence our second and third focus groups, designed after the interviews and an initial survey analysis, explicitly separated members into policy advisors and policymakers.
Social acceptability bias
Also known as social desirability bias, this is the tendency of people to provide answers they unconsciously believe will be viewed favourably by others, even when answering an anonymous survey. This phenomenon has gotten a lot of attention recently, given the failure of polls over the last few years to accurately predict election outcomes.
“… social acceptability bias is likely to have driven certain audience groups (particularly policymakers) to declare greater enthusiasm for functions and features of the platform than was clearly the case when engaging with them in a more unstructured environment, or when able to ask follow-up questions or probe opinion in focus groups” — slide 25, final report
An ongoing process
Overall I agree that we should give greater weight to the interviews and focus groups than the survey. But we remain cautious, and so will:
- continue seeking more relevant audience research — it’s surprisingly hard to find, so contact us if you have some;
- engage with existing users as soon as we have the appropriate tools (which means favouring those tools in our development programme);
- invest in web analytics so we can measure what our users actually do, rather than what they say they’ll do.
What success looks like: one possible story
How to summarise the 80+ page audience research report, not including annexes? Why not tell a story?
The following story tells one way K4P could help get scientific knowledge to policymakers in a form they can use, when they need it. It is only one of many possible stories, of course, but this particular one includes many of the features the audience research demonstrates K4P will need to succeed.
Interestingly, some of these ‘features’ are not web functionalities or content types — they are services. So, with apologies for misappropriating few steps from Joseph Campbell’s Hero’s Journey:
The Call to Adventure
The constantly rolling EU policy agenda raises a new issue over the horizon. A senior EU policymaker asks a junior policy advisor to do some research for a speech next month.
The junior policymaker turns to the K4P platform, which she had previously discovered through its social media, events, press coverage, etc.
Behind the scenes: This requires constant, long-term audience engagement by the platform as a whole towards policymakers around Europe. In practice, different K4P Themes (next) will need to be promoted to different groups of policymakers.
Meeting the Mentor
The junior policymaker discovers a high level Theme covering the areas of her MEP’s Committee, and on it:
- a useful one-page summary, including infographic
- deep links back to the underlying knowledge (data, studies, publications, etc.) which it summarises, and the scientists active in the field who developed that knowledge
- a ‘request help / find me an expert’ feature.
Behind the scenes: K4P’s information architecture will need policy/legislative interfaces — i.e., presenting the work of multiple Knowledge Services in terms of the policy agenda. These must feature supportive, integrated content and service conceived specifically for policymakers, driven by an editorial calendar reflecting the policy agenda. They will also need to be actively promoted to their specific policymaker communities (see above).
Crossing the Threshold
The junior policy advisor Registers to K4P to use the ‘request help / find me an expert’ feature on the Theme she’s looking at.
Her request arrives at the online community manager (OCM) responsible for that Theme, who answers:
“We keep an eye on the policy agenda and knew this was coming. So we recently asked our community of scientists, which we’ve been convening from within and outside the EC for years, to help update that summary and related content. We’ll send it to you when its ready, but if you can’t wait we can help answer your questions now.”
Behind the scenes: K4P features must include Register, About Me and Request Help / Find me an Expert. OCMs will be needed to manage the above Themes, field queries, and help the related Knowledge Services develop their communities of practice in their respective scientific fields.
Tests, Allies & Enemies (mostly Allies)
The junior policy advisor uses the existing material for the speech and then organises, with the OCM’s help, an online Q&A between her boss, experts from the two most relevant Knowledge Services, and scientists from their communities.
That conversation restarts when the summary material is updated, and results in face-to-face briefings when the relevant scientists next visit Brussels.
Behind the scenes: Once the Platform’s OCMs have coordinated multiple Knowledge Services to meet the policymakers’ needs, s/he leaves it to them.
The above story is not the whole story
While the above story is only one of many, it helps illustrate the diversity of online content types, interactive features and offline activities that K4P must master. However, it doesn’t include them all.
For example, while it refers to Communities of Practice — scientists collaborating to develop the ‘knowledge’ part of Knowledge4Policy — it doesn’t foresee Communities of Interest, where policymakers get involved.
That’s because — contrary to survey findings — the interviews and focus groups indicated that few policymakers will join such communities, preferring to talk to scientists face-to-face, or get answers by email or videoconference (see ‘Online Community Management’, previous post).
some policymakers and policy advisors may join a community which is closed and trusted
However, the audience research indicates that some policymakers — and probably more policy advisors — may join a community which is closed and trusted. So perhaps the above story could be extended, with the junior policymaker joining a closed online Working Group, created to continue the conversations begun in the face-to-face briefings. This would reflect my experience that the best way of getting an online community going is to integrate it with offline, face-to-face events.
But we won’t develop that straight away: while the above story will guide the next phase of the Platform development, it will not define the Platform forever. We’ll probably launch online communities for scientists first, and if we see interest from policymakers we’ll adapt to accommodate them.
Audience research, in other words, is not over: we’ll embark on the following developments, but then constantly analyse uptake to ensure we stay on the right track.
What must happen to make these stories come true?
What we’ll need to do
If we’re going to make the above happen, the K4P site must:
- manage data, information, knowledge and wisdom up and down the Linked Knowledge Pyramid (above)
- provide a series of useful services to scientists at the lower three levels: e.g., highly granular knowledge management; access to a cross-EU online community of peers, integrated into the conference circuit; etc.
- provide additional services to policy makers — particularly policy advisors — at the upper two levels (knowledge and wisdom): interfaces reflecting the legislative agenda; dedicated content types for policy; Request Help / Find me an Expert; etc.
- prepare dedicated content for policy reflecting the legislative timetable;
- promote the above content and services to the world of policy;
- provide capacity-building to both groups to help them talk each other’s language.
What we’ll need to be
To do that, the K4P project must become a centre of excellence in knowledge management at all levels of the Pyramid, mastering:
- knowledge brokerage: not just creating (and promoting) scientific knowledge for policy consumption, but also facilitating conversations between the two worlds
- convening knowledge communities, or communities of practice for scientists, so that the JRC has a wide and deep pool of expertise to answer policymakers’ short-term questions
- loads of technical issues: content management; UX; taxonomy; AI-driven Natural Language Processing; linked data management; data visualisation; multilingual federated search, and more.
We clearly have our work cut out for us.
Join in
Transparency is one of our bywords, so we’ll publish as much as possible here. Follow us to stay informed, and contribute by either Responding to this post or submitting your own.
