Beth Simone Noveck and Jason Williams-Bellamy
Opinion: A well-timed survey may be the difference between failure or success
The reason: a shortage of digital skills among public leaders made it difficult to measure progress and share common lessons learned.
The shortcomings of Scotland’s efforts to digitise government services might have been avoided, had they been more aware of the state of digital literacy in the country’s public sector. Scotland’s lack of insights into the skills and abilities of its public servants is not unique.
As public institutions seek to become more innovative in how they work, they generally lack baseline information about the current state of skills in the organisation.
In the private sector, too, we worry about workers at every level having the skills they need to thrive in a high-tech economy. Managers regularly turn to surveys and questionnaires to identify what people know and what they want to know.
But, in the public sector, we know very little about current skills and competencies.
Asking the right questions
Many governments are investing in creating new training programs in an effort to change working practices.
But they are often doing so without knowing either what people know, who needs to know what, or how they want to learn. There is more than a little certain irony that, in an effort to spread the teaching of skills such as data analysis and human-centred design, public organisations are not applying those very same approaches to conduct an assessment of training needs prior to developing their training programs.
Asking people about their skills has its own limitations, as people are notoriously poor and unreliable at assessing their own expertise
For public organisations wishing to boost their performance and improve how they solve public problems, they should first conduct a training needs analysisto measure the current state of innovative problem solving skills and the nature of any skills gaps to understand what people know, what they would like to know and how they learn best.
The OECD team conducted interviews with 90 public servants and surveyed over 150 people focusing on assessing competency in six skills:
- Design thinking,
- Digital thinking,
- Data and evidence use,
- Curiosity and flexibility, and
- New narratives and cooperation
The research, while based on a small sample, enabled the consultants to follow up with in-depth interviews and later produce a 120-page report, documenting that innovation skills exist only in pockets in the Chilean public sector but without any coherent framework to bring them together or systematise them in public practice.
Respondents indicated a particular skills gap in the area of citizen engagement. They also felt that, whereas their organizations were better prepared to use data science skills, managers frequently did not support employee efforts to practice innovative ways of working. Alas, the OECD does not publish the survey questions.
Surveying digital trends
The findings in Chile paved the way for more expansive empirical research into public sector innovation skills. Following suit, the Canada Digital Serviceput out a skills survey at the end of 2018, asking public servants, “What digital training do you need?”
The survey (also not published) sent to a random sample of 5500 recipients about their knowledge of 30 digital trends such as cybersecurity, data visualisation and machine learning.
They used the results to shape the roll-out of the Digital Academy in 2019, which teaches digital literacy through a series of courses that target public servants at different levels of government — from short crash course videosaimed at all 250.000 Canadian public servants to intensive multi-day workshops for executives and senior leaders.
The Academy’s use of a skills survey prior to rolling out its training has led to a better-informed design and a more cost-efficient roll-out.
But asking people about their skills has its own limitations, as people are notoriously poor and unreliable at assessing their own expertise.
Thus, GovLab developed a survey of innovation skills designed to overcome the deficiencies of self-reporting. Both ICMA and Bloomberg Philanthropieshave circulated these currently ongoing surveys (here and here, respectively) to their constituents in the United States and ANZSOGdid so (here) in Australia and New Zealand. Results will be published later this summer.
Saying and doing are different things
In connection with six innovation competencies, we asked people to distinguish between: 1) their ability to use a particular method, or 2) explain it to others.
The reason we ask “can you explain this” is to get at a more honest answer than if we asked “do you know what this is.” If we asked the latter, everyone would say “of course.” More importantly, we ask people “can you do this thing.”
Skills surveys are an important opportunity to take the pulse of innovation health in your organisation
By asking people whether they can use a skill and following up that question with a request to specify how often, the hope is that we will get a more robust assessment.
Also, before any questions are posed, the survey presents the respondent with a description of each skill in words and in a video; a real world example; and additional resources.
This is done to avoid miscommunication about key terms. For instance, while some public servants may use open innovation in their daily work, they may not know the synonymous term “crowdsourcing.”
After all, one man’s design thinking is another man’s human-centred design. We wanted to get to the bottom of what people know regardless of nomenclature.
Mapping knowledge gaps
The other benefit of surveying employees about what they know — and what they want to know — is that it enables us to determine how knowledge is spread across the organisation.
Identifying such gaps gives us the ability to target training and resources to the areas of greatest need. The survey we recently ran with our colleagues at Monash Sustainable Development Institute on behalf of the Australia New Zealand School of Government (ANZSOG) asks people about 9 innovation skills:
- Problem definition
- Human-centred design
- Data analytical thinking
- Open innovation
- Behavioural insights
- Lean-agile methods
- Impact evaluation methods
- Evidence synthesis methods
- Systems thinking
We try to pinpoint the skills gaps by starting with questions on the respondent’s position and role in the Australian Public Service. Questions like: “are you a state or federal public servant,” “what is your organisation’s jurisdiction,” or “what best describes the type of work you do?”
Asking these questions in parallel with an optional request for more detail about what kinds of problems people are tasked with solving could help ANZSOG triangulate which public servants use the most innovative methods and who is lagging behind.
In addition to asking what people know and what they would like to learn, you also find out how public servants want to learn so that you can ensure high participation in any programs you design.
The ANZSOG survey accomplishes this by asking respondents where they like to learn (online, face-to-face, or group setting), when (at home or during work hours), and with what frequency.
Taking the pulse on innovation
To be sure, it is hard to get people to respond to surveys.
But identifying a sample of the eventual audience (across diverse agencies, levels of government, seniority and roles) can help to ensure that all voices are represented. In Canada, they first sent their survey to a random sample prior to inviting anyone to self-select to participate. Publicising the opportunity to participate via popular channels, such as social media and newsletters, is an obvious strategy for encouraging participation.
Finally, using live events to announce a survey and giving people time to respond via their mobile phones or via tablets at a physical station can help to boost responses.
Skills surveys are an important opportunity to take the pulse of innovation health in your organisation and generate sources of data to inform further qualitative research with public servants about how they will make the transition to new ways of working.
As Scotland’s troubled digital strategy shows, building digital capabilities in government — something many public institutions are trying to accelerate — is no easy task; it depends more on human capital than on technology itself.
But when armed with ambition, coordination, and a good skills survey, half the battle is already won. — Beth Simone Noveck and Jason Williams-Bellamy
This is the second article in a series about public entrepreneurship written by GovLab director, Beth Simone Noveck. Click here to read or re-read the first article in the series, outlining what governments need to do to train 21st century leaders for the public sector.
Thank you to the Australia New Zealand School of Government for supporting the research in this post.