The Researcher Skills Framework data—Sept. 2019 update
10 months ago, a group of ReOps volunteers set out to build a “researcher skills framework” around a simple goal: build a tool for researchers that would help them assess their current progress and understand how to move forward in their skills and career. Researchers joined our workshops, met their communities, and also sent in data about their skills & careers—now we can begin to give it all back.
There are tens of publicly available career & skill ladders, and probably hundreds or even thousands more that don’t see the light of day. Only a few apply directly to design/user/UX researchers. And most of them are developed from an organizational perspective, for the sake of administrative legibility¹: from a need to shape and structure a team from the top, or to help HR standardize leveling for pay grades. It’s an organizational necessity that doesn’t help in the way that we — researchers and the ResearchOps Community — would like to see.
Update: Project Status
We’ve run workshops to help build up local researcher-communities and gather data for the framework. More than 60 organizers have conducted over 30 workshops in 25+ cities around the world. (Feel free to see the backstory if you haven’t been following along.)
Over 450 researchers and PWDRs (People Who Do Research) have participated in the workshop. We’ve got response data from 423 of them. A few final workshops are underway and by September’s end we’ll have hosted 500 researchers at this event. It’s a big milestone for the larger research community of practice. For many of our attendees, it was the first time people who did research were able to gather for just that sake, rather than a design or user-experience meetup that happens to focus on research.
It was interesting analyzing myself and realizing the way my career has evolved. I should do it more often. –Participant in Toronto, Canada
And of course, it’s an amazing opportunity to collect unbiased[self-report] data from a lovely [self-selected] set of people who identify as researchers [and are able to attend an after-hours workshop in a city that had one or more people willing and capable of hosting a Researcher Skills workshop]. We’re now cleaning up that data and starting to make sense of what we saw. After 2 more workshops are conducted, we’ll call this phase of the project finally-and-fully complete.
Data Collection
Researcher skills & career data was collected on paper during the workshops. It is not entirely scientific, certainly subjective, and was also collected across a number of cultures in a wide range of different workshop formats. In some cases, even different languages: this workshop was run in English, Spanish, Portuguese, and Japanese.
The data collection instrument was a small piece of the workshop — designed to be lightweight bookends to the event itself, and not to draw focus away from reflection and discussion of skills. Participants filled in their data in two pages of the participant workbook (google slides).
It was good opportunity to go deep in some issues about the research work that sometimes other UX/product workshops can’t discuss because is not the space for that. I felt like I was in a group therapy session, a safe space with people like me. —Participant in Rio de Janeiro, Brazil
After the workshop, facilitators collected the sheets and sent data back to us by a Google form. Our goal was to keep technology out of the workshop itself, not requiring participants to enter their data on a phone or laptop at the event.
Data: Cleanup & Analysis Underway
There are a few key signals we’re following in the data. If you saw the layout of the worksheet, you know the general structure of what we’re collecting—
- Basic descriptives: First is the composition of our attendees themselves. Who considers themselves a researcher, or a person who does research? What titles do they carry? What types of places do they work, how many years have they been in the field, and how long have they been in this specific role? We also asked for a short open-ended responses to how our participants would describe the type of work they’re doing right now.
- Skill selection and rating: Next, we have multiple-choice selections for “3 skills that are currently most important for my job” and “3 skills I would like to improve.” These were coded numerically, and there are some simple trends showing up in basic exploratory analysis: researchers run research and are looking to be more strategic (surprise!) What’s more interesting is the next layer — for a given respondent context, say, researchers with 1–2 years experience, what do they view as their most important skills, and where do they want to grow? What about the same for researchers who have been doing the work for 5, or 7, or even 10 years? We’re almost ready to run these analyses on a fully clean data set.
- Challenges, work, steps for growth: And we asked about the biggest current challenges our respondents face. We asked them how they’re looking to grow next. We asked them what ONE STEP they think they might take. Right now these responses are being manually reviewed, tagged, and coded. When all the data’s in, that same quantitative clustering—challenges vs. years in an industry, next step vs. years in industry—will give us the building blocks we need to chart out a real progression.
Synthesis & Next Steps
We are cleaning up & coding the data to answer the basic questions above. This is the first, descriptive step, one piece on the larger process of making a meaningful framework that researchers can use to identify where they are in a larger progression, and figure out what they can consider next to think about their career, their breadth of skills, the specific resources they might make use of or methods to attempt. As we identify the clusters, make sense of open-and-closed responses, we’ll start to sketch out our framework, a tool for growth and progression.
I feel like I might know a bit more than I thought. —Participant in Austin, Texas, USA
When the data’s all-in, coded, and cleaned of any personally identifiable information, we’ll set it up as open-source on the project’s github page, and do the same for any queries/reports we use to make quantitative sense of what’s happening. Everyone who’s contributed a piece to this project can see the whole picture, and will also be able to make their own sense of the raw & coded data.
Finally, before this year is out we’ll release a test framework within the ResearchOps community, publish more broadly a framework, and (once those last 3 workshops are complete) open-source release the data. Look out for the next update on Medium, or in the #skills_framework channel of the ResearchOps Community Slack if you’re already on board. It will be a lot more interesting than this one. :)
Footnotes:
[1] Recommended read: Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed by James C. Scott
