The Researcher Skills Framework data—Sept. 2019 update

Dave Hora
Dave Hora
Sep 23, 2019 · 6 min read

10 months ago, a group of ReOps volunteers set out to build a “researcher skills framework” around a simple goal: build a tool for researchers that would help them assess their current progress and understand how to move forward in their skills and career. Researchers joined our workshops, met their communities, and also sent in data about their skills & careers—now we can begin to give it all back.

Researchers in London reflecting on the craft

There are tens of publicly available career & skill ladders, and probably hundreds or even thousands more that don’t see the light of day. Only a few apply directly to design/user/UX researchers. And most of them are developed from an organizational perspective, for the sake of administrative legibility¹: from a need to shape and structure a team from the top, or to help HR standardize leveling for pay grades. It’s an organizational necessity that doesn’t help in the way that we — researchers and the ResearchOps Community — would like to see.

Update: Project Status

We’ve run workshops to help build up local researcher-communities and gather data for the framework. More than 60 organizers have conducted over 30 workshops in 25+ cities around the world. (Feel free to see the backstory if you haven’t been following along.)

World map with ~25 pins for all of our workshop cities.
Workshop Locations: June → September 2019

Over 450 researchers and PWDRs (People Who Do Research) have participated in the workshop. We’ve got response data from 423 of them. A few final workshops are underway and by September’s end we’ll have hosted 500 researchers at this event. It’s a big milestone for the larger research community of practice. For many of our attendees, it was the first time people who did research were able to gather for just that sake, rather than a design or user-experience meetup that happens to focus on research.

It was interesting analyzing myself and realizing the way my career has evolved. I should do it more often. –Participant in Toronto, Canada

And of course, it’s an amazing opportunity to collect unbiased[self-report] data from a lovely [self-selected] set of people who identify as researchers [and are able to attend an after-hours workshop in a city that had one or more people willing and capable of hosting a Researcher Skills workshop]. We’re now cleaning up that data and starting to make sense of what we saw. After 2 more workshops are conducted, we’ll call this phase of the project finally-and-fully complete.

Data Collection

Researcher skills & career data was collected on paper during the workshops. It is not entirely scientific, certainly subjective, and was also collected across a number of cultures in a wide range of different workshop formats. In some cases, even different languages: this workshop was run in English, Spanish, Portuguese, and Japanese.

Workshop preparation (and data collection sheets) in Sweden

The data collection instrument was a small piece of the workshop — designed to be lightweight bookends to the event itself, and not to draw focus away from reflection and discussion of skills. Participants filled in their data in two pages of the participant workbook (google slides).

It was good opportunity to go deep in some issues about the research work that sometimes other UX/product workshops can’t discuss because is not the space for that. I felt like I was in a group therapy session, a safe space with people like me. —Participant in Rio de Janeiro, Brazil

After the workshop, facilitators collected the sheets and sent data back to us by a Google form. Our goal was to keep technology out of the workshop itself, not requiring participants to enter their data on a phone or laptop at the event.

Data: Cleanup & Analysis Underway

There are a few key signals we’re following in the data. If you saw the layout of the worksheet, you know the general structure of what we’re collecting—

  1. Basic descriptives: First is the composition of our attendees themselves. Who considers themselves a researcher, or a person who does research? What titles do they carry? What types of places do they work, how many years have they been in the field, and how long have they been in this specific role? We also asked for a short open-ended responses to how our participants would describe the type of work they’re doing right now.
  2. Skill selection and rating: Next, we have multiple-choice selections for “3 skills that are currently most important for my job” and “3 skills I would like to improve.” These were coded numerically, and there are some simple trends showing up in basic exploratory analysis: researchers run research and are looking to be more strategic (surprise!) What’s more interesting is the next layer — for a given respondent context, say, researchers with 1–2 years experience, what do they view as their most important skills, and where do they want to grow? What about the same for researchers who have been doing the work for 5, or 7, or even 10 years? We’re almost ready to run these analyses on a fully clean data set.
  3. Challenges, work, steps for growth: And we asked about the biggest current challenges our respondents face. We asked them how they’re looking to grow next. We asked them what ONE STEP they think they might take. Right now these responses are being manually reviewed, tagged, and coded. When all the data’s in, that same quantitative clustering—challenges vs. years in an industry, next step vs. years in industry—will give us the building blocks we need to chart out a real progression.

Synthesis & Next Steps

We are cleaning up & coding the data to answer the basic questions above. This is the first, descriptive step, one piece on the larger process of making a meaningful framework that researchers can use to identify where they are in a larger progression, and figure out what they can consider next to think about their career, their breadth of skills, the specific resources they might make use of or methods to attempt. As we identify the clusters, make sense of open-and-closed responses, we’ll start to sketch out our framework, a tool for growth and progression.

I feel like I might know a bit more than I thought. —Participant in Austin, Texas, USA

When the data’s all-in, coded, and cleaned of any personally identifiable information, we’ll set it up as open-source on the project’s github page, and do the same for any queries/reports we use to make quantitative sense of what’s happening. Everyone who’s contributed a piece to this project can see the whole picture, and will also be able to make their own sense of the raw & coded data.

Finally, before this year is out we’ll release a test framework within the ResearchOps community, publish more broadly a framework, and (once those last 3 workshops are complete) open-source release the data. Look out for the next update on Medium, or in the #skills_framework channel of the ResearchOps Community Slack if you’re already on board. It will be a lot more interesting than this one. :)

Footnotes:

[1] Recommended read: Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed by James C. Scott

researchops-community

Talking about how to operationalise research

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store