5 Tips On Implementing A Survey Strategy
By Evi K. Hui & Philip Rhodes
Getting feedback from users is integral to the success of a business and the product development process. Over the years, Flexport, like many organizations, organically developed multiple surveys for different users across our business and product portfolio.
Surveys were created and analyzed by different parts of the organization (business, marketing, product, design). This results in poorly coordinated surveys to users and a siloed approach to understanding user sentiment. Some additional problems include:
We needed a holistic approach to surveying, so we took it upon our team to align other teams and drive a strategic framework for gathering feedback. We called this the Survey Strategy, which can be leveraged by any team across Flexport.
Since its initial launch a year ago, we have been generating scalable insights across the org.
We have deployed:
- 18 rounds of CSAT surveys to key user groups with 6257 responses
- 1 continuous NPS survey with 20,772 responses
- 3 in-app Kano surveys
49,176 email and in-app survey impressions were made
12 reports were written
8 presentations were made
In addition, we implemented persistent feedback in our app footer and began experimenting with our first Jobs-To-Be-Done (JTBD) surveys.
Here are the top 5 things we learned, that you can apply to improve survey strategy within your business.
Tip 1: Bring awareness to the problem and align owners of the surveys
Our process started with identifying all the surveys and their owners. By doing this and bringing the owners together, we were able to collectively see the problem (and share that problem). We created a spreadsheet to capture the details of the surveys, including the goals, users, and frequency.
This large group of stakeholders collectively agreed that sending surveys outside of a centralized process was not ideal and needed to be addressed. We asked these survey owners to be part of a committee to meet monthly and to help create a holistic approach going forward. Bringing everyone together to collectively define the problem made aligning on a solution much easier, while allowing everyone to understand each other’s perspectives and needs, and how a holistic approach was needed to address each team’s needs.
Tip 2: Select the right survey methods and communicate how they all fit together
We spent the next few weeks conducting reviews of various survey types and best practices. We also drew inspiration from our team’s previous experiences where we worked with dedicated quantitative research teams and in-house survey experts.
Our overarching strategy became more clear as we started to hone in on the best-fitting surveys that would achieve each team’s goals. For example, we chose to use CSAT across the tech org for understanding user satisfaction with the overall experience and subareas of the app. We picked in-app JTBD to understand the ease of accomplishing key tasks. The visual below shows the other surveys, who in the org owns them, and how they all fit together, with an example question, scale, and purpose.
It was useful to have this visual for proposing the Survey Strategy and getting alignment and support from our partners in business, marketing, product, and UX.
Tip 3: Choose one survey platform for managing, executing, and analyzing
Next, we proposed to the committee that we choose only one survey platform. The committee collectively outlined the benefits of a ‘one platform’ solution. These benefits include:
- One place to host unified survey templates that teams can re-use
- One place to capture survey results
- Ability to manage which users get which surveys and their frequencies by using a directory (pool of potential respondents)
- Ability to view a user’s history of survey responses
- More in-depth analysis by looking at responses across surveys
There was little pushback on this proposal, as the team agreed that these were important reasons to use a single platform. We chose Qualtrics, as some teams were already using this platform, and our analysis concluded that it was one of the most robust for survey building, managing, and analyzing. We knew adoption would be easy to assure because of this.
However, we wanted to ensure that everyone across the business could easily access the results from the surveys without everyone having a Qualtrics ‘seat.’ This was probably our biggest challenge, as we needed to get the data to flow into Snowflake/Looker, which is widely used at our company for storing and viewing user data. There is no easy API integration, so this required our business tech team to implement it.
Tip 4: Ask the right questions and reverse engineer your survey design
Once all surveys are hosted on the same platform, the team can be more effective in creating new surveys, analyzing responses, and synthesizing insights. These benefits include:
- Tying two project data sets together; for example, we can now easily connect CSAT scores to NPS scores
- Conducting more detailed analysis (by role, etc.), as we can splice the responses (from across surveys) using a standardized set of variables including, role, tenure, location, and the like.
- Only asking key questions; now we can use responses from previous surveys, and centrally sorted data to reduce the number of survey questions
An additional benefit of a consolidated survey strategy is that it also allowed for teams to align on simplicity and consistency in survey designs. For example, we opted for a standard practice for CSAT survey design so that all CSAT surveys follow the same structure. This allows teams to compare their survey results with more rigor to historical surveys.
Tip 5: Provide guidance and make it easy for others to run surveys
In order to scale and make it easy for others to succeed, we created a playbook for each survey type and templates that others could use. We expect designers, marketers, product managers, and our team’s researchers to leverage these tools.
This is a post-launch CSAT/JTBD survey template with an example of Flexport Visibility.
In our general CSAT playbook, we included a slide on the frequency of CSAT, its timeline, and owners. The playbook also includes how to test your survey on our platform before deploying and how to target specific users.
We also created a visual of when each type of survey should be deployed in the product development process. The visual also includes other mechanisms for feedback, such as qualitative interviews during the discovery and beta phases.
Our work is not done–implementing a survey strategy across a company is a lot of work and can take quarters, maybe even years. Some of the work we are looking forward to is connecting all the survey results together and synthesizing them to produce top insights by personas and product areas. We are also working on connecting JTBD attitudinal data with analytics behavioral data for a full picture of measuring the user experience and identifying areas that need work. Speaking of work, we are hiring UX researchers for our team. If you are interested in working with us on this and other research on the global supply chain, check out our hiring page.
We would not be where we are without our collaborators Michelle Heckert and Courtney Mauk, so a big thanks to them!
