Getting real survey answers out of smart, busy people
How Quartz got 940 of the world’s business leaders to tell us how they consume news and respond to advertising
In March, we conducted the Quartz Global Executives Study, a survey to better understand how senior business executives—our core users—consume news and engage with advertising.
We learned a lot about executives’ media consumption, some of which we’ve handily summarized here. But perhaps even more interesting was what we learned from the way we ran the study, which was unconventional, a little risky, and ultimately highly successful.
Why this matters
It’s important to our mission: Quartz was built to serve business people in the new global economy: people at the helms of companies and the frontiers of industries. They tend to be early adopters and frequent travelers who move quickly and constantly. They have busier schedules than the average person and, consequently, even less patience for low-quality content.
While much work has been done on how mobile adoption and the social web have changed the broader population’s habits, not a lot is known about senior executives, specifically. There’s a reason for that: survey research is expensive and complicated, particularly when your target respondents are the heads of global companies. The cost to identify, reach, and incentivize senior business executives typically ranges from about $70,000 to well over $100,000, largely driven by the higher incentives they supposedly require.
And cost aside, the accuracy of such surveys is questionable: How many smart, busy people—let alone CEOs—do you know would a) notice the outreach email, b) actually open it, c) click through to start the survey, and d) spend the time to finish it… all for the promise of airline miles?
What we did
To attempt a 65-question survey of C-level and other senior executives was thus an ambitious endeavor. But we attempted it anyway—and the results were highly encouraging.
Here’s a summary by the numbers:
- Number of survey questions: 65
- Survey completion rate: 55%
- Share of responses submitted on a mobile device: 55%
- Average completion time: 16 minutes (12.5 minutes on phones)
- Number of responses fitting Quartz’s “global executives” criteria: 940 (we received a total of 1,797 responses)
- Amount we spent: $0
A 55% completion rate for any survey is considered really good—but considering how much time this particular survey took, and how little time these respondents have in the day for non-essential tasks like this one, it felt especially victorious.
Even more rewarding was the group of individuals we ended up with: 940 global executives, 43% of whom were C-suite (the rest being directors and partners, VPs, heads of business units, and other senior leaders). Executives came from 61 countries and 36 industries.
And, because they willingly provided their personal identification information, they weren’t faceless survey respondents sourced from a purchased list, but real people, with names, job titles, organizations, profile pictures, and LinkedIn bios. A sample:
- A C-level executive at AIG
- A C-level executive at Viacom
- The president of a major enterprise software company
- A managing director at PwC
- A former managing partner at Booz & Company
- A partner at Kleiner Perkins Caufield and Byers
- A director at McKinsey
All this, without spending a dime.
How we did it
By sticking to a simple guiding principle: Don’t just ask.
People in charge of companies get asked for things every day, nonstop. There already aren’t enough minutes in the day for CEOs to read every important email, attend every important meeting, and think through every critical business decision. If some outside entity is going to try to steal away precious minutes that could otherwise be spent pursuing business objectives (or personal ones, for that matter) it had better give a good reason for doing so.
That’s what guided our approach from start to finish: Before launching into a 65-question survey, we gave people a reason to want to answer. In other words, our focus was: user first, survey second.
Here’s what that looked like in practice:
1. We offered no financial incentive
Picture a senior executive who makes $500,000 a year. The amount it would take to offset the opportunity cost of her time being spent on our survey would be significant—certainly larger than the promise of a few airline miles or the chance to win a $250 gift card. Moreover, even if said executive was motivated by such an incentive, who’s to say she wouldn’t blow through the questions as quickly as possible, accuracy of her responses be damned, just to get to the end where a free Jambox was waiting?
Instead of offering a financial incentive and risk getting hollow, rushed responses, we provided context and relevance, producing genuine, thoughtful responses.
We sent the email below to 500 senior executives. You might notice that it spurns a few email marketing standards: It’s long, for one thing, and the call to action doesn’t appear until the fourth paragraph. Before asking people to participate in the study, we set the context for it (“The news business is changing more quickly than ever before…”) and its relevance to them (“…how the world’s smartest, busiest people consume information”):
The result: a 34% clickthrough rate; real answers; and a smattering of nice emails sent to us in response, like this one from the Managing Director of a London-based hedge fund:
“THAT was a masterclass… responses submitted!”
2. We paid attention to what wasn’t working, and changed it
We closely monitored people’s reactions to the way we were talking about the survey, and adjusted our messaging accordingly.
As an example, we ran four announcements in our morning news email, the Quartz Daily Brief. This was our worst performing placement:
How do you like to consume news? Tell us by taking the Quartz Global Executives Study.We asked for (actually, commanded) something too quickly + barely gave any context + sent it on a Friday = 0.41% clickthrough.
This was our best performing one:
Are you reading this on a) a phone, b) a tablet, c) a computer, or d) something else? No, really, we want to know. We’re conducting some research on how you like to consume news—take our survey here.We used a more personal tone + provided some context + led with a more answerable question (“Hey, I know the answer to this!”) + sent it on a Saturday when people actually had the time to do the requested action = 4.48% clickthrough.
3. We adapted our message to whomever we were speaking to
Respondents were sourced from Quartz’s audience and through partner organizations with user demographics similar to ours, and contacted through a range of channels (dedicated emails, integrated placements, tweets, etc.). In each case, we used the two points above to adapt the framing and tone of our message for each channel and its audience.
4. We used design and language to create the easiest, most efficient experience possible
Just as news consumers are now the destination rather than the news itself, and just as ad impressions are nothing without the users bestowing them, it was important that our survey experience be focused on the user first, and the results second.
We used Typeform to create a single-page survey that users could navigate via keyboard if they were on desktop, and with minimal frustration if they were on mobile. Rather than hunt for an answer from a list of a poorly organized options then patiently wait for the information to submit and the next page to load (especially painful on mobile), users stayed on the same page throughout, zipping from question to question:
- We carefully architected the question flow and held mini-focus groups to test logic and clarity, and identify points of fatigue.
- We used human language and images of our own staff to convey empathy to those who were lending us time out of their busy days:
The result: Most of the people who started the survey finished it (55%, to be precise), and some enjoyed it so much that they tweeted about their experience afterward:
Our approach to this project was designed to be emblematic of Quartz’s approach to everything: Focus on the user first, and your product second.
Just as our reporters choose story formats based on what’s most efficient for the reader, just as the Quartz Daily Brief links to stories from around the web and not just the ones on our site, and just as we’re continuously challenging advertisers to create better experiences, so it was with the Global Executives Study. We didn’t just ask. We gave users a relevant, high-quality, and beautifully designed experience. In return, they gave us answers.
The findings of the Quartz Global Executives Study may be found here.
Mia Mabanta runs strategic projects at Quartz. She joined Quartz after subscribing to the Quartz Daily Brief and noticing that, in just a few days, an email newsletter had changed her decade-old morning news habit. You can sign up for the Daily Brief here, and follow Mia on Twitter @miamabanta.
