Measuring Fun: Inside Snapchat’s Blackbox
Twenty-three hours and change later, Marissa Breton logs back on to HubSpot’s Snapchat account, takes a picture of the viewer counts, and then logs these “retention” numbers from the app’s minimal interface into an Excel spreadsheet. As the social media team’s Snapchat guru, she’s trying the company’s investment in the app.
It seems a bit cumbersome, but it’s necessary. Every video on a Snapchat story disappears 24 hours after it’s posted. There is no easily legible dashboard or “Export to CSV” to pull the numbers from, so Marissa has to get her hands dirty.
“We manually track every story,” she explains. “What I’m looking at most closely is what we’re calling ‘retention rate.’ Basically, the number of people who watched the first segment of the story all the way through the last. I literally will set a calendar reminder 24 hours after the first story. It’s brutal.”
But knowing whether Snapchat’s effectively reaching an audience is necessary, and whether audiences tune in for the full experience is a piece of that. Hubspot is in the business of selling marketers solutions that verifiably work. And part of “working” is measuring efficacy.
Still, little on Snapchat is verifiable. For marketers and measurers accustomed to basing their decisions on dashboards like these:
the Snapchat “dashboard” is decidedly primitive:
What data is there?
- How many viewers clicked in,
- What their usernames are,
- Whether they screenshotted.
These metrics are shallow to begin with, and you can’t drill deeper into them. Do users watch with sound? Do they skip through certain takes? It’s like a Tootsi Roll pop: We may never know and even the wisest influencers are more eager to bite in than find out.
In spite of measurement issues, HubSpot decided to establish a presence on the platform. After some initial hesitation, the social media team couldn’t resist the call of the zeitgeist.
“Snapchat was something that had been on our radar for a long time,” Marissa says. “We pride ourselves on being on the cutting edge of this type of thing. That being said, obviously Snapchat’s audience is younger than our traditional audience. The reason we finally decided to start is we saw a major uptick in interest from marketers. Everyone was really interested in Snapchat.”
This part is true. Grubhub has ordered up on Snapchat (but do viewers purchase more food?). Everlane is trying it on (but do viewers buy more shirts?). It’s no longer just for the disaffected youth (but marketers are hoping the disaffected youth stay).
The interest coincides with Snapchat surpassing Twitter in daily active users. When The Signal recently spoke with Hunter Walk, he offered his theory about Snapchat fever: “One of the most wonderful things about technology is lowering the bar for creation, allowing more people to be creative, and having that creation occur within an audience and community leading to collaboration.”
But in simplifying tools for creation, the company has also given its app a bareness that passes for Jobsian minimalism. Unless you’re one of the few media properties paying a high price to sit on its Discover feature, Snapchat remains a blackbox. Even for advertisers, who are clamoring for paid space on the platform, the app can seem like magical thinking. In March’s Bloomberg, Tom Gruhler, a marketing VP at Microsoft, said, “Snapchat is awfully expensive, and there’s pretty much a lack of data and visibility.”
There’s no way to catapult the eyeballs on Snapchat to other channels. There’s no way to record a Snap story’s efficacy (short of hand-compiling a spreadsheet). But for many, there’s this feeling about Snapchat, however hard to articulate. Marissa simply knows there is an ROI there, even if it’s hard to name for the time being.
They don’t want you to measure success
Meanwhile in California, investor and Twitch co-founder Justin Kan was reading up on DJ Khaled. Apparently, Khaled had gotten lost on a jetski and documented the whole harrowing experience on Snapchat:
Although Justin had a crucial role in the early history “lifecasting,” he hadn’t bought into Snapchat at first. For three or so years, even as a partner at bleeding-edge Y Combinator, the app didn’t compel him.
But after hearing about Khaled, Justin didn’t need to see numbers. He needed to see a man lost on a jetski.
In the post-Khaled era, Justin is on Snapchat daily and he refuses to host a subpar channel. Each day begins with a branded title card. He dishes entrepreneurial advice and rides his own jetskis. His Twitter deflects followers to his Snapchat.
“What’s most surprising about adopting it is probably the amount of engagement there is,” Justin says. “What works best right now is if I do a SnapStorm about entrepreneurship. Then, I do my final post with a summary slide. 15% of the people who view it are screenshotting it, which is pretty high.”
Impressed with this engagement, Justin decided to try a new experiment. In May, he accepted applications for Y Combinator’s Fellowship over his Snap channel. The applicants would be able to pitch their startup by creating videos on his channel.
And the platform’s reach was impressive.
“A lot of the applicants for the Fellowship were really good companies,” Justin says. “If we had pre-allocated more of an investment budget, we would’ve invested in more. Typically in YC, application period gets 5500 to 6500 applications. We had 400 applications for a program that was promoted a week in advance of shooting the applications. We took 3 of the companies. That’s almost 8%.”
Justin’s outlook on Snapchat as an engagement tool is similar to his content blend: A little work, a little play.
“Generally, having a microphone to people who are interested in entrepreneurship is pretty valuable from an investor perspective,” he says. “It’s like a pipeline. I have people say, Hey, I quit my job and I went to startup bootcamp, because you said it was so important to program. To me, that’s pretty meaningful and impactful.”
But it’s not yet scientific for Justin.
“Snapchat works with authenticity and spontaneity,” he says. “That’s much more of my personality than other social media. I didn’t say Oh, Snapchat is growing, so I should be there. That is the case, but my interest isn’t strategic. The platform resonates with me as a user.”
At HubSpot, Marissa confirms this tenuous balance. When exploring Snapchat, she knew it would need to be a calculated investment, but also realized stiltedness would render it moot.
“One of my biggest concerns in planning Snapchat content is making sure that it feels organic and true to the platform,” she says. “We don’t ever want to take a webinar and squish it into Snapchat’s functionality. That’s not what we’re trying to do.”
Although HubSpot’s content is planned in advance, the social media team learned that heavily scripted content is bad for engagement. Users don’t tune in.
On the contrary, HubSpot sees high engagement with stories that allow for more spontaneity. Recently, Marissa did a Billy on the Street-style Snapclass. Approaching strangers, HubSpot quizzed them point-blank on “What is a CRO?”
And viewers responded positively, with screenshots and written feedback. Marissa attributes the success to mapping out the story’s educational value while keeping the content organic. After all, spontaneity is why the app became hot.
So while it’d be interesting to have a little more insight into user behavior, part of Snapchat’s secret formula is in its denial. By allowing more data, Snapchat could break the creativity it’s inspired with constraint.
So goes the opinion of venture capitalist and marketing antihero Jay Acunzo. But he summarizes it much more effectively:
“Eventually some company is going to figure out how to game that system, and ruin it, and then write about how they ruined it and everyone else will follow.”
The fun plan
Does anyone remember fun?
That’s what Jay wants to know. Because although he’s been in the industry long enough to know that optimizers come in, ruin things, and write about them, he’s having a lot of fun on Snapchat.
Jay uses his channel to goof off, wax poetic on creativity, and give a behind-the-scenes look at his podcast, Unthinkable. But even though he carries a message, he doesn’t want to be excessively results-driven.
“Oh, it was a strategic investment decision of my time and I was hoping to get a return on investment — No,” Jay says. “Stop thinking like a robot for a second and start thinking like a human being. I’m building a project that I want people to be incredibly engaged with, and stuck to, and excited about.”
His premise is if he approaches Snapchat as a craft, the end results will take care of themselves. And the app, in its closed-circuit creation, is designed to promote this ethos. It provokes you to offer something real, instead of a meticulously-calculated experience.
And Jay confesses he doesn’t necessarily care about more views. He’s angling to get the 150 people who always view his story, to repeatedly do so, and love every ten-second segment.
“There is no robotic brand logo talking to you on Snapchat unless you’re dressing up in a Furry costume,” Jay says. “It’s never been easier to reach people. It’s also never been harder to resonate with people. Start with the resonance part.”
A deeper kind of engagement
There is a way in which Snapchat enables a unique opportunity for deep engagement by eliminating broader metrics.
Very little data is out there on how people engage with Snap content. A view is not worth as much as it is elsewhere, because there’s no data to determine whether it was just a tap, soundless and disinterested, or the real thing. As of today, there is no passive affirmation, a la “Like.”
Instead, Snapchat reports only deep engagement to its users. A screenshot or a typed response requires more energy from the passive user economy, but if a piece of content really moves a user, she’ll make noise about it.
“When someone takes the time to reach out to us, we know we hit it out of the park,” Marissa says.
Or, if you tell users to screenshot a bitly link and they follow-through to the other side, you know you’ve not only engaged, but provided a meaningful experience. If you tell users to vote on atopic they’d like to hear about it, and people actually submit Snap ballots, they’re genuinely invested in what comes next.
Aside from that, it’s somewhat winner-take-all. If you don’t hit that threshold, it’s nothing but dead air.
“Distracted, passive behavior on Snapchat is completely disabled by the way they built their product,” Jay says. “There’s a depth available on Snapchat that is not available anywhere else.”
The only way to reach that depth is through capitalizing on the platform’s constraints: Low budget, high creativity; minimal scripting, high spontaneity.
Whatever hacks Snappers find to justify their engagement in the meantime is up to them. You can lose a lot of hours transplanting other people’s face onto yours (I know I have). But currently, the platform’s ripe for fun because of its limits. Only really strong content can register as effective, and so only really strong content can survive.
Engagement numbers on Snapchat will generally be lower than other channels. But when you do connect, it’ll have a higher impact. It’ll probably verge closer to fun than any other channel you’re using. And, as Jay puts it, “Fun begets more fun.”
“I actually would love if somebody were reading this post and thinking, This guy is a total crack because he’s ignoring data,” Jay says. “Because the people on the opposite end of that spectrum are also responding but they’re shouting to the rooftops, Yes! I agree. I don’t want the middle ground.”
To that point, it seems as though Snapchat’s ROI isn’t yet in a sizable fraction of that 150 million. Instead, it’s in the meaningful relationships you can cultivate in these early days. If the media app continues to grow, the spoils will be for those who engaged with users deeply, early.
And it seems like it’s here to stay:
