Why news publishers are investing in Facebook A/B testing
By Simon Owens
Dan Acton remembers the exact moment when he became sold on A/B testing Facebook content.
Acton is the social media manager for DramaFever, a video streaming company owned by Warner Bros that uses a Hulu-like model to license and stream Korean and other Asian TV content for an English-speaking audience. Many (though not all) of these shows are romantic comedies. To promote the shows DramaFever licenses, Acton and his team produce short videos they then upload to Facebook. “Sometimes it’ll be clips from shows, or trailers and teasers for upcoming episodes.” he told me. “We also get a lot of original content produced from Korea, like shout outs from the actors or behind-the-scenes footage that nobody else has.”
Earlier this year, DramaFever began licensing a new show, called My Amazing Boyfriend, about an alien being with superpowers who falls in love with the show’s female lead. “It was a show that we were really hopeful for,” said Acton. “But it was a different show for us, a new genre, a new kind of thing for our audience. So we weren’t exactly sure what was going to be the marketing angle for our audience.” Not long before DramaFever began broadcasting My Amazing Boyfriend, the company had started using a tool called Naytev that allows publishers to A/B test and optimize Facebook content, and Acton decided to use the tool when posting a video clip from the show. “The clip was a romantic scene, and it included a kiss,” he recalled. Facebook allows you to pick a thumbnail image to appear with any video, and in the past DramaFever’s social team had always used a still image of a kiss as the thumbnail. But Naytev allowed Acton to test out several different thumbnails. “In the same scene there was this shot of a woman lying in the street. It looks romantic, but it was a different type of image than what we’d normally use. But when we tested it, it got — I don’t know what it was about it, it might have been because it was so different than what we normally post — but it got 10 times the reach compared to the other ones using the kiss. And if we didn’t have Naytev we would have never have known to use it.”
I had been introduced to Acton by Patrick Costello, one of Naytev’s cofounders. Though the company is now headquartered in the Bay Area, it was launched in Washington, DC. “We began as a cause-based marketing company,” he told me in a phone interview. “We worked with nonprofits to boost their visibility. In the course of doing that work, we discovered that message testing of content on Facebook was a very impactful way to drive a great lift in visibility.” Eventually, Naytev’s team realized the technology’s appeal extended far beyond the nonprofit world and would be useful for any business that relied on Facebook for marketing.
In its initial incarnation, Naytev allowed publishers to use Facebook’s Open Graph API to test different messages. When you copy and paste a link to an article into your Facebook status bar, Facebook automatically consults that article page’s Open Graph to determine the headline, thumbnail image, and descriptive text that it then shows in the newsfeed. So a publisher would load up to three different messages into Naytev, and whenever a reader would then go and share the article to their Facebook account, Naytev would randomly assign them one of the messages being tested. The tool, using machine learning, would measure which version received the highest engagement, and, after it had collected sufficient data, would choose a “winner” that would then become the default message.
But this version of the Naytev product only worked well when a piece of content was organically shared by a large number of users on Facebook. If an article was only shared, say, five times, then the results wouldn’t be very conclusive. “So we upgraded the system to ensure you get results for every piece of content every time you run a test and actually expanded the number of messages you can test,” said Costello. “And the way we did that was by expanding it to dark testing on Facebook.”
Dark posts on Facebook are sponsored ad posts that aren’t published to your page’s feed. Let’s say you’re a popular pet food brand and you want to promote a special kind of food for diabetic dogs. You know the vast majority of those who like your page don’t have diabetic dogs, and you also want to reach pet owners with diabetic dogs who don’t yet like your page. So you run a dark sponsored post that targets that specific subgroup, and those who go directly to your Facebook page won’t see that post in your feed.
After Naytev updated its tool, publishers could log in and create multiple versions of a post by changing three variables: the status bar text above an embedded link, the headline within the link itself, and the thumbnail image. You then run all these variations as ads targeted to Facebook users similar to those following your page and see which message performs best. Once you’ve collected sufficient data, you can post the winning message to your page. “We’ve optimized the testing structure and we’re able to help people get great results for as little as a $30 budget,” said Costello. “In the past people have had to spend orders of magnitudes more than that and had to wait significantly longer to get results. We’ve optimized the structure across the board to make it so it’s not only something that’s less expensive but also fast running and very easy to scale.”
A/B testing as a practice is certainly not new. Major web companies ranging from Google to Netflix use it constantly to tweak their products. President Obama’s campaign famously tested out multiple subject lines to great effect for its email marketing. And any Madison Avenue ad buyer has been optimizing ads on Facebook for years. But Naytev’s claim is that it makes this process easier and quicker. It’s also been widely adopted by news publishers that traditionally rely more on organic Facebook sharing instead of paid distribution. The news organizations I spoke to for this piece will typically only spend a small amount of money to optimize a message and then will rely on their organic followings to promulgate it.
[LIKE THIS ARTICLE SO FAR? THEN YOU’LL REALLY WANT TO SIGN UP FOR MY NEWSLETTER. IT’S DELIVERED ONCE A WEEK AND PACKED WITH MY TECH AND MEDIA ANALYSIS, STUFF YOU WON’T FIND ANYWHERE ELSE ON THE WEB. SUBSCRIBE OVER HERE]
In 2014, the Naytev team got accepted into and attended the Y Combinator bootcamp. Upon completing the program, it secured enough funding to move its entire team from DC to the Bay Area and focused on scaling its client base. It also continued to expand its offerings to include A/B testing for native Facebook videos, which have grown increasingly prominent on the platform.
Fusion, the Millennial-focused media company owned by Univision, began using Naytev in January. Prior to joining the company as its executive director of audience engagement, Alton Phillips had similar roles at OZY and NBC. “At both of those places, A/B testing dark posts with Facebook had been part of our bread and butter for years,” he told me. “But the great thing about Naytev is that it really reduces a lot of friction in the process and enabled us to take that work to scale, whereas before we had teams having to manually create hundreds of ads and it was super laborious. Now, the level of effort is much more manageable.”
Fusion’s senior growth editor Gennady Kolker was drawn to Naytev because it allowed his team to optimize content that wouldn’t typically be considered clickbait. “Particularly for stories in topic areas where we historically found it a challenge connecting with our audience,” he said. “Green issues, gun control — those can be tricky to connect with an audience. That’s where we find testing provides some of the biggest dividends for us.” Since implementing the tool, Fusion has seen a 230 percent lift in click-through rate for its Facebook posts and a 151 percent increase in post engagement. Its videos have experienced a 93 percent increase in views of at least 10 seconds.
DramaFever’s Dan Acton leveraged Naytev to help him establish a voice for the Facebook page. “I’ll use copy that is kind of in the voice of our young audience,” he said. “Instead of saying, ‘Watch this on DramaFever’ and posting a link, I’ll post a kind of natural reaction to the clip in the status update and then we’ll follow up in the comments with, ‘Hey, here’s where to watch it.’” For a recent episode clip of the Korean television series Uncontrollably Fond, for instance, DramaFever simply posted “oh my gosh!!!” as the status update. “The cool thing is that you can try stuff like that and see that it’s working,” he said. “There are tons of social media optimization tools that I’ve tried, and this is the one I’ve been happiest with, because it makes the decision making a lot clearer.”
I asked Acton to walk me through a hypothetical scenario in which he wants to test out a new video clip. “I’ll get the clip from my video team,” he said. “They’ll make a special thumbnail in Photoshop, but when I upload the video to Naytev, it autogenerates options for thumbnails, so I’ll upload the ones we made and then check a couple of the options. I’ll use three thumbnail options, and then for the status copy I’ll do three or four variations. Sometimes I’ll just ask, ‘Who’s this actor?’ because our fans love identifying people. I’ll do 12 variations, I’ll run the ad, and I usually give it 24 hours. Naytev shows a little meter of how confident it is in its results. We set the budget at $30 per test, and once it’s very confident of the results, we place more money behind the winner.”
Costello said that Naytev clients will typically see between two and three times more clicks on their posts as well as a significant uptick in overall audience growth. Given that Facebook isn’t the only social platform in town (though it’s by far the largest), I asked if Naytev’s functionality could be extended to, say, Twitter, which also allows dark posts. “We’re fully focused on Facebook because it drives the overwhelming majority of traffic and engagement for publishes,” he said. “Other networks and channels are still important to focus on but won’t drive nearly as significant returns as Facebook will. That being said, we do allow teams to optimize their content for visibility and distribution on other platforms by allowing them to carry over winning elements from their Facebook tests.” What he means by that is an image that does well in a Facebook test is fairly likely to do well on Twitter.
The ability to squeeze more organic engagement out of posts has become especially important as Facebook has tweaked its newsfeed algorithm in ways that have made it more difficult for publishers to reach consumers. In June, the company announced that page content would be deemphasized at the expense of posts shared by your personal friends and contacts. And Facebook recently indicated that it would identify and punish “clickbait” headlines in the feed. “With the increased pressure to stand out, you have to get your message right every single time,” said Costello. “And as an editor or social media manager, you might be able to come up with a pretty good message, but how do you consistently produce a winning message? We try to eliminate the guesswork about what’s the best-performing version of your message so you get it right every single time.”
***
Did you like this article? Do you want me to create awesome content like this for you? Go here to learn how you can hire me.
Simon Owens is a tech and media journalist living in Washington, DC. Follow him on Twitter, Facebook, or LinkedIn. Email him at simonowens@gmail.com. For a full bio, go here.
Image via AlexisGrant.com
