5 Things We Learned While Working With Axios on Product Development at NYU
Co-written with Aldana Vales. Thanks to Axios for letting us learn from them.
During the 2019 spring semester at NYU Studio 20, we partnered with Axios for our Studio II course. Designed to tackle one large project in innovation, the class functioned as a product and audience development lab for the media company.
Axios is known primarily for its newsletters. They have “close to a half-million subscribers”, up to “10 million uniques a month” and made “$25 million in revenue in 2018”, making them one of the hottest media startups in the American market. The task we had from them was simple: to extend its signature Smart Brevity format to both new platforms and new audiences, with scalable products that could use existing content with little investment from the company.
This post will talk about what we learned and the Agile process we used to develop prototypes of 3 products that could help Axios get a sense of how to expand its presence to other platforms and audiences.
1.- Make data-based decisions
Right after we started working, Axios gave us access to a big chunk of data from their products. Open and click-through rate, podcast downloads, uniques from their website, banners performance, and social media. However, although we had all that information, we just went through their existing products and made conclusions based on our intuition, and preexisting ideas. The result was a presentation full of proposals that anyone could have made. No data, no KPIs, no focus on audiences needs for the products we pitched.
After failing miserably, we took a step back and went into the pool of data we had in our hands. The following is what we found:
Media companies like Axios, QZ or The Information are trying to reach aspirational leaders. That is young professionals or people who are about to graduate from college and become part of those young professionals. They are on Linkedin, but the chances are that they are also on Instagram, because 71% of Instagram users are under the age of 35.
Our initial question was this: What does Smart Brevity look like on Instagram? How do you translate newsletters to posts and stories? First, we wanted to know how they were doing, and we used CrowdTangle data for that. After an analysis, we set two goals for this project: increasing engagement and converting followers into subscribers.
It’s not big news that audio, specifically podcast, is and will be a booming business in the US for years to come as it has room to grow both in revenue and users. Also, some of us knew about Danish media company Zetland and their successful experiment that made the organization a case study: they offer 100% of their text content into audio, available for all paying members. Since Axios content is already packaged to be commuter-friendly, we thought that we could use the “Smart Brevity” ethos of Axios and translate it into audio: Audio Newsletters in podcast format to be distributed through all major podcast platforms.
Axios already has an audio product. Pro Rata Podcast. This meant we had data to dig into audio consumption habits from Axios’ users. We went through the most successful episodes, their subjects, the day and time that people listened and crossed several parameters. After auditing all this, we finally pitched the Audio Newsletter project. This time, we had data that backed us and KPIs that could measure our success.
2.- Mind the newsroom workflow
We had a pitch based on data. Hence we thought that it was robust enough not to fail. Until it did. Reality slapped us on our faces.
The workflow of the newsroom made it impossible to do our Audio Newsletter project with neither of the two daily products we pitched. We just needed the writers to be available to record audio right after sending the newsletters, but it was either the time of the day that the email were written or the time availability of the authors that made our ideas impossible. Our plans were down. We had to come up with new ones.
We went back to the data and got an alternative yet better plan. Sara Fischer’s Media Trends newsletter. It’s a weekly product, meaning that we had more time to work with the writer, and the subject was technology-related, one of the findings we got after immersing ourselves in Axios’ data.
3.- Face your ideas with those of the users — a shock of reality.
We were thrilled. We had two ideas that had been vetted by data and newsroom workflow standards. Now we needed to research on audiences to develop our products as close to their needs as possible. For this, we gathered a sample of users, asked them questions from a news, IG and audio consumption habits questionnaire we made and then got users to play with:
- A fake Axios profile we created. The results were what we expected based on the data we had: users liked less text, minimal image templates and didn’t like data-viz as much as images or quotes on image cards.
- Four prototypes of an Audio Newsletter: it was a voice reading the newsletter. It had some audio effects, and the host tried to make the reading as friendly as possible. Here is where we failed again.
None of the prototypes worked. The takeaways we got suggested that users wanted something way different from what we developed. We learned the hard way that what we think is the right product, might not be what audiences like or need.
At least by then we had feedback on what users disliked from our prototypes and, thanks to the questionnaire, we had identified their pain points when consuming news. We could work from there.
4.- Meanwhile, something similar was happening with IG
We had some ideas: weekly specials, specific content referring to newsletters. It was a set of 12 potential ideas for Instagram Stories that we came up with after analyzing young audiences’ behavior on IG and data from Axios’ newsletters. Two of those ideas made the cut for live testing on Axios real account on Instagram: One Big Thing, a special for a very relevant topic explained; The Big Picture, a quiz-like series with information from the newsletters. Both of them represented the spirit of Axios’ signature “Smart Brevity”.
The prototypes weren’t very successful. Because we wanted to focus on spreading the word of Axios account, we created a card to be screenshotted and shared. We had some interactions, but we didn’t have as many as we expected. We might have asked too much from users.
However, we noticed one clear thing: users did respond to Instagram Stories’ native features like the emoji slider or the question tool. With that in mind, we re-shaped the format of the two series. It was a big win: users engaged much more with the account when we asked little from them.
An example of this is that halfway through our work, Instagram launched the quiz sticker. Until then, we had been working with the poll tool, so we thought that the Quiz Sticker was ideal for our series. The first time we used it, we included four possible answers for each question, but It didn’t work as well as the poll tool. We later understood that reading four answers took too much time, and that was discouraging our users. The next time, we used two answers, and the completion rate of stories improved.
5.- Keep testing. Keep iterating
After the first round of testing/interviews and the live testing on Instagram, we had identified user needs and prototyped solutions to solve them.
However, one of the most recurring pain points for those who didn’t consume Axios was the challenge to understand a specific news cycle without prior information.
More testing: Three weeks after the first round of testing, we held a new round — this time with a sample of users twice as big. Half of them were Axios users; the other half consumed news but not Axios. To make the testing as close to reality as possible, we invited Axios’ Sara Fischer to record the prototype of Media Trends Audio Newsletter.
As expected, the prototype made sense for the audience. Later, we ran a new round of testing with Media Trends subscribers to make sure that our ideas made sense for those who, out of all people, would be the early adopters.
After 4 rounds of testing, we were ready to deliver a MVP. We even took Axios’ visual pack and made the logo of our prototype.
During the 14 weeks that we worked with Axios, we interviewed more than 20 people, conducted 4 rounds of interviews and testing, designed 7 audio prototypes and ran 3 live testing on Instagram Stories. From this experience, we were able to:
- Develop final prototypes for an audio version of the newsletters. This included working directly with Media Trends reporter, Sara Fischer. Axios used this prototype to further explore its own audio strategy.
- Improve Axios’ interaction rate on Instagram by 13% via new templates.
- Gather information on Axios’ audience that can be used by the company in other projects.
After all this process, we came to the conclusion that when developing new products for news organizations, decisions have to come from a) available data, and b) insights gathered through user interviews and testing. Media companies need to identify consumption habits and pain points from their audiences to build solutions to solve them. If not, products will only be appealing for those who work in the creation of them. This is what happened to us with the first prototypes.
However, when we switched to an audience-based approach, we were not only able to improve interaction rates on an existing product like Axios’ Instagram, but also to design audio solutions that appealed to those who tried our prototypes.