Understanding audience and data as the foundation for newsroom product discovery

Read about the Local News Lab’s first three out of seven product discovery workshops along the path to our cohort-based data product

Photo by Reed Mok on Unsplash

Following our formal project kick-off meeting, we started our product discovery process which spanned seven workshops altogether. However, before digging into the meat of each workshop, we led the group through our reflections from the previous session. Establishing this habit from the start, recapping the high level discussion points and insights from the workshop prior after a break of up to a few weeks, gave the process both a sense of continuity and momentum.

It is worth noting that the lab uses the double diamond product design model (which you can read up on here). This is a process through which we had periods of “flaring” when we thought expansively, generating new ideas and potential problem spaces our project could address, and times of “focussing” during which we distilled those new ideas along with workshop insights and converged our thinking into a narrower realm of possibilities, and eventually a single product.

Double Diamond Design Process phases via Digi-ark

Our product discovery spanned seven workshops over twelve weeks. Overviews of the first three workshops are outlined below and illustrate how we created some common understanding: of data (in general), our partners’ audiences, and the existing work produced by the lab (to explore potential new product extensions). These sessions represent how we ensured everyone involved in the project had a shared language around these concepts which set the stage to “flare” and open up to the possible directions the project could take.

Workshop 1: Audience + Data: What do we know?

The goal of this first workshop was to answer three main questions with our newsroom partners:

1. Who is the primary audience for your publication?
2. How do you get data about who they are?
3. What are the top three ways you know your audience is engaging with your publication?

In answering these questions, several themes emerged:

• Each partner could identify multiple audiences
• Some had an audience they didn’t necessarily expect
• Lots of different audience data tools are available, but many have limitations. Ideally the tools could talk to one another, but do not, which makes it a challenge to analyze

Each partner had distinct audiences they targeted and some of these target audiences were different from their actual audience, a quandary that is not unique to these specific newsrooms (how do we retain our loyal readers while attracting those we feel we can serve but whom we have not yet converted?).

By the end of this workshop we had identified a dozen ways our partners measure audience engagement ranging from digital analytics tools to in-person conversations. But three measures rose to the top as being critical signals of engagement and feedback these small newsrooms get from their communities: social media engagement; newsletter and email engagement; and surveys and direct conversations. These became focal points for further investigation as we deepened our product discovery process.

Finally, we surfaced some specific challenges and made note of them, in case they became relevant later on:

• Reaching an ideologically diverse audience
• The limitations of weighting feedback from survey takers
• Deciding whether to adapt audience strategy to a tangibly younger audience
• Paying consistent attention to audiences, beyond pledge drives and campaigns
• Taking care of and serving communities’ needs while not being extractive
• Gauging the usefulness of tracking various entry points to content

Workshop 2: Audience (where are we at + how can we do better)

As always, we opened with a quick recap of reflections from the previous session.

For this workshop, we dove deeper into the top ways in which our partners said they know their audiences are engaging with their publication, which are:

• Newsletter engagement
• Social media engagement
• Feedback from surveys and conversations with readers

Through group discussion, we distilled the metrics used, the process of analysis, and identified some gaps and potential areas of improvement in the way we understand audiences and what their behavior can tell us about our publications.

The main metrics in these areas that our partners pay attention to are:

Newsletter engagement data:
Net subscribers, growth rate, unsubscribes, clickthrough rate, number and quality of direct replies, open rate, bounce rate, subscriber cadence

Social media engagement data:
Net followers, overall and by platform (Fb, Insta, Twitter), growth rate, unfollows, comments, re/tweets, referrals to the website, # and quality of replies to posts, shares, reach/popularity of posts

The ways in which our partners go about analyzing these data range from manual analysis (they do a lot of it — including sentiment analysis, responses to call-outs, observing what is amplified on different platforms and by whom), to habitual checking of quantitative analytics dashboards and team-based qualitative analysis of data captured through surveys and direct feedback.

We put our heads together to come up with ideas for improvement. Given our project’s purpose is to be of service to our partners and the journalism community at large, we continually asked ourselves how we might use computation and automation to improve upon both audience engagement and user experience as well as how newsrooms might refine their processes and better understand their data. It is clear, through our work to date, that small newsrooms crave both deeper proficiency and efficiency in their work with audience engagement data in addition to the raw necessity of understanding audience engagement overall that is so critical to their sustainability. Some of the questions and ideas we surfaced as we concluded this session were:

• How might we measure the impact of content and visual design updates?
• How might we best identify which topics resonate most with our audience and how topical engagement might differ across social and web platforms?
• How might we scale qualitative analyses?
• How might we connect data across social, web and newsletter platforms for more precise engagement data in order to optimize content or user experience?

With the foundations of how our partners collected and analyzed their audience data, and these questions in mind, we proceeded to our next session.

Workshop 3: Existing Recommendation Tech + idea sparks

For this session, we looked at the technology the lab has already built that supports machine learning (ML) driven content recommendations. The technology can be broken down into its three component parts (we have written about this work previously):

1. Gather (data collection)
2. Process (analyze data + using ML)
3. Serve (API + audience-facing recommendations)

Then we talked as a group about what, if any, ideas were sparked by knowing that we have this technology available to us. We discussed what opportunities this opens up to help us achieve our goal of building a computational data product that supports newsroom sustainability for our cohort partners.

What follows are the ideas that sparked for us:

• Social media engagement analysis
• Gathering and analyzing data from social media (comments, conversations, observations) that provide insights into how to better listen to and serve your audience (bonus: avoids some privacy issues, given the public nature of social media)
• Content recommendation tool
• Creating a content recommendation dashboard that is easily manipulatable by non-technical folks to tweak, test and measure the effectiveness of different kinds of recommendations
• Deep dive on existing content recommendation data & impact
• Do a deep dive on the way that the existing recommendation system affects overall traffic — does it increase traffic to already popular stories, or add/redistribute traffic across a broader array of content. Embedded in this: explore the value of recommendation to a reader, which is inherently related to sustainability.
• Automated data collection at sign-up / subscription
• Automated data collection at the time of sign-up or subscription that can be used to inform sustainability strategy (i.e. race, ethnicity, where they live) to help newsrooms know which community residents they are serving (bonus: census tie-in, given in-house expertise at the Brown Institute!)
• Multi-format content recommendations
• Build a system to recommend articles, as well as other pieces of content that serve audiences, including links to primary/source materials, community events, “scooplets”, newsletter content, maps / tools, etc.
• Develop new guidelines for local/community content recommendations
• Develop an entirely new set of questions that a newsroom may ask themselves when deciding to recommend a piece of content to an audience member. What are the consequences of a recommendation? What are the values behind them? How might we automate and scale those determinations?
• Automating the ask (for a donation, a newsletter sign-up, a membership)
• An automated system that can predict the best time to make an ask or CTA that will lead to a conversion

These ideas formed the basis for the next phase of our product discovery process in which we zoom in, close the first of the “double diamond” and embark into the second diamond in which we hone in on our project definition.

About the Local News Lab

The Brown Institute’s Local News Lab is a team of engineers, designers, and data scientists working to build AI-powered, open-source products to help support local newsrooms and their businesses. The team’s work is collaborative, partnering with small- to medium-sized publishers across the country.

While large national news outlets can have groups of data scientists in-house producing insights and products that optimize audience engagement and drive reader revenue, helping ensure their sustainability and survival; local and community-based newsrooms are often left stretching their limited resources in every direction. We aim to help our partner organizations overcome the barriers to mobilizing the best data science to support their business strategies.

We are grateful to the Charina Endowment and the John S. Knight Foundation for supporting our work.

Curious about what we do? Send me an email and get in touch.