How using questions and values to evaluate options helped us tackle the paradox of choice
Read about the Local News Lab’s process of narrowing down to one final data product idea from many
Workshop 4: Zooming in to the ideas that sparked
Building on our previous workshop, in which we looked at the machine learning technology the lab already built for content recommendations to see if it sparked any ideas, we met subsequently to zoom into each of the seven ideas that had surfaced in order to narrow down to three ideas (at this point, these “ideas” are rough sketches of what the product could be, not well-defined with feature requirements). The intention was to give the cohort an opportunity to identify which idea most resonated with them. We prompted our partners to consider which idea might best support their strategic goals, and also, which piqued their interest as an exciting area to explore. They then met in smaller groups to ask — and answer — specific questions about their chosen product idea in order to build out more fulsome pictures of what opportunities, risks, and barriers it might present.
The group self-organized around three product ideas:
- Automated content recommendation platform
- Automated audience data collection and analysis at sign-up / subscription
- Automating the ask (for a donation, a newsletter sign-up or membership)
To help us zoom in to these ideas, we asked and answered some big questions about each (below are just some of the answers we generated):
1. Automated content recommendation platform: A centralized platform where non-technical staff can manipulate content recommendations rules and easily see and interpret results from each deployment
Q: What are some types of recommendations that you would want to test and compare?
- Showing more stories from the topic that is being read
- By author
- By type of stories: Q&A, breaking news, “service” stories, etc.
Q: What types of audiences do you think would engage with each type of recommendations? Who are they? What value would they get from the recommendation? What might those recommendations look like on your site?
- Inline w/ photo for 1st story
- Grid w/ 4 stories, all curated by hand
- Home-page teases: different stories in different layouts, can be made into automated feed instead of curated by hand
Q: What might some of the risks be, or barriers to automating these recommendations?
- For topics: we don’t know what kind of audience comes to us for a particular topic.
- To what extent can our data be “actionable”
- What could we learn about reading habits? Slice by referral to see how it changes behavior?
2. Automated audience data collection and analysis at sign-up / subscription: An automated data intake method when an audience member signs up to a product or subscribes, the centralizes and automatically interprets the data to produce insights that drive strategic improvements
Q: What data points might you gather? And from which sign-up or subscription touch points?
- Email addresses
- ZIP code (can be connected to census)
- Membership & sign-up through newsletters
Q: What questions would you want the data to help you answer?
- Type of content consumed (podcast, article, etc.)
- What info do community members need? (Who are we serving? Who is being left out?)
Q: What might some of the risks be, or barriers to automating analysis of the data you gather?
- Privacy
- Interpretation of data is wrong
3. Automating the ask (for a donation, a newsletter sign-up, a membership): An automated system that can predict the best time to make an ask or CTA that will lead to a conversion
Q: What kind of “conversions” do you offer your readers? Eg newsletter signups, subscription, etc.
- Newsletters, donations, contacting us
- Follow on social (org and also specific people)
Q: How do you typically ask or invite them to convert? What is the language and tone, and how do you offer it?
- Membership is part of drives, campaign
- Big membership asks around major breaking news
Q: When do readers currently encounter these asks, and how do you make sure that they’re seen?
- RSVPing to events
- Sidebar CTA on desktop site, in-line for mobile
Q: How do you think that automation could help improve the quality of timing for these asks, or the volume of asks?
- Save time (producer not needed to manually place)
- Loyal vs new audiences (can the system distinguish and serve accordingly)
- More effective but fewer CTAs
Some key insights and questions emerged in these discussions:
Data privacy is of paramount importance
Consider how we avoid annoying the audience with too much *and* having the product be static “furniture” that doesn’t capture attention
An interest in more effective, but fewer CTAs
Can the system distinguish between loyal and new audiences?
How might the product help us gain more insight into audience characteristics and behavior?
Having a platform to test out different rules would be neat. You can read other case studies, but ultimately that’s not our audience.
Would love to see more reporting led by data that exists about what people engage with. Is a social visitor more likely to do “x”? Is a search visitor more likely to do “y”?
Our hunch was that we would see one emerge as the top choice with some further probing in our next workshop.
Workshop 5: Ideas + Opportunities = Path
With this workshop, we begin to close one of the “diamonds” in our double diamond design process, by narrowing down from three to one final product idea that we will focus on. How did we do it?
We came up with a rating exercise that each newsroom weighed in on via a simple Google form. We gave them a scale of 1–4 to encourage an opinion (there was no neutral ground allowed!), and asked them to rate the three project ideas according to each of the seven goals below from whether they strongly disagreed (1) to strongly agreed (4) how well the project would support each goal for their organization and your audience. These criteria were rooted in feedback from our conversations leading up to this workshop along with the lab’s guiding principles (of serving reader communities, identifying business value, supporting cross-disciplinary collaboration, etc.):
- Strengthen relationship with readers
- Provide a good user experience
- Offer revenue potential or add business value
- Feasible to launch on your site or products
- Learn from the product process or output
- Spark cross-disciplinary collaboration within your organization
- Support or communicate your newsroom values
In the end, the ratings each idea received was frustratingly close! Perhaps a good problem to have (they’re all good, practical and useful product ideas — terrific!) but presented us with a challenge (argh — how do we break the tie?). Thus, the lab team and each partner came up with a weight for each goal, which we then averaged, and our talented data scientist, Duy Nguyen, ran our results again using the weighted rating and we finally arrived at a winner (albeit another frustratingly close race, but at some point a product team has to make a call and clearly all choices here were good, so we were thrilled to be making this big step forward). Our new product idea would be: “Automating the Ask”.
To ensure we were staying on track, we mapped the idea against the project goals we arrived at during our kick-off:
💪 Build an alliance!
🔧 Implement a feature or data product that wouldn’t have been possible on our own!
📚 Learn how we to better use traffic data and analytics to build a loyal community and drive sustainability.
📚 Learn about how to center readers and listen to their feedback when making product, tech and tool choices.
🔁 Exchange ideas about (tech, x-functional collab, using data)
Satisfied the project remained aligned with our goals, we set forth. This point in the process represented the end of another phase of “flaring”, the peak of one diamond, as we set out to define and scope what “Automating the Ask” could be.
About the Local News Lab
The Brown Institute’s Local News Lab is a team of engineers, designers, and data scientists working to build AI-powered, open-source products to help support local newsrooms and their businesses. The team’s work is collaborative, partnering with small- to medium-sized publishers across the country.
While large national news outlets can have groups of data scientists in-house producing insights and products that optimize audience engagement and drive reader revenue, helping ensure their sustainability and survival; local and community-based newsrooms are often left stretching their limited resources in every direction. We aim to help our partner organizations overcome the barriers to mobilizing the best data science to support their business strategies.
We are grateful to the Charina Endowment and the John S. Knight Foundation for supporting our work.
Curious about what we do? Send me an email and get in touch.
