Closing a product discovery process with definition and hypotheses to test
How the Local News Lab defined our product and generated some hypotheses for how an automated “ask” might increase audience engagement
With the product idea identified coming out of Workshops 4 and 5, the lab dedicated our final two product discovery workshops on better understanding our partners’ current calls-to-action, in what ways they could be better, and finally, on generating a number of hypotheses we could use data science to test.
Workshop 6: Digging into “The Ask”
After our customary reflections presentation (where we presented findings and insights from the previous session), we spent this workshop asking our partners deeper questions about their current newsletter and donation “asks” (aka calls-to-action or CTAs); questions ranging from technical (where is it appearing and how it is made?) to data-oriented (what data do you have that tells you whether it is working well or not?) to more design/UX (what do you think could make them better?) and even sort of existential (what does better mean to you and how might you measure it?).
The commonalities we observed across all our partners’ sites were as follows:
- Most current placements are permanently displayed
- There are more types of asks for donations than newsletters, but just a few
- We imagined a similar number and type of “automation levers”, centered on:
- Engagement: Visits, No. of stories, Scroll depth, dwell time
- Status: Donated/newsletter subscriber status
- Past performance: Types of articles that have converted people in the past - There’s an interest in looking back at past data to improve future asks
- UX matters, including transparency, tact; being non-extractive and not distracting
We also took the opportunity to review our guiding insights and questions from Workshop 4, and highlighted three particularly relevant points:
- An interest in more effective, but fewer CTAs
- Can the system distinguish between loyal and new audiences?
- How might the product help us gain more insight into audience characteristics and behavior?
Important points that we made a concerted effort to consider as we shifted to the next phase of our work: to devise some hypotheses we can test that might help us determine the effectiveness and utility of our product.
Workshop 7: Generating hypotheses to test
Taking stock of all the “ask placements” identified in Workshop 6, we brainstormed questions and hypotheses we might be able to use to gauge our product’s benefit using this template:
If {this} happens, Then {this will happen}, Because {this reason}.
For example: “If someone reads three stories in one session, they will donate if we ask, because they really appreciate the content.”
We divided these up between newsletters and donations/memberships. In total, we generated 21 hypotheses! All were based on a combination of the expertise of our partners and their knowledge and understanding of their audience analytics.
The lab then conducted a prioritization exercise which was then shared with the cohort (because listen, sometimes consensus is critical, but other times, in the interest of expediency, someone or a small group can help shortcut to the essentials). We used these criteria to narrow down the list to 6 hypotheses:
- Technical feasibility
- Interesting / exciting
- Aligned with our values
- New or fresh
- Applicable to multiple newsrooms
- Applicable across newsletter and donations/membership
- Potential to learn / build capacity
- Has clear metrics for success
These enabled us to arrive at the following as the top six hypotheses (each applies to both newsletters and donations/membership):
- If someone reaches the bottom of a story many times, they’re more likely to subscribe to newsletters.
- If someone reads an increasing number of articles over a time period, or some other engagement that increases over time, they are becoming more loyal and more likely to subscribe to newsletters or donate.
- A user would be more likely to subscribe to newsletters or donate if they’re reading an explainer/solutions/community asset piece because it might provide an answer for them and might want more, or want to give back to the org.
- Someone may be more likely to donate if they are reading about a topic that is trending (more generally, nationally, social, etc.) because it would mean the reader is making an explicit choice to come to our website.
- Someone who makes a direct visit to the homepage may be more likely to donate because they are very intentionally coming to our site (bookmark, etc.)
- If someone visits an “About us/Support us” page they may be more likely to donate or subscribe to a newsletter because they’re curious about the mission.
Finally, we did an assessment of scope made two decisions: first, to leave hypotheses #3 and #4 out of scope for now — while having high potential, we noted they also will involve a high level of effort (that may be beyond the scope of this project) and will come back to these if time permits. Second, we decided to focus on the newsletter sign-up first as the data sources and technical platforms were more straightforward, and to come back to donations (which are more complex technically) time permitting.
And with that, our product discovery process wrapped up. The next phase will consist of hypothesis testing (using both exploratory data analysis and machine learning modeling approaches) and more specific feature requirements and technical planning leading us into our build.
About the Local News Lab
The Brown Institute’s Local News Lab is a team of engineers, designers, and data scientists working to build AI-powered, open-source products to help support local newsrooms and their businesses. The team’s work is collaborative, partnering with small- to medium-sized publishers across the country.
While large national news outlets can have groups of data scientists in-house producing insights and products that optimize audience engagement and drive reader revenue, helping ensure their sustainability and survival; local and community-based newsrooms are often left stretching their limited resources in every direction. We aim to help our partner organizations overcome the barriers to mobilizing the best data science to support their business strategies.
We are grateful to the Charina Endowment and the John S. Knight Foundation for supporting our work.
Curious about what we do? Send me an email and get in touch.
