When Time is a Luxury, Validate Product Roadmaps With A/B Testing
Interview with Deepa Subramaniam, Director of Product at Hillary For America.
When we think about influential, innovative product teams, we tend to picture the most modern, tech forward companies — Apple, Amazon and Google particularly come to mind. However, one of the most diverse and skilled tech teams of 2016 belonged to a more traditional cause: Hillary for America.
It’s been just over a year since the conclusion of the last American presidential election, and this has Deepa Subramaniam reflecting on her time as the Director of Product at Hillary for America. Her insightful talk at the First Mark Capital Design Driven NYC event last May highlighted the importance that Deepa and her team placed on experimentation as they created digital products for online fundraising, volunteering, grassroots engagement and election analytics, all in pursuit of putting Hillary Clinton in the White House.
“We tested everything, we had to. We used data-driven testing to identify areas we should spend precious time and money.”
We were thrilled for the opportunity to chat with Deepa about this unique experience of hers and to learn more about the campaign’s data-driven culture.
Taplytics: Can you tell us a little bit more about the Hillary for America team, the products you built and the role that experimentation played in the decisions you made?
The Hillary for America Tech team was made up of 90+ engineers, designers, product managers and user-researches. In 579 days we built well over 55 applications and services. This roughly amounts to shipping 1 new application or service every 10.5 days for 19 months straight!
These 55+ products covered a wide-range of needs: from mobilizing voters, to raising money and getting our candidate’s message out into the world. Testing helped us understand which product ideas were worth investing limited time, energy and manpower in. This was advantageous because it helped our team build confidence that the ideas we were investing in would have the right, measurable impact.
I’m a strong proponent of testing, especially when used to identify new feature areas or to optimize existing impact. Evergreen testing is valuable for organizations to continue to level-up and improve the key performance indicators of their product portfolio. Dates drive urgency, not relevancy of testing.
Do you think it’s better to experiment on general features and flows that all users will experience the same, or on nuances and variations that will be personalized to different user segments?
This comes down to volume. Broad product flows that touch all or many of your users are ripe for testing because it’s more likely you will get enough usage in a reasonable amount of time to have statistically sound results.
For example, A/B tests on our campaign fundraising flow often hit significance in an average of 7–10 days because of the volume of grassroots donors we received; but sometimes this would be as short as 1–3 days during high-volume moments like primaries, debates or townhalls.
Product flows with narrower usage, like those customized to particular segments of users, will take longer to get to significance. Statistical soundness is important for valid testing, so it really comes down to a product team’s ability to estimate how long it will take to get sound results and whether that time is worth waiting for.
When do you find experimentation has its disadvantages?
Testing does come with disadvantages. Some product areas had limited volume of users at particular moments of the campaign lifecycle. This meant that some ideas couldn’t be testing to statistical significance until specific moments during the campaign.
For example, organizing national volunteers hits an inflection point after the Democratic National Convention and onwards to election day. Because of this timing, we couldn’t test certain ideas and had to operate on intuition, domain experts and past data to guide that product roadmap until the right time when testing made sense.
Speaking of intuition, how do you decide what to run tests on — data or intuition?
Both — using data and intuition is critical. By listening to one or the other too much, you can skew your understanding of potential product impact. I also believe in having the right stakeholders in the room to help evaluate both the quantitative and qualitative aspects that go into creating your testing roadmap. Broad stakeholder input improves the quality of your team’s testing roadmap by fostering healthy debate, along with the added bonus of roadmap transparency and (hopefully) buy-in!
Do you approach building products now the same way that you did at Hillary for America? Were there any profound learnings from the campaign that you brought back to your work after?
Yes! Building product on Hillary for America was exhilarating — we worked at an incredible pace building our planned roadmaps while also being reactive to the twists and turns the presidential race went through.
At all my roles since Hillary for America, I’ve brought the same sense of urgency to ideate, build and iterate quickly. Best practices like A/B testing can help guide product roadmaps when time is a luxury.
Time for Change.
Whether or not you are working against a deadline with high stakes like the Hillary for America team, time and money are precious resources that you need to prioritize. Embedding experimentation in your culture and routines early on in the product roadmap can help define bold, impactful ideas that will guide you down the right path.
A big thank you to Deepa for sharing her time and insights with us!
Deepa Subramaniam has spent 15 years leading teams to build products with broad reach and impact. She has been a leader at numerous organizations including Kickstarter, Hillary for America, charity: water and Adobe. She champions data-driven design, achieving high-velocity in fast-paced environments, and building collaboration and joy into company culture. You can follow her on Twitter here.
Click here to watch Deepa’s Building Product for Campaign talk at Design Driven NYC.
Taplytics is a product experimentation solution committed to helping teams build digital products for their users and get the most out of their technology budget. By encouraging companies to execute, listen then iterate, Taplytics helps them validate product decisions with live user data to prove that they are making a positive impact.
