Developing a data informed product culture
We live in an age where we have vast quantities of data and information, so much so that it can be difficult to filter out the noise. At Monese, we’ve always been data driven as a leadership team, tracking key performance metrics to monitor business growth, revenue, retention etc. Our Infrastructure team have also used monitoring from the start to track system performance and health. However, it took us a while to start building a monitoring mentality to help inform the product decisions we make.
There’s a fantastic quote in Patty McCord’s new book on the culture at Netflix, where Patty explains that as a company, they are ‘data informed’ versus ‘data driven’. That’s the philosophy we’re trying to adopt at Monese, using data as one input to help us improve our decision making as opposed to data being the only input that drives our decision making. This post looks at five ways we use data insights to inform our everyday decision making and prioritisation processes at Monese.
Customer Support Data
As a business that deals with customers’ money, we’re fortunate in that we get to talk to our customers every day. We have a fantastic support team who works round the clock to improve the overall experience for our customers. Many tech companies have less personal interactions with customers, often only by email, user testing sessions or via app store reviews. We’re chatting to our customers on the phone, over video call, instant messaging or face to face every week of the year. This puts us in a privileged position where we’re able to leverage the customer data gained from hundreds of daily interactions to improve the product. We also have a Slack channel where engineers and members of the product team interact directly with our support agents, often troubleshooting problems in real time or investigating and triaging more complex issues for prioritisation. We look at the top issues customers contact us about and then use this information to place items on our roadmap. This helps ensure we’re constantly delivering value for customers and solving issues with our core product. We involve our Head of Customer Loyalty in product discovery sessions and they actively play a part in helping shape our roadmap. The Monese app supports 9 different languages (at the time of writing) and we provide support for customers in each of these languages. This ensures we’re building a product that works for each language, as opposed to a UK-centric view of the world where decisions about additional languages are an afterthought. We’re by no means perfect at this right now, but through the insights our support team is able to provide, we’re able to better plan features with multiple languages in mind.
A/B Testing
Most product teams are now A/B testing. We lived without it for a while - as with any startup we had to ruthlessly focus our time on a handful of key projects. This meant A/B testing got put on the back burner. Instead of building out our own A/B testing framework, we decided to partner with Taplytics to get us up and running quickly and enable us to start experimenting on both iOS and Android. A key focus was to improve the customer experience from initial install to account opening. As an FCA regulated company that provides both UK current accounts and European IBAN accounts, we have to ensure we’re verifying our customers and performing full due diligence checks. From a UX perspective, this presents friction points when trying to onboard new customers — they need to sign up, have one or two forms of ID ready, and record a video selfie before opening an account with us. Our reporting systems identified funnel conversion opportunities, but we didn’t have a quick way of testing changes and measuring the impact on overall sign-up to account opening conversion. Through A/B testing tweaks to the funnel, we were able to make decisions about which variants to implement, iterate on or scrap without endless app store updates and regression testing.
Anomaly Detection
Often anomaly detection is used in dev ops and network security to alert an analyst that something may be wrong with the system. We decided to take the concept and apply it to our onboarding data, so that we could identify areas of customer drop-off. As an example, our BI system showed that we were experiencing a reduction in the number of accounts being opened later in the evening on Android versus the daytime. Through having this data available, we were able to follow up and investigate potential causes of this drop-off. We realised that poorer quality cameras on Android devices, combined with asking customers to take a portrait image on Android, resulted in poorer conversion rates. Customers got stuck, unable to successfully scan their ID. They usually progressed from this state via customer support contact. With a reduced number of staff in the evenings, this led to a drop-off in account openings. We were able to solve the issue largely through technical improvements and some additional staffing in the evenings. We saw a dramatic rise in evening account openings. Without having the reporting in place to alert us to a glitch in the matrix, we wouldn’t have started our investigation into the causes of the issue. Additionally, we can set up anomaly events for the number of ID scans or the number of customers failing ID scans per hour versus the baseline for that hour over x amount of days. This can hint to something having gone wrong with one of our partners, our own systems or with one of our marketing platforms. Having fast alerts means that we can act swiftly and ensure a smoother onboarding experience for customers.
User Testing and Customer Surveys
When working on a new project, we first start off with a design sprint or prototype a first version of a feature. We then schedule user testing sessions where we invite 5 customers to try out the feature and observe their interactions with the product. From the common patterns that emerge in testing, we then explore this further through a broader customer survey. The data we receive from both qualitative and quantitative testing better informs our product design choices and helps validate, invalidate or pivot on our initial assumptions. The key here is that early testing can provide the team with confidence that we’re heading in the right direction, or that we’re potentially building something that customers don’t want. Many other decision points go into shipping a feature, including competitor analysis, executive feedback, internal feedback and sometimes gut feeling (dread!). We don’t solely rely on the data, but it certainly helps provide valuable insight for iteration before committing to any extensive development work.
Business Intelligence Insights — Behaviour Nudges
We use business intelligence tools to mine our data for short and long term retention insights. We break down cohorts of customers by sign-up month, by the type of account they open (Euro or UK) and by the pricing plan they select (Starter or Plus). The cohort-based data can look at the correlation between performing a certain action, say making a Monese to Monese payment within the first 30 days of account opening, and the likelihood that you will remain an active customer 90 or 120 days later. Certain behaviours show a strong correlation with long term retention and thus increase Customer Lifetime Value (CLTV). We can leverage this data to optimise our on boarding flows to better promote these behaviours. This can be taken a step further, by implementing a machine learning model that can nudge users into positive behaviours over time, thus increasing the likelihood that they remain a customer over the long term. This actually enhances the experience for the customer, as this longer term product education and discovery recognises that each customer is different. What Customer X signing up from Country A may find valuable about the product, could be totally different from Customer Y signing up from Country B. Machine learning can help personalise the experience for customers, making the product more relevant to them.
So that’s a summary of five things we’re doing today to try and build a data informed product culture. I hope you enjoyed the read, and it would be great to get a discussion going below on how you’re using data to drive your business decisions.
This article was written by Tom Barbour, Head of Product @Monese. This is the third post in a weekly recurring series from the Product & Engineering team @Monese.
