Delivering value through continuous discovery, delivery and deployment
Today’s continuous-deployment systems are transforming the way tech operations work, and not just for software teams. Here, User Experience (UX) Researcher Marley Dizney Swanson explains how CI/CD platforms can be coupled with UX insights to create new customer experiences that deliver in record time.
Do our customers want what we’re building? Do our stakeholders like what we’re building? Do our developers have the information they need to build it?
There is a triangle of needs within any tech company: customers, stakeholders and developers. At the best of times the three perform an elegant waltz of checks and balances. But getting the steps right requires strong processes and solid research.
I’m Marley, the Senior User Experience (UX) Researcher at MPB. I’m on a mission to keep customers at the heart of what we do.
You may be wondering, “How can I deliver value to my customers?” The MPB answer: through continuous discovery, integration and deployment.
Since we upgraded our platform to a continuous-development continuous-integration (CI/CD) system, with twice-daily updates to the codebase and automated monitoring, we have been able to turn user feedback and testing into real-world features faster than ever before.
This can be done before any code is written, during development, or after shipment. Let’s break down how we do it.
Before any code is written
This is where UX research comes in. We use a continuous discovery strategy in which we talk to users in small, frequent sessions throughout the product development life cycle. We interview two to five customers every week to identify areas for improvement.
Two kinds of research help us discover these nuggets of knowledge: generative and evaluative.
In generative research, we ask our customers broad questions about pain points and moments of joy in their transactions with MPB. Evaluative research gathers customer feedback on existing ideas, designs and prototypes.
Putting designs in front of customers can save us days of development by weeding out features that customers don’t want and refining the ones they do. We show several design iterations of designs to both customers and internal users to make sure we’re solving the core problem.
For example, we received feedback that some people selling camera gear to MPB were unclear about which accessories to send in (Do I include the lens case? Neck strap? Original packaging?). We built two prototypes — one that told sellers what to send (non-interactive) and another where sellers told us what they wanted to send (interactive).
Customer interviews showed us that sellers preferred the interactive option, but the non-interactive option would be better than no information at all.
But our developers pointed out that building an interactive list would require significantly more time and effort. Together, we decided to build the non-interactive feature first to bring value to our customers sooner. This gives us time to iterate towards interactive versions.
We’re lucky that MPB customers are generous with their feedback. In order to harness that creativity, we established the MPB Research Panel. It is made up of visual storytellers who are willing to give us feedback on new concepts, designs and features. (Sound interesting? Here’s the link.)
Because our research panel is maintained internally and composed of participants eager to improve the MPB experience, we are able to test ideas seven times faster than we were a year ago, and significantly more cost-effective. The research panel is made up of almost 100 fabulous individuals who have genuinely shaped the platform with us. By testing ideas with them early and often, we’re able to ensure value from the very beginning of the product development life cycle.
During development
How can you deliver value to customers during development? Through strong change management processes — otherwise known as continuous integration and continuous delivery (CI/CD).
CI/CD is “a modern software development practice in which incremental code changes are made frequently and reliably.” It enables the speed, stability and availability of changes that contribute to organisational performance.
Fundamentally, CI/CD means having an effective process and tooling to manage change so we can continue to rapidly and safely release the features that drive organisational performance. It can be costly, but luckily for you we have an article on how to keep CI/CD system hosting low.
At MPB, this has taken form in four distinct areas:
- Feature environments: These live outside our customer-facing platform so we can test new (you guessed it) features and catch issues internally before they affect the customer. We use GitOps, Helm and Kubernetes.
- Observability: We started with Service Level Objectives (SLOs), which define reliability around key business objectives (for example, 99% of quotes to be generated in under a minute). We have also increased observability on our internal operations platform, to alert our support engineers if a key part of the infrastructure fails.
- Automation: We have automated many parts of the CI/CD process including regression runs and promotions, freeing developers to do more … well, developing. And we have automated the release process to reduce risk and standardise incident response.
- Post-mortems: Retrospectives help us avoid making the same mistake twice. They mitigate risk and strengthen existing processes by identifying weaknesses.
In the past year we have:
- Cut the number of changes that lead to failures from 5–10% to zero
- Reduce lead time for changes to a week, from 2–3 weeks last year
- Automate processes to free engineers to work on more productive projects.
We track this data through DevOps Research and Assessment (DORA) measurements, a programme supported by Google. This ranks performance in four categories: low, medium, high or elite. MPB’s goal is to be “elite” in each DORA metric: deployment frequency, lead time for changes, change failure rate, and time to restore service. We are currently “elite” for two of the four and working to improve in the other two.
After shipment
After we ship a feature, we evaluate. Did it add value to our customers? We answer this question in three main ways: conversion rates, NPS scores and Voice of the Customer reports.
Conversion rates
To obtain conversion rates we essentially divide the number of people who completed a transaction (buy, sell or trade) by the number who started one.
For example, we recently tracked the impact of a new sticky footer on our sales form (see screenshot below).
We hypothesised that potential sellers were abandoning sales because they felt they did not have enough information about the process.The new footer contains explanations of the selling and trading processes in addition to popular FAQ items.
In the past three months, we have seen sellers who open the sticky footer convert at twice the rate of the general population. This gives us confidence that the sticky footer provides value to our customers, and it has proven to be a space ripe for iteration.
NPS scores
NPS scores are calculated via a survey sent out to all customers who complete a transaction. They’re asked to rate us on a scale of 1–10 and then given an opportunity to provide feedback. The NPS scale goes from -100 to +100 and our goal is to stay above +70 (which we are achieving!).
We recently changed courier services in France and wanted to evaluate the impact. We used Chattermill to analyse post-transaction survey data to understand customer sentiment. This showed that the “overall turnaround speed” category had risen by 23 NPS points since the switch. We can therefore say with confidence that we’re on the right track with our new courier.
We followed up this analysis with customer interviews (shout out to Ariane on the Seller Experience Team for supporting us with this research) to dig into specifics about how we can continue to improve our delivery services.
Voice Of The Customer
We produce a fortnightly Voice Of The Customer report which compiles the biggest themes from qualitative feedback. We collate this data from our NPS surveys, Trustpilot, Google Reviews, social media, interviews and anywhere else customers give us feedback.
We use this information to identify opportunities for growth and understand what customers like about our services. The resulting insights are fed into Product Managers, who work with Design and Development teams to find solutions.
Improving continuous discovery, integration and deployment
There’s plenty of work in progress to improve the way we deliver value to our customers. One key area for growth is tracking the impact of our work.
We currently track high-level impact, such as conversion rates for buying, selling and trading. But how can we assess the value delivered by fixing a back-end bug? My answer: by setting success metrics that narrow in on a specific goal. Localised success metrics reduce the number of unknown variables feeding in, and allow us to clearly evaluate (and celebrate!) the value we’ve delivered.
We are also moving towards prioritising work based on customer data. Potential avenues include counting the number of feature requests from customers, calculating an impact/effort score for a new idea, or estimating a predicted Gross Merchandise Value (GMV) change after a release.
This work will include teaching interested people to carry out their own research using our suite of tools, including Chattermill for qualitative feedback and FullStory for quantitative insights. Democratising research means that we all learn, and faster.
We’re constantly looking for ways to deliver value to our customers. If you’ve made it this far and are interested in joining our research panel, we’d love to hear from you. And if you’ve ever given us feedback, thank you — we have so much still to learn, and appreciate you taking the time to teach us.
Here’s to continuous growth!
Marley Dizney Swanson is the Senior User Experience (UX) Researcher at MPB, the largest global platform to buy, sell and trade used photo and video gear. www.mpb.com
