Alpha Phase: leveraging our team’s existing knowledge for improving the quality of our output

Pennylane’s product design team faces a big challenge: defining a complex and feature-dense product while keeping simplicity and usability at the centre.

Understanding our users and their craft is essential, and we seize every opportunity to expand that knowledge.

Dealing with a complex topic like accounting is challenging, but luckily we are not alone facing that feat! It is not just up to the product teams to do constant research, interviews, testing, or feedback gathering. As a company, we’re united in a collective, cross-discipline effort that keeps our users at the centre.

Today we’d like to share a process we’ve recently implemented, and is already working wonders to capitalise on the team’s existing knowledge: Alpha testing. For this initiative, we count on the help of our Partner Success team to enhance the quality of the features we put in the hands of our users.

With the help of our Partner Success team, we’re enhancing the quality of the features we put in the hands of our users.

The Partner Success team at Pennylane is responsible for guiding users in the use of our product’s accounting features. This includes providing day-to-day support as well as in-depth training. The team is composed of individuals with accounting knowledge, many of whom have worked as full-time accountants. As a result, the Partner Success team is a highly valuable source of first-hand feedback about our product and our users, their daily struggles, and their experience from day one.

The Alpha phase

This new initiative is born with four objectives:

• Improving communication between the teams building the product (Product Design, Product Management and Developers) and the teams in closest touch with our user-base (Partner Success).
• Having a dedicated, available team with extensive knowledge of the product and deep empathy with our users.
• Standardising and facilitating a process to guarantee the quality of our releases.
• Reducing the workload of our support teams.

It’s a win-win: better chances for the success of our releases and more focus for our support teams, by anticipating problems and improvements earlier on.

How does it work

At Pennylane, we work hard to ensure the quality of our output by testing with users, investing in QA, performing beta phases, and doing progressive rollouts. However, having a continuous improvement mindset means acknowledging that these methods have their limitations:

• Testing requires preparation and we risk not anticipating some scenarios.
• With QA, we also risk not anticipating some scenarios.
• Beta phases are a black-box system: we receive feedback but can’t really get all the details of how things went.
• Progressive rollouts help us avoid oversights reaching all users, but inevitably require reaching some.

All this results in having some preventable issues worsen the experience of our users and create additional pressure for the support teams.

The Alpha phase takes place before a Beta phase or the start of the rollout. A dedicated team of colleagues from the Partner Success team — with accounting knowledge — uses the soon-to-be-released feature with real accounting data. Then, feedback is provided.

Because the partner Success team is in such close contact with users, their returns include pet peeves, edge cases or expectations that would have been easily overlooked otherwise.

The Squad in charge of the new feature then acts in consequence, that can take many shapes: iterations and improvements, additions to our knowledge base (Pennylane Academy) or even freezing the rollout when deeper changes are needed. Explanations are provided to how and why we did or didn’t solve each issue.

Afterwards, the Alpha testers have a final say and give a rating to the feature:

• Positive review: the feature can move on to be released
• Positive review with reservations: the feature can move on to be released, but there are points of attention
• Negative review: the feature needs more work

Based on the overall rating, a decision is made by the Squad in charge of building the feature. It’s not an exact science: only one negative review among all positive ones might be worth a whole rework if the argument is solid.

Moving forward

Alpha testing seems to have come to stay, but won’t replace other methods of research or quality assurance. It helps us capitalise on existing knowledge within the company to increase our chances of success in releasing great features.

Additionally, we have discovered its advantages as a white-box system, providing visibility into the entire process and the factors that contribute to a good or bad user experience.

This goes beyond the surface-level feedback that users might proactively provide, allowing for highly insightful conversations. These insights then help the design and product teams make better use of other research methods that involve direct contact with users, which remains unquestionably essential.