What to Look Out For When Conducting An Alpha Test

And why it matters

A series of screens designed for the Nine Tales app. Graphic created by author.

There are a ton of tests to be completed in a developmental process when you are launching a product. You may have heard of a usability test, end-to-end testing, or a beta test, but have you heard of an alpha test?

What is an alpha test, and why is it important?

Alpha testing is one of the many user acceptance tests (UATs); just like the definition of the word “alpha,” it is the first form of acceptance test performed. It is carried out to validate key functionalities and/or features of a product, which is tested out by internal staff, including stakeholders.

More often than not, most teams disregard an alpha test. Some clients may look at it as an additional step to the whole developmental process, signifying that extra time is needed, which may potentially cause a delay in the launch of a product. You (or some clients) may also ask, why should we run a test ourselves when we should get feedback from real users instead? They are ultimately the ones using the product, and hence isn’t it more important to know how and what they think?

These points and concerns are valid, however, by testing it ourselves, we get to experience first-hand how key functionalities and/or features work. An alpha test also helps to:

• Save initial costs and time of recruiting participants to test your product
• Spot flaws in user flow and user interfaces early, allowing the design team to improve on them before moving on to run a beta test
• Allow stakeholders to identify major green and/or red flags of any features and functions in the product early on before any time and cost is wasted (on recruiting participants)

Hence when we were designing an initial prototype for Nine Tales, a startup focusing on helping children build emotional intelligence through storytelling, the idea of conducting an alpha test sparked. We had been working on an initial prototype, and we had some doubts about certain key features and functionalities of the product.

What better way to clear those uncertainties than to conduct an alpha test?

3 Learnings from Conducting an Alpha Test

After conducting an alpha test for Nine Tales, here’s what we think you should look out for when running a test of your own:

1. Set your objectives

Before running through this idea of an alpha test with your client or stakeholders, it’s important to justify conducting the test by setting objectives.

What are you trying to find out?
What key functions and/or features are you specifically testing?

Without setting objectives, running the test would be insignificant as there is no premise to fall back on.

With Nine Tales, we questioned ourselves if certain key features and functionalities would be easy to use and understand:

• Would assigning characters in stories be enjoyable and easy to use?
• How would the different recording flows (assigning of characters, round-robin, and recording on your own) work on different stories?
• Would recording a story be intuitive and enjoyable?
• Are there any initial bugs and issues we should be aware of?

These objectives help set a premise and at the same time, help determine your next steps in crafting the test.

2. Determine the test plan, time, and resources needed

A key to a successful alpha test (or any test for that matter) is planning. With a detailed plan, the test has a higher probability of running smoothly and efficiently. It also reduces risks and might help avoid potential issues that might arise.

Two factors to take into consideration while crafting a test plan are time and resources needed. You might ask:

• How long will the whole test take?
• With the set of features and functions to test, how are we going to execute the test?
• How many testers will be needed?
• How should we gather our feedback/insights? (e.g. surveys, interviews)

How we split the tests, roles, and responsibilities for Nine Tales’s alpha test. Actual names of testers are hidden due to confidentiality.

As we wanted to test how family units in real-time work together to record a story in Nine Tales, we designed the alpha test to be conducted over a course of one week. Every single person in the team, including stakeholders, was involved in the alpha test. All of us were split into different family units, with each family unit testing two stories. There were three different tests, defined based on the objectives we had set. The one-week duration allowed us to record stories at our own pace, resembling the reality of how busy each family member might be on a daily basis.

3. Prepare your client for possible outcomes

With objectives defined and a detailed test plan, you might feel excited and more than ready to pitch the idea of an alpha test to your client. But here’s what we learned: While you are pitching, you should always, always, prepare clients for possible outcomes.

Have an honest conversation and lay out all possible outcomes, be it a success or a failure, as this keeps expectations to a minimum. Clients might expect the test to be completed in a week when it might take two weeks or more (depending on what your objectives are), or they might expect to move on to conduct a beta test immediately after, or they might not understand the amount of time needed to improve on the designs before conducting another test, and so on. These assumptions are valid from a business standpoint, and hence preparing them for all possible outcomes goes a long way. Should insights gathered be unpleasant, the advanced preparation would allow both parties (your team and the client) to expect the necessary steps that will follow.

Key findings from Nine Tales’s alpha test

Fortunately and thankfully, Nine Tales was immediately on board with the idea of an alpha test. They valued the benefits that come with it and had already expected several issues to arise from the test, which both of our teams sat down together to resolve. In general, here are our summarized findings from the alpha test:

• 100% agreed that assigning characters was fun!
• 67% of the testers found it easy to record for the first time, and 100% agreed that it gets easier the second time onwards.
• 91% of the testers found the process of recording stories to be enjoyable.

We also tweaked and added some features to elevate the whole experience of using Nine Tales. Below are a few graphics on some improvements we made:

Improvement 1: Since the feature of assigning characters was a hit, we decided to make the assignments viewable to all family members.

Improvement 2: Introduce “Recording in Progress” section on Home Page.

Improvement 3: Introduce a step-by-step tutorial for users recording a story for the first time.

I hope our learnings convinced you (and hopefully your clients) to conduct an alpha test. It is the first step to ensure the product meets the business requirements and functions, and how you can redefine it to better serve your consumers. It is often overlooked, but it is as important as other UATs because of the many benefits that come with it.

To learn more about how real users react to Nine Tales’s updated features and functions after the alpha test, look out for our next article where we will share more about it! 👀

Elaine does UI/UX design at 55 Minutes. Inspired by architecture, art, fashion, nature, photography, and people, Elaine’s approach to design is to tap into what makes us human — our emotions. She believes good design involves understanding who she’s designing for, and how design makes them feel. For her to be pixel-perfect in her work, she needs a good breakfast. Coffee is a must too.