Do Not Disturb — the value of remote qualitative testing
Here at Elsewhen we are always trying to find the best way to learn from our audience and iterate our products according to their needs. So, when we got the chance to work with pre-school children to create of Hopster’s Apple TV App , we were thrilled.
When working with such young kids it’s important to know that there is a big difference between a 2-year-old and a 5-year-old. At those ages, children’s knowledge and experience curve changes very fast, as does their understanding of how to navigate a digital interface. For example a 5-year-old has probably been using digital products for some time, while a 2-year-old probably hasn’t.
In order to understand such variety of needs and wants and progress quickly with our product, we decided to establish a group of families to try, test and feedback on the Hopster Apple TV product on a daily basis. Hopster has a fantastic fan base so, with their help, we recruited about 15 families. As we wanted to know exactly how children and parents interact with content on the Apple TV, we decided to carry out the tests in their homes, rather than getting them to come to us. The production process was on a 2-week sprint basis, and after every sprint we released a version of the product with this test group.
From the start, we wanted to have a feedback cycle that was as short as possible. We also wanted to use our time in the product team efficiently, without interrupting families too often. So, we set up a journal system — a basic journal where every family had a Google Doc with their name and a set format that they used to report back to us every time they used the latest iteration of the product. This meant that the product was used as intended, while giving us good qualitative data that we could then match with the product analytics to measure core metrics, such as the average session time.
This system had many benefits.
Because the test group expected to have something new to test after every sprint, the product team had to work hard on having something shippable and releasable with every cycle. This made discussions and prioritisation in our planning meetings very easy, because the objective was set. What’s more, as the data was delivered to the team regularly, reporting findings and validating decisions became more efficient. Finally, with data that was coming directly and unfiltered from their core audience, all stakeholders felt reassured throughout the process.
Having a short feedback loop and easy access to data from the very start of the project is key for successful product/market fit. By making the most of an already established fan base and following a process where data and feedback come to you regularly, almost on tap — you have more time to concentrate on using those insights to create a great product.
