Part 2

TL;DR: Conducting a pilot test is an excellent opportunity to validate your product in a real-life setup before its release. It’s no small feat. Very exciting in the beginning, time-consuming in the making and exhausting around the insight processing but in the end looking at the massive results it’s truly rewarding.

István Tóth

Follow

Dec 3, 2020 · 8 min read

Intro

We work in digital healthcare, getting something out for actual users to test is not as easy as you’d think. Luckily, we had the chance to lead a 3.5-month pilot study preceded by a 1-month pre-pilot test with an internal user. After months of preparation, we were ready to get started with the real deal. (For more about pilot study in general see Part 1: Preparations and dry run.)

Goals of the pilot study

• Investigate whether the proposed feature set of the product provides enough value to the customers.
• Get an overall understanding of how the product is used in a real-life setup.
• Discover errors in both the design and the implementation.
• Explore improvement suggestions and constructive feedback about the product.

Userbase

A solid userbase was recruited out of our customers. The ideal number of pilot users are neither too small nor too many. To avoid lopsided results they belong to different customers and are a good match with the prime persona.

For us, there could have been 2 bottlenecks to the ideal number of users.

Customers’ capacity to deal with the pilot study — can result in:

• A small number of users
• Low user activity

UX team’s capacity to observe, gain insights and interpret them within a reasonable time — to name a few possible consequences:

• Unnoticed important insights
• Overlooked bugs
• Unimportant features
• Late understanding of user needs (out of sync with product development)

Thankfully, none of these jeopardized the success of our pilot testing.

How things kicked off?

Each user had their own access to the software. The environment was unique to the respective customers. After the initial briefing pilot users were let alone. Intentionally. They were expected to act as if they just gained access to the software for the first time and the product was to be discovered and then used for their daily work in a real-life setup. These happened at their location in their own environment (thanks to COVID–19 for some folks this shifted to their actual homes). From time-to-time, we checked in to see how they were doing with the product and the pandemic. These all took place remotely.

Research

We planned a lot of things: weekly interviews, on-site shadowing, analytics to gather quantitative data and daily diary studies from the users. For details, see Part 1: Preparations and dry run.

However, there are things you cannot prepare for like an unprecedented pandemic. Needless to say, all in-person meetings (observations, shadowing) were canceled. Our users had to figure out how to work in the lab safely and how to work from home, which they didn’t do much before due to the nature of their jobs. This made remote interviews hard too as our already quite busy users got even less available. Some of their projects were canceled and a lot of data they uploaded into our software became obsolete — they had to restart with new projects. So, unfortunately, they didn’t send us daily feedback; we received very few diary studies from our users. Even weekly interviews were rescheduled to biweekly with some customers. The silver lining is once we got the analytics plugged in it worked smoothly.

Synthesis

We recorded interview notes into an online note-taking app, to make it easily accessible and editable for anyone in our team observing the interviews. Afterwards, we continuously categorized feedback (positive, neutral, improvement need) and added them to a huge Excel spreadsheet by customer and by date. The improvement needs were also copied on a Trello board where we measured how many times users mentioned the same problems and prioritized them on the go. This was necessary because we wanted to iterate and release new versions of the software during the pilot, so we couldn’t wait until the end of the study to process the most important insights.

These preliminary results were then further prioritized by business and UX together, while effort points were estimated by development and UX. We then held a workshop where the team put the improvement needs (not features yet!) into a value-effort matrix. This helped us go forward with the following releases.

In the end, we gathered all the insights and copied them into Miro (pre-pilot, pilot, analytics). We ended up with more than 670 sticky notes. We held a 2.5-hour Affinity Mapping session with our multidisciplinary project team where we grouped most of the post-its. Then the UX refined the groups and drew conclusions, which took around 8 hours for 3 people.

Lessons learned

Research should be done by researchers

Most of the pilot testing activities were planned, conducted and facilitated by UX designers. By the ones who designed the product itself. Clearly, it’s suboptimal. No matter how much you strive it’s hard to remain unbiased. All the activities of the pilot test require more researcher practice than designer.

If we had more researcher support from the beginning to the end of the testing, the way of execution would have been different. It might have had an impact on the final results and some small-scale insights could have emerged.

Web analytics is not that straightforward in healthcare

Patient data security and privacy is a sensitive topic especially in the era of digital healthcare. It’s a long process to get a tool sanctioned by cybersecurity. Even though the world is moving into the cloud most often the quickest and most-secure solution is to use tools that can be installed on your own server, which is probably already secured. For us, Matomo turned out to be a great and inexpensive web analytics platform (with some limitations compared to Google Analytics) that provided us 100% data ownership — just as they advertise.

However, it was problematic to install it on our server and integrate it properly, because of strict security measures. Defining how the UI can reach it and configuring it was also less straightforward, so it took some time for our tech team to figure it out. It caused some delay in the implementation of Matomo and we could start using it later than expected.

Users are busy and not as excited as you are

We had to reschedule the interviews with some users from weekly to a biweekly basis. They didn’t spend as much time with the software as we expected. Although they were excited to try something new and do something different from their everyday work, working with our product was just another task on their busy daily agenda. And it really surprised us but some of them couldn’t spare one hour per week for an interview either.

For the same reason, the diary study didn’t work out as we planned either. They took notes during their usage of the software, some of them even created presentations with screenshots for the interviews. But for some unrevealed reason, they really didn’t like sending these to us ahead of time. We asked and reminded them several times, then we just let go of this part of the study for the sake of a good relationship with the users. If they don’t want to do something, you can’t force them.

The MVP

In general, users liked the product for being easy to use, quick to learn and intuitive. In healthcare and life sciences, this is a big deal. Working with pieces of robust desktop software that require reading hundreds of pages long user manuals, this was refreshing for our users. They also liked the main concept of the product. They expressed hopes that it will solve many of their current pain points. Unfortunately, the MVP version of the software couldn’t do that just yet. But we got tons of feedback, so we know how to proceed.

Iterations

During these 3.5 months, we tried to react to user feedback as fast as possible. However, here the limitation is the development and implementation and not the design. We had two Product Increments (PIs) during the pilot, based on the most important improvement needs mentioned by our users. Their feedback was a great help in aligning business and UX goals and to determine the direction in which we wanted to improve the product.

Unfortunately, in a strict environment such as healthcare, pushing out new versions of software cannot happen overnight. It took longer to develop and release new iterations to the pilot users than we expected in the beginning. Sadly, the second iteration couldn’t even be finished before the pilot ended, mainly for security reasons. The first iteration, however, was a great delight for the users and we were happy to learn that we’re heading in the right way. It was very useful and felt fulfilling to get feedback right away.

Team effort

Needless to say, it’s been an effort by a larger team than solely the UX. Without the tireless efforts of business, product and engineering teams the pilot environment couldn’t be established, bug popping up couldn’t be quickly fixed, and the future state of the product couldn’t be defined. These folks were involved in interviews too, so valuable questions were raised and these observers could better empathize with the end-users.

Even though it was tough to carry out a 3-month long pilot study with all the ups and downs, it was totally worth it.

Key takeaways:

1. Do it. Do the pilot. Convince your team, get help from researchers, do whatever’s needed because it’s really worth it and you can save a lot of time and struggle by getting the right product to real users as soon as possible.
2. Don’t expect though that your users will be as eager as you. They are busy and your product is probably an MVP, so they will be somewhat unsatisfied and frustrated. However, every bit of feedback they give is very precious, so don’t forget to appreciate that.
3. Prepare to do adjustments to the design and to your own processes. Development, on the other hand, can be slower and more disrupted, so don’t expect everything you design to be implemented. Though, all of them will be useful later, when you synthesized everything and decide which direction to go.

The case study was written by Tünde Gál and István Tóth.