Finastra Labs’s guide how to save time on testing.
Testing is a (must have) burden, isn’t it?
Feedback (therefore testing) is king. You should start it as early as you can and rigorously repeat it long after your solution is ‘ready’ — sounds the popular mantra. Although this is true without question, in reality it is easier said than done. After a team spends a lot of time searching for the best test methodologies an inevitable question emerges: are there any best practices or shortcuts to use without jeopardising the outcome’s value?
When and what should be tested?
Anytime and every time. Unfortunately even tech giants lack the resources to follow it, still, the principle is true. Regardless of the resources testing and the following modifications lengthen release lifecycles — particularly in prototyping — dramatically. So one should play smart when it comes to assembling testing plans.
Our team, Finastra Labs has the luxury to experiment (a lot): idea trials, release/standalone feature, high/low fidelity prototype testing, we even start sometimes small with quickly drawn paper cards. All performed well in certain aspects and failed in others. Here are a few things we learned so far during our in-person tests:
- Idea testing is ok if you reverse the situation: instead of telling your idea to users expecting them to judge it (they can’t) ask a lot about the topic, sniff around and see if your solution is aligned with their mental model or not. It requires routine to ask the proper questions but due to the simplicity of the interview process it’s input/outcome ratio is great .
- Low fidelity didn’t really work for us. Although we love to play around with pen and paper while brainstorming, as soon as you put it in front of your customers it fails, a lot. It’s no question that its good for simple site layout or navigation testing but when using digital banking users become more attentive which does not perform well with paper prototypes that creates a lot of uncertainty in users. Mostly these two things happen: you get a lot of unrelated feedback (mostly due to the missing details and GUI) then spend a hell a lot of testing time explaining the prototype itself.
- Fake it till you make it, another favourite of ours. Instead of paper we use moderately designed screens with click(tap)able areas, connected together (mostly in Invision) on the users own devices. We only include the red-route content but place most of the options and navigation items on the screens. Once we compared with two test sessions, the outcomes (first with this method then with the final product) and the value of feedbacks barely differed.
As stated above we learned all of these during moderated in-person tests. Although these are great occasions to discover things we haven’t even thought about before but the time spent on them is a big burden. Therefore we try to cut the user journeys into smaller pieces and align the scenarios to the specifics of unmoderated remote test environments. Combining it with in-person test grants both the deep insights and the cardinality for us.
Ok but are there any shortcuts?
Without losing value, barely. Although we found a few techniques that can significantly reduce the workload, especially in the first phase of your project. Our favourite is called design critique. We use it to decrease the number of external tests by discovering the design mistakes as early as possible. As you can see this chat example below, the technique defines the terms ‘design’ and ‘designer’ (aka owner) broadly: it can be a visual interpretation-less chat flow or a high fidelity product feature as well.
It’s an in-house technique taking up only your colleagues time. It starts with the idea or prototype owner inviting 4–6 non-team members (various roles!) to the session. At the beginning I only announced the topic of the critique prior to the meeting but later started to distribute descriptions and pictures or sometimes the prototype as well. Only the minority will check it but who actually does, will feel much more comfortable with the whole process. As part of the preparation the designer should print out and put up all the (major) steps of the prototype on the wall. Make sure to cover all states that will give sense for your flow without a detailed explanation. This wall will give a dedicated space to look at and work with during the session.
When the meeting starts, the owner should always briefly introduce the agenda and clarify the rules of criticism. Reminding everyone how to give good feedback (either positive or negative) is crucial to succeed. Here are a few important points to note (even in a pre-meeting mail).
- Be constructive. Each piece of comment should be formulated in a way to include a self-explanatory reasoning for the bottom line. It is always a good idea to relate to the prototype’s business goal, how the criticised piece supports or damages it.
- Be sincere and active. everyone had been invited to express their feelings not to stay silent. The meeting is about to help the designer not to nurture him.
- Never forget to be positive. It is natural that participants focus on things that need to be corrected (in order to feel productive) but highlighting those points that you like is equally important feedback (plus creates a good mood).
After everyone reset their minds according this critique guideline, the designer pitches the idea, the business and customer goal(s), characteristic of the targeted users and the environment of usage (we love to use the as a/I want to/so that formula!). In some cases it worth to even describe a few steps prior to the flow: how the customer got there, what was the last screen or action he took?
Thereafter the designer has the freedom to either walk the group through (shortly) each steps with additional info (like: it is a proven fact that emojis help people to choose from multiple options) or let the group observe the flow on their own. Either way, this part should not take more than 5 minutes.
When these basics are done, the attendees start to take notes, individually, silently, one note/one post-it. Depending on the length of the flow this should take max up to 10 minutes (we found out that the tipping point for valuable feedback is around 9 minutes). When time is up, the first participant starts to put his post-its on the wall for the first step (or the general idea itself). Thereafter the next person follows until everyone run out of comments for that step, so you move on to the next one.
It is important to say the content of your post-it out loud as much as to formulate it in a constructive way (see above): avoid personal and emotional comments (like: I find this step pointless) and try to reason your thoughts (e.g. I think this step is unnecessary since the customers already know this info from the previous one). We know that the temptation is strong but everyone (including the designer) should avoid to solve or even reply to the raised problems, otherwise group-discussions will spoil the session. The designer should sit silently and make notes (a lot!) while the others put up their post-its (of course clarification questions may be asked).
The wrap up should be a simple ‘Thank you all for coming’ since your time will most likely be up by this point. If all went well the designer will leave with a tons of feedback while the participants feel accomplished, fun, being part of something important.
30 mins’s impact on our life
We absolutely love it! If you follow these steps and tips above, you will be able to finish a session like this in 30 mins and end up with a long list of valuable comments. Our experience is that these sessions helped us to detect and correct 50–60% of the issues we faced in the first round of the external user tests. It means that we did not just saved half our time and money but also moved faster since we got twice as much hard-to-detect issues instead of testing the obvious ones.
We started to do them as a routine half a year ago and will probably keep continuing :) I’m personally also delighted about it’s “side effects”, like getting to know my colleagues’ project better, to see how other valuate the same things differently and to simply break away from my daily routine for 30 mins.
