What we’ve learned experimenting in-house usability test
Building recurring, quick and not-so-dirty usability test sessions to ensure the product team keeps delivering quality product at low cost; even if you don’t have enough resources! How we ended up in-house testing at Algolia. 🚨 in-house testing cannot replace testing with real users! 🚨
In Q3 2019, the squad in charge of the Analytics at Algolia went through some major changes. The main goal was to improve the overall user experience of the foundational analytics features. As a user researcher, I’ve been asked to help them iterate quickly and focus on usability.
To do so, we put in place a usability testing process: every other week we would test iterative mockups to validate and implement them on the go (Happy to share thoughts on how we did this in detail if you are curious!).
When we talk about putting a usability testing process in place, a few typical constraints are often considered, namely:
First — squads have limited resources
Delivering qualitative product updates should not reduce the speed of shipping (or only minimally impact).
👉 Usability tests have to fit into the dense product development process.
Second — user researchers & designers have tight time constraints
We often have several projects for different squads going on simultaneously.
👉 User research & product design time has to be used wisely and optimized to deliver value.
Third — recruiting test participants is hard and can take a lot of time
(*note- this is not a valuable use of user researchers & designers time 👆)
This is particularly true if you work in B2B, and you have to find, let’s say, search PM or Search marketers (if you are one of them: I’ve been looking for you).
👉 Recruiting can not be a blocker (If you want to know more about our recruiting strategy for user research, you can read this article)
To prevent recruiting from becoming a blocker, I’ve put together a recruiting strategy scale of confidence.
In an ideal world, we would test our mockups with our customers (and the right profiles among our customers). In actuality, with time, budget and resources we sometimes have to test in with relevant profiles within Algolia.
This internal quick and “not-so-dirty” way of gathering feedback is far from new. But let me tell you how we put it in place at Algolia and what we have learnt in doing so.
🥚 Birth of the testing corner
At the 5th test session (out of 6 in the corner), we struggled with recruiting, it was the end of September, our CSM team and our clients had a lot to do, and no one answered our calls, emails or slack messages. We wanted to test our new compare-range features which allows you to compare two time periods in the analytics section. What we were trying to learn was more about making the time comparison clear than having a deeper understanding and use of the displayed metrics. Since this was less specific to a very targeted user group we decided to test with our own Algolia team members.
Our tests were scheduled for Wednesday, so we ensured the Bridge (Algolia’s epicenter) was free that day so we could set up the scene for our test there. We needed more “business profiles” than techy ones, so targeted our marketing folks, sales team and our CSMs, rather than our engineers; we are lucky to have so many diverse folks in one office
We then had to communicate to the company “The Testing Corner” was coming, what it was and why they should join us. Communication here is super important to ensure your colleagues know what’s happening and can more easily participate. We even drafted posters to inform them where to go and what we were doing.
Wednesday was upon us and we arranged a proper scene for the tests. We needed to be seen by everyone in the company yet ensure some privacy for the brave Algolians who agreed to take the test. We ingeniously displayed the crepes our office manager ordered for the Algolia employees on the test table; help us out, then grab a crepe 🥞!
To handle the test, we were paired: the designer taking notes and myself asking the questions. The PM was not far away either 😉
The mockups were displayed on the computer and a screen recorder started before the test to have every test recorded (just because it’s internal doesn’t mean one should forget good researcher’s habits!).
We were all set. It was all about smiling and saying “Do you have 10min? That’s all it takes!”.
The test was a real success. In less than 3 hours, we had 10 testers, great conversations around the product with people who were not used to getting involved at that stage and enough valuable feedback to iterate with confidence.
In the end, everyone benefited from this method:
- Gave the opportunity for everyone to participate in the product development process
- Helped other teams better understand UX issues
- Super good for the visibility of UX in the company (and with an increased awareness comes more projects)
- We had a lot of fun
🐣 The Testing Corner becoming a thing
After the initial success, we decided to offer the Testing Corner to any squad who needs to iterate fast. We help to smoothly set up a quick usability test in the Bridge, to get rapid, valuable feedback and thus started the “Usability-testing-as-a-service” in Q4.
The idea was simple: every other week, any squad could register to have its own usability test running in the testing corner. All they have to do is register one week in advance, so I can get back to them and define the test goals, and make sure the PM or designer is available to assist on testing day.
Results in less than 2 days guaranteed.
If no one registers, no testing corner (and I have time to get other work done).
We wanted to make sure we only tested when we needed to and not just because testing is fun and cool. So we put some rules in place:
Note: I took some inspiration in this great article to establish when to test or not.
🐥 The Testing Corner still has room for improvement
We are quite happy with what we have already created — we even have inspired other teams who want to handle their own testing corner now. Plus, Algolia is definitely fostering a more user-centered culture- and this is great.
However, this is not the end of the story!
We still have to improve and learn from this quarter, namely:
- We don’t have registrants for every open time slot. I bet it’s not because we don’t have things to test, but rather because we still have to increase awareness around testing: explain better when it is needed, why it is important, and encourage the teams to do it (even if it means spending a few hours).
- Empower the different squads to do their own testing so the user research team can focus on more strategic projects. To achieve that, we are working on internal training for designers and PMs; when and how to run a usability test properly. We are also creating templates and checklists to make it easier to jump in, get going and be consistent.
Excited to hear from you about what worked for your teams!
Do you have any questions? Have you and your teams tried similar initiatives? What happened?
Thanks to Marlin De May for her precious help!
🎁 Gift: the testing corner checklist
Organize your very own testing corner.
🗓 Week -1
- Review the mockups/sync with pm/designer
- Define who is the main target user
- Send email to the company (or specific team)
- Prepare the posters 🎨
🗓 Day -1
- Slack the company
- Final check-in with the team
- Make sure you have all the material you need (computer + recording tools, posters, table)
🗓 Testing-Day
Before
- Buy food (or other bribes) on your way to the office
- Arrive early to setup the corner (posters + table + display food)
- Slack the company with a picture (get them excited)
During
- Standby near the morning coffee machines for recruiting
- Be ready for the second round of coffee breaks as well
- Ask someone to take pictures for you as advertisement
- By lunch time you’ll probably have enough feedback already ;)
🗓 After Testing
- Review the notes/videos for synthesis (this can be processed & done the same day!)
- Send a slack message to thank everybody with a picture (specifically shout out to testers and make them feel good)
- Get some rest — you have earned it! 🕊
