Product testing: what we learnt from testing Service Recipes with new users
6 insights from 71 total, gained from usability testing with 5 new site users.
Two weeks ago we tested a load of changes we were making to the Service Recipes website. This post explains:
- why we tested again
- how we tested
- 6 key insights, with a few others woven in
Summary
Testing led to 71 total insights, 6 do-now design tweaks, 9 visual design improvement ideas and 9 style guide ideas for the new recipe content pattern.
Why we tested again
Back in November we tested the old version with users who already knew about Service Recipes. This generated over 160 insights, many of which we acted on during December’s development sprint. This led to lots of changes:
- Several small design changes
- Search functionality improvements
- A new content pattern
- A different back-end system for creating recipes.
But before making these changes life we needed to test them with people who hadn’t used the Service Recipes site before — who didn’t know what Service Recipes were. That way we could:
- understand if and how the site and its name helped them understand what recipes are
- learn if the design changes we made met their needs.
How we tested
We used a similar approach to November’s sessions:
- Invite 130 people from Catalyst’s research list to complete a screening questionnaire. Screening was important to ensure they didn’t know what recipes were yet, worked for a non-profit and were available the week of the testing sessions.
- Select 6 users from the 57 who expressed interest. We used their response to a personal characteristics question to make sure we had a diverse range of people taking part.
- Invite users to book in a 1hr testing session. We used Calendly to do this. It was so simple.
- Run the sessions: each one a brief interview then a set of tasks:
- We asked testers to carry out some tasks on the site and to commentate on what they were seeing, thinking and expecting on the site. We also checked how their understanding of recipes changed as they completed the tasks.
- We showed them some different homepage images to get a sense of what they liked and associated with their (new) understanding of recipes
- Ran a card sorting exercise to see how they grouped different recipe titles
5. Synthesise the notes to generate insight statements.
1. Visual design: the homepage is clear and calming
Like before people found the colour and feel of homepage to be calming. This seems to encourage people to scroll the home page rather than using the menu. In turn this led to them often and easily finding an action to take on the page.
“The blue, calming, nice to look at, illustration not too busy”
“Love the look (like the dark blue with the light blue and the graphic)”
“That’s quite unusual actually — nice to be able to go straight in”
Observation: scrolls slowly, all the way down — hovers on logos briefly
Observation: scrolls first
What we will do
- Keep the homepage layout as is.
- Keep the clear and calming feel in the new visual design
- Create a new design that shows what recipes are, while reflecting diversity of recipes and contributors
2. Branding: contributed by other charities
People responded very positively when they saw on the homepage that recipes were contributed by other charities. No one knew about it before the session.
Three things are significant here:
- Their response was unsolicited — we weren’t looking to gauge their reaction to learning this
- Their response made us realise we had underestimated the value of this aspect of recipes — that they come, freely shared, from their peers
- The name Service Recipes doesn’t communicate this value (a good reason to consider changing the name — see insight 3!).
What we will do
- Adjust homepage strapline to promote that recipes are shared by charities
- Explore alternative names for Service Recipes
3. Understanding what recipes are
Before taking them to the site we asked people what they thought recipes were. Most people broadly guessed at some kind of guide or ‘how to’. However:
- People were unsure what the format would be and how interactive it would be
- After exploring recipes some people found the site name incongruous with what the site offered. This happened enough that we are considering changing the site name, something we considered in the Autumn (for the same reasons) but didn’t think there was enough of a case for.
Before quotes
“Recipes is intriguing but I don’t really know what that means.”
“I wanna know what sort of recipes we’re talking about.”
“Definitely less boring (Service Recipes as a title)- but I’m not sure I know what it meant”
After
People generally described the recipes as posts or guides to how charities or organisations have used digital. However, they found the name did not match what they found
“I don’t really know what the service part means — I slightly understand more recipes — the step by step part”
“Having gone through the guides I don’t think that term (recipes) fits with what is actually being provided”
“I’m not that clear on the service part — but I’m getting the recipe side more — what it means and what you are showing”
What we will do
- Explore alternative product names. I’ll write about this in a weeknote.
4. Value: people find value in other recipes to those they were looking for
As people browse the list of all recipes they connect titles to work they are doing. They do this even when they come are actually looking for something else.
They also do this on search results pages, even when some results that aren’t what they were looking for.
We observed this by people either:
- commenting that a recipe title was interesting, because of its relevance to work they are doing, want to do, or even have done
- hovering over a recipe title, then responding to us asking why they did this
This behaviour matched that of November’s testing cohort. One person would use recipes about things they hadn’t yet done to lobby for support from their organisation to do it.
What we will do
- Rewrite the titles of all old recipes so they are easier for users to scan and decide if interesting enough to view. We were doing this anyway.
- Make a couple of tweaks to how the search feature is configured to make sure it offers results with enough possible breadth and relevance to what the user is looking for
5. Visual design: very readable but not engaging enough
People generally find the visual design clear and readable. But at times they don’t find it engaging or interactive enough. They are put off by stock photos on recipes.
“Stressed again wanting to see different colours or fonts or something to mark out boxes”
“Barely see the box — think maybe it needs to have a brighter background — to make it more distinct”
Observation: Talks more about how the page has very muted tones which does make it easier to read but there’s not a lot of action on the site
What we will do
- Commission a new visual design. We’re doing this because we’ve saved some budget by not making any small visual design changes in this funding round and have some content budget to add to it. If there wasn’t this budget we wouldn’t be doing this. New design should launch around Easter.
6. Usability: expanders in recipes
Expanders are a new feature that make recipe steps easier to browse and navigate. Expanders are also known as accordions or toggle headings.
Everyone was positive about using expanders to explore recipe steps. Within expanders people were most engaged by the example part which described what the organisation did.
“Repeated pattern is useful for a dyslexic brain”
“I like these parts of having the insight of what they did it — I like having that context to read”
“Likes the “this how you could do it and then “this is how we did it”
What we will do
- Keep this new design for steps.
Sign up for email updates
Get email updates and find out when the new recipes site launches in April 2023. Sign up.
