Testing times for our new designs
Back in May, we quietly released new versions of our article and gallery pages to 1% of the Mirror.co.uk audience. The pages weren’t finished, but we wanted to know how they would perform for real users, on real hardware, over real Internet connections — so out they went, blinking and staggering, into the harsh light of the real world.
Testing ideas on part of the audience isn’t anything new. We frequently run multivariate tests using Optimizely and other tools, for example to determine the best design for the play button on our video player. Optimizely segments the users and overwrites the standard code with the designs we want to test. We monitor the stats to see which one performs best, and then implement the winner.
For the article and gallery pages, we needed a different approach. One of the key goals of the current project is to make the pages faster, and you can’t test load speed by sending users the existing (slow) page, then overwriting it with the new (faster) code. Instead, we had to split the users as the page requests came in, then send out new pages to 1% of them. We did this using Akamai’s Audience Segmentation Cloudlet, which splits the users and then cookies them so they get a consistent experience.
New pages? Check
Audience segmentation? Check
Ready to go? Not quite…
Simple complexity
If you want a team of 100 people to move in the same direction, they need a goal that’s clear and easy to remember. “Serve the new pages to 1% of the audience” meets those criteria.
I could fluff that up to a user story:
As a grumpy and sceptical product manager, I want to serve the new pages to 1% of the audience, so I can see what we need to fix
Or I could write it like this:
Given a user is in the test segment
When they view an article page
Then they should see the new design
Both formats have their uses, but if you want focus a team, simpler is better, so “Serve the new pages to 1% of the audience” will do.
Do I really mean that? Of course not.
There are complexities around the new pages, and even more around the 1%. Do we serve articles with features that aren’t supported yet? Yes, but with appropriate messaging. Do we include people with old versions of Internet Explorer? No, because the new pages have some browser-specific issues that will affect their experience and skew the data. Do we include people on our network, who might get confused and raise support calls? Yes, we want their buy in and need their feedback, so we’d better let them know about the test. Do we have the right analytics in place to measure the impact. Hmm, work to do there…
I could write those things down as acceptance criteria, but that’d turn a simple statement into an essay. As a Head of Product managing a handful of product managers, it isn’t my job to spell out the requirements or prescribe the solution. The product managers and engineers in our cross-functional teams are closer to the details and better placed to work things out. If I start doing their jobs, what are they supposed to do?
I like to do stuff. I like to build things, solve problems, turn new ideas into something tangible. That’s not my job, though, unless we’re going to count creating a presentation to persuade a group of senior stakeholders that releasing a set of unfinished pages is actually a good idea. My job is to make complicated things simple, so my team can make them complicated again.
It boils down to two things:
If I can’t state the goal in simple terms, I won’t get what I want
If I can’t trust my team to work out the details, I’m not managing them right
What they said, and what they did
So, we served the new pages to 1% of the Mirror audience, and in June we started serving them to 1% of the Liverpool Echo audience as well.
We invited users in the test group to tell us what they liked, and what could be improved. Some of them said nice things:
Looks more professional and sharp in terms of design. Layout is clear.
It’s much quicker to load.
Some were very specific:
How do I print off a crossword? It falls over two pages now, and won’t print off.
I don’t understand why the menu bar along the top keeps disappearing. It is important to have it permanently there.
Auto hide the header, it is distracting.
And, of course, some of them hated it:
Stop trying to make it look hip and modern. Why is everything BIG? It looks stupid and it makes the page difficult to read.
Go back to normality please.
Given people don’t like change, and are more motivated to complain than to praise, the overall mix was encouraging.
User feedback provides helpful insights, but what users actually do is equally important. Throughout the trial, we’ve been tracking how many people click the ‘Go to old site’ link and drop through the escape hatch to ‘normality’. Fewer than 1 in 500 people are taking that option.
There are nuances and complexities to the data, and there’s more work to be done, but if I frame the results with the same simplicity as the test, the conclusion is that users like the new experience better.
