When I was at the Guardian, we used to do a ton of user testing. Not just on the website, where I worked, but on our native apps, subscription and events businesses, dating and jobs products and tools for journalists. We even had one of those rooms with one-way glass. Creepy.
One of the challenges when you’re testing a news product is that the content and context is really critical to the experience. If you’re designing a feature around breaking news, you probably need to mock up a scenario that’s going to feel real to the user. If you’re playing with a new content format, you’ll get more useful feedback if you have a version using the kind of content the user would actually read.
Tuning tests to get the most actionable insight was something we did a lot of. The idea was usually to remove the content itself from what the user was talking about, and focus them on what we’d think of as the “producty” bits of the product. User experience, functionality, look and feel, usability.
Despite our efforts though, most people would go on and on about the content. You’d show someone a new live blog template with data from a recent football game, and they’d comment on the writing, or say they really loved this or that author, and why didn’t we have more from her.
Over time, I came to see this as a problem not with how users were talking, or how we were running the sessions. It seemed like a problem with how we thought about product in an established company like the Guardian.
Users saw our product as one experience, blending content, platform, distribution and monetisation. To them, it was all just the thing they got when they tapped or picked up the Guardian. They fed back on all of it.
Internally though, we had all the usual divisions of responsibility typical of a legacy media org. The church (editorial) and state (commercial) divide continues in many news companies, often with a third entity called “product” or “digital” added to the mix.
So back in the UX lab, that team sitting there listening to the user’s feedback is probably composed only of digital folks. Talking to them about the content, or even what kind of ads are being shown, is like telling a ship’s captain to calm the sea.
(This isn’t to say digital people are necessarily disempowered. UX quality has gone through the roof since news companies started hiring pros. And for now digital folks have way more data and their fingertips than anyone else at the company. They have a good measure of influence, just not much direct control.)
If the skills in the product team matched how the user saw the product, there’d be no insight wasted from the process of building it. Editors and business people would be right there listening. Even better, decisions which might have taken a long time to get to (because you have to go find somebody who knows about that, beg their time, get referred to their boss, etc.) could be made within the team. It shifts what BuzzFeed’s Alice DuBois calls the Wrangle-to-Work Ratio in your favor.
Best of all, it would encourage folks on the team to think big. You would know your team has the skills to play with the whole beast, from what you publish to when you publish to how and how long and to where and what it looks like and where it’s written. Rather than zeroing in on the little bit someone told them was their job, the team can think about the total experience of what they’re building.
There are some non-trivial logistical challenges to working this way, which are boring to go through and probably surmountable. Today’s reality— product in a silo — will play out much worse. And it’s why the teams involved find it hard to stomach listening to users who don’t see it that way.
