Quant backing up the qual

Maz penguino
Aug 24, 2017 · 3 min read

At SEEK we’ve been running a lot of A/B tests in my teams. We’re focussed on optimising the create a job ad (CAJA) experience and in doing so, getting more jobs posted on SEEK.

Sometimes, we have some tests that result in some really interesting and surprising outcomes. One of our latest tests, dubbed ‘the pink test’ had one of these results. Here’s how I see it…

The survey.

We sent out a survey in response to a failed A/B test where we were trying to increase the uptake of our premium product aimed at hirers who have hard-to-fill jobs. We wanted to learn a bit more about why the test had failed and what it was that we needed to change to get a better result.

In this survey, whilst the results were much like the test and inconclusive, the most interesting and telling themes were found in the free text responses in relation to the layout and design of the page. We thought we weren’t being overt with our sales pitches and upsells but our hirers definitely didn’t think so…

“Pushy and undervalues classic ad”

“Nicer look, but still misleading as the standout is the default.”

“Don’t try the hard sell on me. If you want to upsell, let me not feel intimidated. Put your advertising elsewhere.”

“I don’t need a marketing pitch. I’ve already made my decision.”

Ouch.

In face-to-face research we hadn’t heard these strong sentiments in regards to this page before. The anonymity of a survey seemed to bring this to the fore and was fabulous as we were uncovering the idea that how we present the products can really affect someone in a highly emotional manner which may have a flow on to their purchasing decision.

Unfortunately, the project was wrapping up (for the moment), so I filed these learnings into the memory banks for a later date.

The pink test.

Control — pink version (left) and the New Experience (right)

The results.

The paring back of the StandOut initially led to a small decrease in people posting a StandOut ad which initially concerned us, but as the test went along, things moved a bit. By the end of the test, the number of ads that were posted in all the flows of the new experience had led to an increase of 4.12% compared with the control.


Lessons learnt.

I love this test and result as it’s such a great story…and has a few lessons in it for me.

  • Listen to what people are saying even if it’s super uncomfortable and may involve some difficult conversations with product owners
  • Selling too hard can really make people angry and not want to deal with you.
  • Quant backs up qual. How can we frame our qualitative research so it’s acted upon earlier?
  • A tiny visual change can have a huge impact so always A/B it!
)

Thanks to Mimi Turner

Maz penguino

Written by

Maker of web stuff, crazy cat lady, lover of travelling and design-y things, manic DIYer. Head of UX.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade