During our most recent AMA designer and author Nick Disabato answered your questions about how research-driven A/B testing can benefit your startup and more.
We compiled the most popular questions and answers here. Enjoy!
Q: What are some quick ways to get into conversion optimisation besides using heat maps as described in your recent newsletter?
asked by Marc Köhlbrugge
There are many ways to get research on a slim budget, little to no administrative support, and little to no time. 3 other easy wins:
- Google Analytics. Everyone hates it, but it’s insanely powerful. I strongly recommend configuring conversion-focused goals in GA and analyzing what browsers, devices, and platforms (e.g. mobile v. desktop) convert, relative to one another. Here’s ConversionXL’s deep dive on GA, should be useful.
- Running a survey. Wufoo & Typeform are cheap. You can configure a survey in 2 hours, let it run, and spend an afternoon picking apart the results.
- Usability tests. Pay UserTesting $495 for 5 tests on the company’s petty cash card. Failing that, make your entire team go through the whole checkout form with a dummy credit card, and write up at least 3 things the form could do better. If they complain about doing this, well… isn’t that a perfect sign that your checkout form is hard to use?
Q: You say that A/B testing is only to be applied to revenue generating transactions. Why is this the case when there are lots of sites who have a goal outside of generating revenue that they would like to optimise? For instance a company may want to design its landing page to maximise sign ups etc?
asked by Nikhil Shah
I recommend testing for additional signups if and only if you can ascribe an economic value to those signups. If you can do that, then it’s not a bad idea!
But you’re still running a business. A business exists to make more money than it spends. Towards that end, I run tests in order to increase revenue or decrease costs. Around your point, I don’t really believe in engagement or eyeballs as valid business metrics — there’s usually something money-driven beneath those.
Q: Why is there the need to ascribe an economic value to actions? For instance a newsletter may want to simply increase reach with the view to monetise that audience at a later date. So why the focus on knowing the economic value of the action you are optimising?
asked by Nikhil Shah
The focus on revenue generation is because I’d prefer to spend time, resources, and my clients’ money on actions that will create a positive ROI for their businesses. I’ve never encountered a client that was excited to drop 5 figures on something that didn’t make them any money.
Marc Köhlbrugge: To add to that, I think that even if you know increasing newsletter signups will eventually make you more money, it’s still critical to know the economical value of a signup, because otherwise you have no idea whether the ROI is there to optimise it and how it compares to other tests you could be running.
Say you spend half a day optimising the signup flow and get a 2% conversion increase. Was that worth it? Should you invest even more time? How do you know? Knowing the rough economical value of those signups will help you answer these questions.
Q: Which are the things you A/B test more? Text variation, design, or functionality?
asked by GitMarket
Test text probably more than anything else. It’s easy to put together and it’s easy to research. I consider functionality to be a design decision — how something works matters a lot more than how it looks. And usability matters a lot more when you’re testing e.g. changes to a checkout form.
But I wonder about your question itself. I suspect you’re wondering whether testing text or graphic design is going to yield a higher ROI. That’s totally valid, and I understand why you’re asking it, but the answer is always: it depends heavily on the research. It may be that your research uncovers significant issues with functionality. Then you have an opportunity to test it. You should always, always research first and then test later. You might be surprised at what you find!
Know who’s flipping great at A/B testing new copy, BTW? Joanna Wiebe: https://copyhackers.com She’s one of the expert interviews in my new course and super brilliant at what she does. Strong recommendation.
Q: Why is research so important?
asked by Kurt Elster
Research is foundational for understanding who your customers are and their motivations. So many business owners think they know this, and most really don’t. Research is also critical for vetting the success of design decisions — and shaping what those decisions should be.
Q: Do you create mock ups or wire frames for A/B tests? If so, are they vetted by the client? Do you replicate production environments locally before deploying tests or, like Bill O’Reilly, prefer to do A/B tests live? And at what point are you performing research? If it’s research driven, does research come first?
asked by Rich Cornish
Yes, especially for more ambitious reworks. 90% of my mockups are just pen & paper sketches in my notebook, though. The client will have a few questions, and then I’ll build a prototype.
I don’t really need to recreate a staging environment for most of my tests, especially those using the WYSIWYG editors in my testing framework. Both big A/B testing frameworks (VWO & Optimizely) allow logged-in testers to preview their work. I use those to QA my changes in as many browsers as possible. So yes, this is perhaps the only way in which I am like Bill O’Reilly.
I am always performing research first. Research is the most effective way I’ve found to create revenue-generating test ideas, bar none. That said, I often use test results as research for future tests! For example, VWO generates heat maps of successful variants, which I use as grist for future research. And it’s always great to have a corpus of test results — even if a test shows negative revenue, that’s actually helpful, because it shows we’re on to something.
Q: How do you deliver testing results to clients? Do you have weekly meetings to discuss progress or use something more one-way like email or project management app.
asked by Josh Frank
We have a shared Basecamp project and Slack room. I post monthly PDF reports in each — and if I call tests faster, I whip up a PDF and post it to Slack. I try to avoid email, because I’m usually communicating with a larger team and want a more permanent, widely-shared record of my work.
Q: Do you have a template where you put down your A/B testing results? Is there a specific service/software you use?
asked by Michel
Yes, I’ve configured a PDF template where I just fill in the blanks. I make the documents in Pages, nothing fancy.
Q: What type of feedback/questions do you get from clients after you submit your Revise Express report?
asked by Michael Bellina
As feedback I’ve gotten one “that’s it” in 3 years and 48 clients. “Wow” is far more common. Some people wonder how to make sense of all the recommendations — at which point I send them the template for my clients’ Trello board. Here that is, for reference: https://trello.com/b/7V53QBjl
Others want to get on a call to chat about big strategic things, which I’m happy to oblige. For example, I once recommended — strongly, in a 10 minute rant, with no uncertain words — that a SaaS completely wrecking-ball their pricing model and start with something else. That was totally worth getting on a call to defend. I’m sure they were surprised by my recommendation, especially since they had not settled on their existing configuration lightly.
Q: Can you describe the basic fundamental blocks of what you do? How is your approach to A/B testing different? How do you find A/B testing clients?
asked by letsworkshop
Let’s handle the questions in order:
On the basics:
- First, I analyze your funnel to determine any places that may be leaking revenue. Nobody has a perfect shopping cart that works across all browsers, mobile devices, etc. And nobody knows how to configure Google Analytics, so I put together the right goals in there.
- Then, I research your customers — by running heat maps, scroll maps, behavior recordings (following your pointer/finger, etc), and an “annual $COMPANY_NAME survey” that asks about past customers’ purchasing motivations. I also get your customers on the phone if they have interesting survey responses.
- Next, I configure a Trello board for all current & future A/B testing ideas. I fill the board with research insights and corresponding test ideas. For example, if there are a zillion customers clicking on a link — and they are unlikely to convert from that link — then I propose removing or reworking that link, and determining what the impact will be on the page. I spend the plurality of my job writing Trello comments & Slack posts, chatting with clients about what to test next and why.
- Then I configure and deploy a test. We should always have a test running. Once the test is ready to call, I analyze it, write a report, and kick off the next test.
On my approach being unique:
I don’t think it’s all that unique, to be honest. ConversionXL and other agencies do very similar things, and many are ahead of my game. I think the thing that keeps me unique is my background as an interaction design researcher — I am very good at teasing out customers’ motivations in user interviews. But others could be good at that, too.
On finding clients:
Mostly my mailing list, referral, and past customers of my books (abtestingmanual.com, cadence.cc, draft.nu/evidence, etc).
Q: What’s a typical day for Nick Disabato like? Do you wake up and reach for your phone, if so why? How does the rest of your workday come together?
asked by letsworkshop
Yes, I keep my iPhone on my nightstand. I’d like to say I check my phone first because I’m making sure no disasters unfolded overnight, but in reality I’m mostly just checking my email and RSS feeds, and reading my Instapaper queue. I’m just as addicted to technology as anyone. We all smoke our own dope in this industry.
I read for a couple of hours, and then I shower and roll downstairs. I spend most of my mornings working on writing, podcasting, and video recording. I spend most of my afternoons on meetings & client work. I walk my dog twice a day; Erin walks him in the late evening. I cook lunch for anyone who comes to my house every other Thursday.
That’s really it. I try to keep IRL meetings to a bare minimum. You can find me wandering my neighborhood during the summer and fall, when it’s nice out. I mostly work at home and dislike co-working spaces.
My friend Kai & I talk more about our daily routines.
Q: Would you say there is currently a large market for freelance A/B testers? How did you discover this market?
asked by letsworkshop
Yes, I would, although I would focus your positioning more around optimization than A/B testing. Think “mercenary who uncovers & fixes all the revenue leaks” and less “contractor who configures A/B testing JavaScript.”
I discovered the market by accident, to be honest. I began a service, Draft Revise, in mid-2012 and it sold out quickly. That told me that I was on to something. I had no idea!
Q: Lastly, at which point in a startup’s life does A/B testing become relevant, what should people start A/B testing first, and what kind of results could you expect?
asked by Marc Köhlbrugge
To answer them in order:
- A business should be getting at least 1,000 revenue-generating transactions per month in order to get statistical significance and a high ROI from A/B testing. Yes, that number is high — but it’s what you need to get really safe findings. Others recommend as low as 500 — I’ve even seen 200.
- That depends heavily on what you find from your research! And it also depends on your type of business, addressable market, etc. I wrote more about my research process in another AMA answer.
- You are running a marathon, not a sprint. You may not get results right away. Things might be broken, and bugfixing them takes time. I’d say a 10% bump in revenue is a reasonable outcome in 6–9 months.
Major thanks to Nick Disabato, you can subscribe to his newsletter here. Also make sure to check out his A/B Testing Manual for more insights.
If you enjoy articles like these please help us out with a 💚 below and follow us to get notified of new articles. 🙌
