Highlights from the conference formerly known as Conversion Hotel

What I learned about conversion rate optimization and business growth (bonus: further reading)

Remco Wietsma

Follow

Published in

Kaartje2go

19 min readDec 2, 2019

--

This year I had the honor to be one of 250 attendees at the sixth edition of the conference formerly known as Conversion Hotel. colloquially named Experimentation Island (it takes place on the dutch island Texel), this event brings attendees and speakers together to discuss conversion, data, A/B testing, and customer behavior. In 48 hours keynotes and unconference sessions are interspersed with late-night bar sessions and social talks for a deep dive into the world of conversion optimization.

TL;DR: The weekend was fun. I’ve learned a lot.

Readers guide

These are highlights from the keynotes and break-out sessions given at Conversion Hotel. These sessions provide a guideline for this document. While I intended to discuss just the keynotes, I ended up including many noteworthy sources, and even added a few of my thoughts too, which I hope you’ll find inspiring. I recommend not to read it in one sitting (unless you skip all the interesting references).

The intention is not to give complete coverage. Even so, if I missed a speaker or skipped a key idea, I apologize in advance. Nonetheless, it should give you a chance to learn new things, explore different topics and study mental models and if that’s not your appetite, it may serve to convince your boss you should be at the 2020 edition.

I’m not in any way affiliated with Conversion Hotel or any of its partners. Just an attendee with an attitude of life long learning.

Enjoy and happy learning!

Topics discussed:

Make better choices
Something to think about: learning from others
Embrace failure
Intermission 1: Stage fright
Friction
Intermission 2: The Pacman rule
Marginal utility
Something to think about: The cost of measuring
A/B Retesting
Something to think about: Uncertainty principle
Sample Ratio Mismatch
Something to think about: Who are we?
Artificial intelligence
Authentic intelligence
Recommended books
Reference list

Make better choices

It’s very fitting to open with Stephen Pavlovich’s statement: “To make better products we need to make better choices.”

When you’re in the business of optimization and experimentation, you’ll likely make decisions regularly: Which tests to run, what to implement and how to cooperate effectively.

Decision making is hard. As Shane Parish states on Farnam Street Blog: “Even smart people are awful at making decisions. Think about decisions like these:

Napoleon deciding to invade Russia (and Hitler doing it again 130 years later)
An editor deciding to publish O.J Simpson’s If I Did It
Chris Webber choosing the timeout he didn’t have in the 1993 Final Four
NASA’s decision to ignore the O-ring issues on the Challenger.”

We base many of our decisions on wrong information, shallow knowledge or biased evidence. Evidence like editorials, case studies, and ‘expert’ opinions have a high risk of bias and are at the bottom of the hierarchy of evidence pyramid. The evidence makes us feel confident, but may not properly display the facts. This is harmful to good decision making.

**Hierarchy of evidence pyramid** designed by *Patrick Teunissen*

Fortunately, Stephen provides us with four strategies to make better decisions, run accurate experiments and build successful businesses:

Solve the biggest problem
Be bold
Test early, test often
Start small and scale

Solve the biggest problem

Charlie Munger once said: “All I want to know is where I’m going to die, so I’ll never go there.” This principle was derived from German mathematician Carl Gustav Jacob Jacobi.

“[Jacobi] knew that it is in the nature of things that many hard problems are best solved when they are addressed backward,” Munger counsels.

Inversion is one of the best mental models you can apply in life. If you want to improve, you need to invert. See what’s hurting your success and remove that problem.

With conversion rates, you’ll often encounter a multiplication by zero effect. 1000 x 0 = zero. It doesn’t matter if you add another 1000 or 100.000 visitors, if the conversion at any point reaches zero you’ll end up with nothing. Combining both mental models will help you in your search to solve the biggest problem.

Be bold

Sometimes we need to make bold decisions. While the strategy is simple to understand, practicing it is much harder. The Decision Matrix proves a useful tool, again from Farnam Street:

**Decision Matrix** designed by *Patrick Teunissen* *| Copyright* Farnam Street

My strategy for triaging was simple. I separated decisions into four possibilities based on the type of decision I was making.

Irreversible and inconsequential
Irreversible and consequential
Reversible and inconsequential
Reversible and consequential

Do you want to make bold decisions? Make sure they are reversible, inconsequential or a combination of both.

Test early, test often

You ran your test, waited six weeks for the outcome and now the results are finally in. You open the A/B tool and your test comes up negative. You ponder: “How could that be? The hypothesis stated it would be a winner?” But your case could still be a winner, so you find a different angle and now the test is successful. At least now all the hard work wasn’t for nothing. You go on and implement the successful test.

Why would you operate in such a way? Aren’t we rational professionals?

Unfortunately, there’s a powerful psychological effect at play. We hate losses and don’t want to be wasteful. This may cause us to make irrational decisions, based on our emotional engagement to prevent losses.

If you want to be a valuable experimenter, you must watch out for the puppy dog close. Don’t fall in love with the puppy or you’ll never bring it back to the shelter. A test is merely a test, or at least that’s what it should be. Accept it failed, view it as a sunk cost and move on.

Start small and scale

In experiments, it’s easy to think big. But a complete overhaul is seldom a good idea. This black and white, all-in thinking is a normal human tendency, but it leads nowhere. Often it’s much better to start small and scale. It’s okay to think big, as long as you start small.

Something to think about: learning from others

Are you agitated by other’s failures? Do you feel threatened by other’s success? Regard each human as a teacher, his story as a lesson. It will make you a better person.

I regard it as a criminal waste of time to go through the slow and painful ordeal of ascertaining things for one’s self if these same things have already been ascertained and made available by others.
— Thomas Edison

Embrace failure

I thought it telling that the session “share your failures” had so few attendees. If you are an experimenter or run A/B tests regularly, you are familiar with failing.

But we can’t seem to accept failure as a teacher. As Robert Cialdini mentions in his book Influence, The Psychology of Persuasion, we don’t like to be associated with bad news. No one wants to be a bad messenger.

You’ve probably heard about shooting the messenger. And while not literal, sharing failure can feel like an execution. This is regrettable. First of all, we’re all humans, we all fail sometimes. Second, failure is a great opportunity to learn, and often a better teacher than success. If only we could look at failure like Thomas Edison did.

I have not failed. I’ve just found 10,000 ways that won’t work.
— Thomas Edison

Failure boards and failure meetings

What if we see failure as a learning opportunity? How can we learn from other people’s mistakes? One way is to have failure meetings. This is very common in the medical world, where morbidity and mortality (M&M) conferences are regularly held.

M&M conferences involve the analysis of adverse outcomes in patient care, through peer review. The objectives of a well-run M&M conference are to identify adverse outcomes associated with medical error, to modify behavior and judgment based on previous experiences, and to prevent the repetition of errors leading to complications.

You can set-up an M&M meeting in your organization yourself (although you should probably change the name and topic). With a few simple guidelines, you can have successful failure meetings:

The topics should revolve around failure (not success)
The environment should be safe (no judgment/consequences)
The meeting should be voluntary and open to all teams (no obligations)

If this is a step too far, at least make sure you have a failure board (not digital) in a regularly visited place.

Success is not final, failure is not fatal: it is the courage to continue that counts.
— Winston Churchill

Intermission 1: stage fright

We all have to present sometimes. Whether it’s for ten, hundred or thousand people, being out in the open can feel terrifying. Although Michael Aagard wasn’t a keynote speaker this year, he was still generous enough to share a few useful tips on stage fright.

Stage fright is normal, even a top keynote speaker like Michael is terrified to go on stage.
Anxiety is your body helping your brain with something important to you, something you care about.
Save some brain capacity with intensive practice of the first two minutes of your talk. This will help you get through the most frightening stage, while also leaving a good first impression.
Create a ritual before you have an important conversation. This ritual is highly personal, you can arouse yourself, meditate or whatever works for you.

Friction

When you’re in the business of optimization, you’ll come across friction. Before diving into Roger Dooley’s keynote talk, let’s first examine the model of mechanical friction, and look at it’s up- and downsides.

Friction is desirable and important because:

[It supplies] traction to facilitate motion on land. Most land vehicles rely on friction for acceleration, deceleration and changing direction. Sudden reductions in traction can cause loss of control and accidents.

Friction is undesirable because:

[An] important consequence of many types of friction [is] wear, which may lead to performance degradation or damage to components.

Lower friction > faster growth

Friction is everywhere. In nature but also online. Even the best-optimized check-out process has some form of friction.

Customers don’t like friction and will abandon the check-out process if it costs too much effort.

Animals, people, even well-designed machines will naturally choose the path of least resistance or “effort”. Optimizing the path of least resistance and removing friction results in better conversion rates and faster growth.

B-MAT

For a behavior to occur we need motivation, ability, and triggers. When a behavior does not occur, at least one of those three elements is missing.

**Fogg Behavior Model** designed by *Patrick Teunissen* *| Copyright* BehaviorModel.org

Ability
By removing friction, we essentially provide more ability. A thing becomes easier to do. This increases the odds of successful triggering.

In essence, the colors we use can produce a psychological feeling of friction. The red color of a back button is associated with unwanted behavior, adding perceived friction. A button with green color, on the other hand, has much lower perceived friction, pushing the customer forward in the check-out process.

Remember the desirable features of mechanical friction? Sometimes we want friction to steer a customer in the right direction. For instance, a difficult HTML form may lead to more phone calls with a better conversion rate.

Motivation
Some customers will accept more friction. The motivation to complete a task is higher even with limited ability. These customers are loyal to your business. Loyal customers are somewhat irrational. Their behavior to follow a path of resistance is unconscious and their willingness to buy is emotional. That’s not to say you shouldn’t work on removing friction.

Loyalty = emotional, unconscious, irrational
Think of it this way: motivation is based on an external locus of control (you have little influence over it) whereas ability is based on an internal locus of control (something which you have greater influence over). If you want long term success, provide an effortless customer journey.

Friction detection

Detecting friction can be done in a myriad of ways:

Be your customer (buy your products)
Use event tracking (measure rage clicks)
Collect analytics data (time on site, bounce)
Ask customer service (frequently asked questions)
Don’t be afraid to collect messy data (quantitative research)
Measure Customer Effect Score (poll data)

Internal friction

When reducing friction, occasionally you will run into resistance from co-workers, different teams or management. If their reply is somewhere in the line of “friction doesn’t matter”, tell the story about Amazon’s one-click button and how Apple paid Amazon to use the patented one-click checkout. After you finish the story ask them this question:

Are you smarter than Steve Jobs and Jeff Bezos?

Intermission 2: The Pacman rule

When it comes to friction, there’s no better example than joining a group of people talking. A closed circle of conversationalists is especially effortful to join. You need high motivation.

Most of the time, the group, has no desire to keep you from joining the conversation, so how could they provide a greater ability to join?

Eric Holscher thought about this as well and invented the Pacman rule. Keep the circle a quarter of the way open, so a new member can join the conversation. Make sure the circle stays open by making the circle a bit larger. So next time you’re the member of a group, apply the pac-man rule. Be inviting and I promise you’ll meet some nice people along the way.

Marginal utility

Marketers, data scientists, and CRO specialists beware of marginal utility. What is utility? It’s the satisfaction or benefit derived by consuming a product. Marginal utility is the change from an increase in consumption of the good or service. In other words:

Your first sandwich will still your hunger more than the second sandwich, or the third.
In much the same way, your first successful A/B test will yield a higher conversion rate. While your 20th A/B test will result in a smaller increase, and a 50th might have no benefit at all.
Likewise adding more data, our spending more on ad campaigns has a diminishing return on marginal utility.

Sometimes you have to hit a threshold before you reach any utility. For instance, writing this post as a draft has no benefit. Even though it took me several days to write, it didn’t have any value until I hit the publish button. Aiming for perfection prevents you from reaching utility.

From nothing to something is a finite distance
From something to perfection is an infinite distance

Don’t be fooled by the man with the hammer. Running A/B tests, gathering data and utilizing advertisement is certainly necessary. Combine marginal utility with compounding effects for better decision making.

Something to think about: The cost of measuring

We live in a world with millions of data points. Everything online can be measured, but should you?

What gets measured gets managed, yet often we measure too much, which adds huge opportunity costs to management.

A/B (re)testing and seasonality

**Seasonality** designed by *Patrick Teunissen*

The Dutch have a saying: “Voor niets komt de zon op” which means the sun is the only thing [that rises] for free.

A/B test is not free and has its opportunity costs:

Doing one thing means not being able to do another. We live in a world of trade-offs, and the concept of opportunity cost rules all. Most aptly summarized as “there is no such thing as a free lunch.

Time might be a reason not to retest, it’s also the best argument why you should retest. Seasonality could lead to subtle differences.

Seasonality

In his unconference session, Kevin Heiner talks about the top 3 most common blindspots in CRO programs, one of which is seasonality.

Seasonality is the presence of variations that occur at specific regular intervals less than a year, such as weekly, monthly, or quarterly.

Different time = different emotion
Different time = different people
Different time = different market

Seasonality is predictable, the influence on A/B test is not. On top of that, cyclical patterns are even more unpredictable modifiers.

Simply put, timing is everything. Revisiting your A/B tests may be worthwhile.

Something to think about: Uncertainty principle

Even though this quote is wrongly attributed to Heisenberg, it is still too good to pass up on.

When you measure something it changes
— Heisenberg

Think about the tools you use. How much friction do they add in the form of flickering, load times, etc? It’s good to be aware of the observer effect.

Sample Ratio Mismatch (SRM)

**Sample Ratio Mismatch** designed by *Patrick Teunissen*

In statistical hypothesis testing, the p-value or probability value is the probability of obtaining test results at least as extreme as the results actually observed during the test.

How likely is it that what is measured in a small group also counts for the entire population? For A/B tests, to make an appropriate decision based on the data, you want both buckets to be the same size. If there are 500 people in group A, you want 500 people in group B.

Of course, this is not as easy as it seems when using random selection to appoint a visitor to bucket A or bucket B. Use Lukas Vermeer’s SRM checker to help you discover if there’s a case of Sample Ratio Mismatch.

Lukas Vermeer had an excellent keynote about SRM which he called the one-neat trick to run better experiments. If you work with data, I urge you to read this paper he co-authored.

SRMs can happen due to issues with experiment design, the randomization process, variant deployment, data logging, data collection, data cooking, as well as due to data analysis conditions.

And

Many SRMs were caused by a human intervening into the experiment and unknowingly breaking it. This could be prevented by introducing controls that show warnings or block users from making changes to running experiments.

Something to think about: Who are we?

Humans have exceptional qualities that are found nowhere else. From a philosophical standpoint, our ability to choose is rather unique.

Who you are is defined by what your willing to struggle for.
— Mark Manson

Artificial Intelligence

Artificial intelligence is a hot topic. While AI has its uses in regression, classification, recommendation, image recognition, clustering, and anomaly detection, we shouldn’t be afraid they take over our jobs.

Machines are great for large data sets and personalization, they are generally better at classifying, recommending, clustering and recognizing. AI learns in one of three ways:

Supervised learning
Unsupervised learning
Reinforcement learning

Supervised learning — Right answer is known

For supervised learning, human validation is required until an inferred function is produced, which uses inductive bias to predict unseen situations in a reasonable way. On such example is classification, where humans determine correct values.

Reinforcement learning — Only the outcome is known

Reinforcement learning differs from the supervised learning in that labeled input/output pairs need not be presented, and sub-optimal actions need not be explicitly corrected. Instead, the focus is finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge). AlphaZero Go is built with reinforcement learning.

AI needs empathetic humans and creativity, to solve problems human can’t solve.

Unsupervised learning — Right answer is unknown

Unsupervised learning is commonly used in cluster analysis. Instead of responding to feedback, cluster analysis identifies commonalities in the data and reacts based on the presence or absence of such commonalities in each new piece of data.

This technique can be used for image recognition, where each image is checked against a set of images wherein a pattern lies.

AI doesn’t care if it fails and will do so thousands of times, before a model is sufficiently trained.

Read this article if you want to learn more about Machine Learning.

Why people always beat computers

Humans have what Elke Geraerts calls authentic intelligence, which makes us uniquely qualified compared to robots.

If robots’ strength is artificial intelligence, then human strength is authentic intelligence. We, humans, have four distinct characteristics which make us unique.

Reprogramming
Willpower
Imagination
Relationships

Reprogramming

Who you are is 60% defined by nature and 40% by nurture. Warren Buffet, one of the greatest American investors call this the ovarian lottery:

I’ve had it so good in this world, you know. The odds were fifty-to-one against me being born in the United States in 1930. I won the lottery the day I emerged from the womb by being in the United States instead of in some other country where my chances would have been way different.
— Warren Buffet

In the same way, your intelligence and physique are shaped by who your parents, and their parents, where. It defines what your tendencies are and how big a chance you had to get into a great school. Most of us have all been very lucky.

What’s even better, 40% is nurture, so we can reprogramme ourselves. That’s why your mindset matters. Your brain is a muscle that, like any other muscle, requires training.

Dare to say: I am unfinished

Willpower

To reprogram, you need willpower. Unfortunately, we are ill-suited for this. There’s a chimpanzee in your brain that wants to procrastinate. Your monkey brain doesn’t want to work and increase performance, it wants you to eat, have sex and it’s very complaintive.

In the Netherlands and Belgium, the term “Calimerocomplex” is used to denote people who are staunchly convinced that their position as an underdog is due to their smaller size, either literally or symbolically, which covers up for their shortcomings. Often the character’s lines from the show are cited,

They are big and I is small and that is not fair, oh no!”
— Calimero

Furthermore, we’ve become a yes generation. Increasingly distracted by our mobile phones. 150 times a day we look at our mobile a recent study shows, on average 3 hours per day. Although there is a counterstudy showing significantly lower phone check-ups per day, the main argument still applies. We need to counterattack our attention deficit and beat procrastination:

1. Attention ritual
The best athletes in the world have attention rituals. It’s how they mentally prepare for big games and save precious brainpower to gain a competitive advantage. Rafael Nadal is a great example. Do you want to beat your opponents? Stop wasting your willpower and set up attention rituals

2. Kill the elephant
We are easily distracted. Many of us start each day by opening our inbox and engaging with whatever finds our attention. Because we follow up on our emails, there’s not enough time before lunch and in the afternoon we face a post-lunch dip. By then it’s too late to finish anything. We leave the elephant in the room and vow to hunt it down the next morning (or so we say).

When you’re in the game of hunting elephants, don’t get distracted by rabbits.

Don’t start your day working on small tasks. Don’t open up your inbox and don’t get distracted. The morning is the best time to complete critical assignments.

If you want to read more about this, I wrote a post about morning rituals and how to make your mornings rock, not suck.

Creativity

The third advantage we have over computers is our creative ability. Our imaginative ability has allowed us to construct civilizations, build skyscrapers and invent artificial intelligence. Our mind wanders between 30–50% of the time, making connections that no computer or animal ever could.

Furthermore, a tired brain is more creative but cognitively impaired. That’s why you should divide your workday into two parts.

The morning for cognitive tasks
The afternoon for creative tasks

We have a brilliant mind capable of abstract or concrete thinking. In social psychology, this is termed construal level theory. High-level construals are a way of thinking in a big-picture way or abstractly. At the low level, more emphasis is placed on how the situation is different from others.

If you want to perform better as an individual or if you want to understand customer behavior, I highly recommend the Wikipedia entry about construal level theory.

We can steer our thinking, just like Charlie’s big glass elevator in Charlie and the Chocolate Factory.

Relationships

The fourth advantage humans have over robots is the ability to form meaningful relations.

We form teams, groups, and tribes in which we find trust, communication, and identity. It’s the reason we find our why.

Relationships are the best medicine against social disorders. A machine left alone will never cry. Your phone doesn’t care if you touch it. But we do. That’s what makes us special.

For the Dutch-speaking, there’s an excerpt version of Elke Geraerts authentic intelligence talk available on youtube.

As you have read above, the conference formerly known as Conversion Hotel was a great inspiration for me. If you got this far, congratulations! I hope you enjoyed every minute and learned a thing or two. If you’d like to further explore topics, sift through the reference list or read the recommended books listed below. If you have any questions or if you’d like to learn more about me, feel free to contact me.

Recommended books:

If you want to dive deeper still and learn more about the topics discussed in this post, I recommend these books:

Authentic Intelligence

Authentic Intelligence, Elke Geraerts (dutch only)

Friction

Friction, Roger Dooley (buy on Amazon)

Psychology

Habits

Procrastination

Eat That Frog! 21 Great Ways to Stop Procrastinating and Get More Done in Less Time, Brian Tracy (buy on Amazon)

Business