Sign in

Growth Book
(image by peoplecreations — www.freepik.com)

There are many prioritization frameworks out there for A/B tests (PIE, ICE, PXL, etc.), but they all suffer from one critical problem — subjectivity. The reason A/B testing is so powerful in the first place is because people are really bad at guessing user behavior and the impact of changes. Why are we then using those same bad guesses to prioritize? In addition, many of these frameworks place far too much emphasis on effort — how long a test takes to implement. Except in the rarest of cases, the time it takes to implement a test is far shorter than…


Tesla rolled out a new user interface this year, and the reception was mixed to quite negative. Although I am not as bothered by most of the changes as others, it actually highlighted a problem that Tesla has had with their user interface that has bugged me for some time: mixing states and actions.

Tesla’s designers are faced with a problem that designers have been solving for years. How can you clearly show the state of the system, and how to perform an action to change the state? …


Multi-Arm Bandit testing comes from the Multi-Arm Bandit problem in mathematics. The problem posed is as follows: given a limited amount of resources, what is the best way to maximize returns when gambling on one-arm bandit machines that have different rates of return? This is often phrased as a choice between “exploitation,” or maximizing return, and “exploration,” or maximizing information. When applied to A/B testing it can be a valuable tool. …


As an engineer, I like to build things, but as a practical business matter, it’s usually less straight forward. As much as it pains me to say it, these days for A/B testing platforms it is usually more prudent to use a 3rd party platform. In this article we’ll discuss pros and cons of building vs buying A/B testing platforms.

Building

Building your own A/B testing platform has a few undeniable advantages. If you have an engineering team that has the ability to take this on, the tight coupling to your code and deployment systems can create a great developer…


This article covers how to run A/B tests at different levels of your tech stack and the pros and cons of each option.

Front-End

This is the most common and easiest method for running A/B tests and is used by Optimizely and Google Optimize. This approach is best for testing subtle front-end changes or things the user doesn’t see immediately (e.g. something in a modal or below the fold).

Front-end A/B testing works in the following way:

  1. Randomly assign the user to a variation in a consistent way (with cookies, local storage, or hashing).
  2. If the user is not in the…


The other day I was speaking to a friend of mine who lamented that he didn’t really believe in A/B testing. He noticed over the years that at his company they have had plenty of successful test results but haven’t been able to see the effects of the tests on the overall metrics they cared about. He was expecting to see inflection points around the implementation of successful experiments. Quite often companies run A/B tests which show positive results, and yet when overall metrics are examined, the impact of these tests is invisible. …


Six years ago, my previous company had a problem. We had the habit of building large, time-intensive projects based on perceived market trends or tenuous research, and far too often these projects were unsuccessful at growing our business. We also had far too many meetings debating small details like the position of buttons or which colors to use, and some of these meetings even got contentious. Agile delivery counteracts a lot of these problems, but without the right systems and tools in place, we still ran into issues that slowed down progress to our goals. The solution we came up…

Growth Book

Building a platform for better data-driven decisions

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store