The Dawn of Conversion Optimization

The monolith from 2001: A Space Odyssey

Conversion optimization is still in its infancy. The job description barely existed when I exited college in 2005. A decade of wild growth has led to a broad market recognition of the concept and a lucrative job market (for which I am thankful). While the glamorous idea of split testing (and the math that goes with it) has won many adherents, the less glamorous idea that good testing requires administration and planning has lagged. We now have testing tools that are sharp enough for most tasks, but most conversion optimizers still have no idea how to properly design and build a testing program around the mathematical whiz bangery that is Optimizely (or other similar testing solutions).

There has been a startling tool gap: testers are expected to make do with tools designed for developers because program administration is seen as annoying at best, unnecessary at worst. This is a fast way to run a testing program into the ground, but no better solution exists and responsible testers in the meantime must make do with tools that require an inordinate effort to use for their purposes.

What might a dedicated test administration tool look like?

A long view of conversion rates, tied to events

Conversion optimizers are currently reduced to putting the big picture in word documents and power points. A good road map would serve these communication needs while continually supplying the test administrator with the data needed to take the long view in conversations with colleagues. I want to see at a glance how my conversion rate has been doing the last year or so and where I am against the projected goal for the current year. I want the system to automatically tie in the dates of significant wins so I can easily see how those are contributing toward my goal (and justify my work to my boss).

Currently these sorts of graph views are preserved in tools like Google Analytics and Adobe Analytics. This raw data is inadequate. It’s not contextualized, so a conversion optimizer (or data analyst at some firms) must contextualize it with relevant dates when building explanations for management. Since the internet is inherently a very noisy place this is often futile, but the effort is nonetheless required in order to validate assumptions and it could be made much easier.

A few key road map views of the funnel (with automatic and manual event tracking) would help bleary-eyed conversion optimizers focus on the long view every morning and would be an invaluable asset when the inevitable what happened when questions come around.

Hypotheses, results and conclusions, every time

A graph is not a road map by itself. A good testing tool would also include a clear ticketing system that would outline what was tested, what happened and why. This is not Jira. There are lots of developer tools designed to organize cards into sprints (like Jira or Trello) but these tools are really designed as digital notepads: take a quick note of what you’d like to do, tick it off and go on with the rest of the sprint. Testing that way leads to sloppy documentation and confused learnings.

The dirty secret of conversion optimization is that for most businesses Jira (or something like it) is the source of truth for test history. The poor documentation this encapsulates in most cases leads to a lot of difficult conversations with developers or designers about what they remember from months or years ago.

Could this be solved with additional Jira fields? Partially. While properly configured ticket fields (if used) allow a tester to document thinking, Jira does not integrate with the raw testing data necessary to draw informed conclusions. It’s difficult to explain why a decision has been made in the absence of clear data sets and screenshots. Sloppy documentation is expensive (in terms of knowledge and test opportunities lost) and a better tool would offer real savings.

What’s missing? Data, hypotheses and conclusions. Although extremely formal hypothesis writing is out of vogue, it’s always important to understand and explain testing rationale and expectations. This description may be brief but that additional context around test results is priceless when you (or your successor) returns to the data in six months or a year or two. Filling out a hypothesis should be required for any testing tool.

Test conclusions are the other side of the picture. They answer the question formed by the hypothesis through test data. A good tool would integrate cleanly with popular testing solutions and pipe raw test data right into the ticket (along with screenshots of each treatment). A test conclusion answering the hypothesis with data would be required to close the ticket and results would be tagged, logged and mapped to the conversion road map before being archived in an easily searchable manner.

Shifting, multi-factorial prioritization

Prioritization is another big problem with testing administration. Again, Jira is the default choice for many conversion optimizers, but it’s not good enough. Jira only allows single-factor prioritization. A card may be on top of another card in a sprint, but there is no contextual information about the relative value within the stack.

Jira rightly considers relative value inside deep stacks irrelevant for development, but testers must take a more nuanced view. Road map development requires continual progress along multiple testing themes and considerable prioritization agility where tests are ranked and re-ranked against shifting external prioritization factors. A Jira view leads to a tempting dependence on the memory of the test administrator — a faulty and selective way to run a testing program.

Testing well requires understanding the multiple themes in play in any given test. Imagine I have 100 test ideas and need to build an intelligent and flexible test road map. I have many factors to consider as I evaluate prioritization, including whether a test pertains to the new product we’ll be launching (+bonus points there), whether it moves the ball forward on mobile (+more bonus points), whether it touches a sensitive part of the funnel (++ points), whether it’s “above the fold,” and any number of other factors.

Current state-of-the-art in conversion optimization requires the development of an enormous Excel spreadsheet with test ideas indexed and sorted by rows against columns with particular weighting values. Test ideas are then keyed manually into Jira or another development tool of choice. This is awkward and unproductive (not least because the brightest and best idea often feels flat when logged as a single phrase in a spreadsheet).

A better solution would include prioritization factors in the ticket creation process. As a test administrator, I would be required to step through a screen where I ticked the appropriate boxes to indicate the key touch points affected by my test idea. The system would then weight and sum the values according to a formula set by the program administrator. The result would be a zen-like flow of test priorities, sorted by their total value (although other filters could of course be applied). Adding a little color and tagging would make it easy to see at a glance whether a particular test in the flow touched a key conversion theme (like mobile or checkout). Jira integration would allow one-click export of a selected ticket and pipe back in a status update and expected launch date.

None of this is impossible, but no tool is on the horizon either. What is lacking is demand. The field has matured to the point where most conversion optimizers can and do take the testing math itself for granted thanks to a series of enterprise class testing solutions. These are great for running individual tests but none of them offer compelling test administration features (nor should they). Conversion optimizers can and should demand their own test administration tool. It is time for the industry to evolve.