Automating AB-testing boilerplate generation with a CLI

At Hootsuite, we run a lot of AB tests. These tests require a lot of boilerplate code. During my co-op at Hootsuite, I had the opportunity to automate that process with a Command Line Interface (CLI).

AB Testing Boilerplate

AB testing is essentially an experiment where two or more variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal. (Optimizely)

An experiment has a few steps.

1. Audience Targeting: Decide if the user should be part of an experiment. For example, we might want to only target certain plan types.
2. Bucketing: Decide which variation (Control vs variation A) the user should see, and trigger that variation for the user. Sometimes we re-render the page, set cookies, or set a few variables for that user.
3. Statistical analysis: Compare metrics between variants such as how many times a button was clicked or retention rate.

Once a user is bucketed, we trigger a variation using a script . We created an AB-testing project skeleton/template (“skeleton”) for that script. The skeleton contains a lot of boilerplate code, things such as polyfills, tracking, and event dispatching.

In the project skeleton, there are several variables that developers have to rename. The same, or very similar edits have to be made multiple times. This process is tedious and prone to copy-paste errors.

At the end of 2019 we refactored and updated the skeleton, and reduced the boilerplate code required for each experiment from 400 lines to less than 150 lines. I had the opportunity to further reduce that to under 100 lines of code. This made the skeleton much easier to read and navigate.

However, there was still room for improvement. To reduce copy-paste errors or developers forgetting to rename variables, I built a Command Line Interface (CLI) that automatically generates a fleshed out skeleton.

A Tutorial for Creating your own CLI

In this tutorial, we will cover the following:

• Introduce the basics of creating a Command line interface
• Make a Node.js CLI that can generate the code for a webpage
• How to use template files
• How to write/save files to your computer

At the end of the tutorial you will be able to modify the final product to generate other types of project skeletons.

Hopefully you already have Node installed. We will be using two libraries for this. Run yarn add inquirer mustache or npm install inquirer mustache to install the libraries. (See yarn vs npm). I will use yarn in this tutorial. If you don’t have yarn, get it here or use npm, which comes with Node.

Inquirer is a library that makes building CLIs very easy. enquirer and prompt are very similar, but I picked Inquirer because it seems to have more people using it.

Mustache is a templating library that helps us generate code from templates. Alternatives include handlebars and pug. Mustache was the simplest library that met my needs.

Create a fileindex.js with the starter code. I’ve created mine inside a subdirectory cli/

cli/index.js

A few things are happening here:

#!/usr/bin/env node

A shebang line: the very first line in an executable plain-text file on Unix-like platforms that tells the system what interpreter (Node) to pass that file to for execution.

const fs = require(“fs”)

fs is a built in Node module for file systems operations like reading and writing files.

const path = require(“path”)

Path is a built in Node module that provides utilities for working with file and directory paths.

Running our script

You can run the script from your terminal using node ./cli/index.js. However, it is best to run our command with a package.json script. If the file path or directory were to be changed, it can still be run using the same command such as yarn gen. Mark the file as executable using chmod +x ./cli/index.js to avoid permission errors. If you don’t already have a package.json file, use yarn init to generate one. Then add the following to your package.json file:

“scripts”: {
    “gen”: “./cli/index.js”
},

Now our script can be run using yarn gen.

For my use case, the CLI was only needed in one repo. If you want your CLI to be available globally (can be run from any directory), you can add the following to package.json

“bin”: {
    “generate-project”: “./cli/index.js”
}

In most cases, we use yarn global add {package} to install a global package, but because we haven’t published our package yet, we need to link our local directory as a global package. Run yarn global add $PWD (npm install -g ./) to link your current directory to the global npm folder. Now your command generate-project is available globally.

Asking questions with Inquirer

Inquirer provides a method prompt which will prompt for each question in an array. It returns a promise containing the answers to those questions. There are different input types such as true/false or string available. See https://github.com/SBoudrias/Inquirer.js#question for a list.

Let’s add another question. What is your favourite hex code?

cli/index.js

Creating files using fs

Now that we have user input, we can start generating project files! For this example, let’s generate a simple HTML page. You can generate any kind of file or code. At Hootsuite, the CLI that I created generates JavaScript code.

Let’s create a new folder for our project with fs.mkdirSync, which takes a path as a string and creates a new folder. Set recursive to true if you are creating nested folders. (Note: the recursive option requires a node version >=10.12.0. If you run into Error: ENOENT: no such file or directory, mkdir, try upgrading Node.)

cli/index.js

Let’s create a new method called createHtmlFile

cli/index.js

fs.writeFileSync creates a file using file name/path and data. Here we are passing a formatted string as data. This isn’t ideal but we will clean this up later using mustache templates.

And a createCssFile. I’ve added a line to check if the hex code provided includes a #, and an appending one if it does not.

cli/index.js

Templating with Mustache

Using formatted strings is pretty ugly and hard to reason about or modify. Mustache allows us to use templates to create our files instead of hard-coded strings.

Create a new file index.html.mustache. I like to use a subdirectory for all the templates, ie templates/index.html.mustache

Both IntelliJ and VSCode automatically recommend plugins to support mustache templates, which should make editing the HTML and CSS files much easier than editing formatted strings.

I’ve added more usages of the name variable. This is a good example of why templating is preferred over changing the file by hand. As the file grows, and as the process is repeated many times, the chance of making a mistake is quite high.

cli/templates/index.html.mustache

We will use the method mustache.render which takes a template and a view object that contains the data to be used in the template.

createHtmlFile becomes

cli/index.js

Because of some nuances of where the script is run from, fs.readFileSync needs a full path to read a file. path.resolve helps us get the full path to the file we need. Then, after getting the template, we apply our variables to the template with mustache.render.

And for our CSS file, create cli/templates/styles.css.mustache

cli/templates/styles.css.mustache

createCssFile becomes

cli/index.js

It is good to note that Mustache is ‘logic-less’. All logic, such as appending a # to the hex should be done before rendering the template.

As you can see, there is a bit of code duplication between our two create functions. I’ll leave it up to you to reduce that :)

The final product

Now try running yarn gen and enter your info…

What is your name? Ian
What is your favourite hex code? 078ee8

Let’s see what we’ve made!

cli/pages/Ian/index.html

Hooray! We’ve generated a webpage based on our CLI tool with variables. Next, try adding more questions to the CLI and adding more content to the page.

This example is a trivial example, but the process can be applied to other types of projects, such as generating boilerplate code project skeletons.