Split testing in an MVC Project

Background

Split testing (or A/B testing) is a way of objectively testing a new feature with your customer base to see if it is what is actually wanted.

The mechanism is to show a proportion of the customers a new feature (or improved existing feature) and measure whether these customers buy more or less (or whatever it is you want your readers to do) compared to the customers who see the original set of features (i.e. the control). The assumption is that your change is the cause of the change in customer behaviour.

More info on wikipedia.

The Goal

We wanted to be able to easily trial new features and screen layouts to see if our ideas actually made any real difference to the company. The context is an on-line shopping website, so our success measure was quite simple to define as a completed sale.

Do our ideas actually make any real difference?

Other things like newsletter subscription signups, or clicking through to a second article/page/product might also make good success measures.

The Approach

The technology stack used for the site is .NET C# MVC 5. Usually, the request life-cycle is to receive a request, allow the controller to nominate a view and pass the data model to it.

Request -> Controller -> View (with data)

By creating a FilterAttribute and adding it to the controller class we can meddle with this process and serve up an alternate view without changing the controller itself.

[SplitTest]
public class MyController: Controller {
….

Request -> Controller -> Filter -> Alternate View

The Code

As usual what seems simple ended up being a tiny bit trickier, but surprisingly not by much. Let’s start with this:

 1.    public class SplitTestAttribute : ActionFilterAttribute, IActionFilter
 2.    {
 3.        private static readonly Random R = new Random();
 4.
 5.        void IActionFilter.OnActionExecuted(ActionExecutedContext filterContext)
 6.        {
 7.            var result = filterContext.Result as ViewResult;
 8.            if (result == null)
 9.                return;
10.            var split = "!"; //failure placeholder
11.            var context = new ActionContext();
12.            try
13.            {
14.                var httpContext = filterContext.HttpContext.ApplicationInstance.Context;
15.                context = new ActionContext(filterContext, result);
16.                split = Split();
17.                var alternateViewName = AlternateViewName(context, split);
18.                exists = ViewExists(filterContext, result, alternateViewName);
19.                if (exists && split != "A")
20.                {
21.                    result.ViewName = alternateViewName;
22.                    result.View = null;
23.                    filterContext.Result = result;
24.                }
25.                result.ViewData["AB_Split"] = split;
26.                this.Log(context, split, exists, "OK");
27.            }
28.            catch (Exception exc)
29.            {
30.                this.Log(context, split, exists, exc.Message);
31.            }
32.        }
33.        
34.        private string Split()
35.        {
36.            private static readonly string[] SPLIT_OPTIONS = new[] {"A", "B"};
37.            var toss = R.Next(0, 2);
38.            split = SPLIT_OPTIONS[toss];
39.        }
40.    
41.        private string AlternateViewName(ActionContext context, string split)
42.       {
43.            if (split == "A")
44.                return context.OriginalViewName;
45.            return context.OriginalViewName + "_" + split;
46.        }
47.
48.        private bool ViewExists(ActionExecutedContext filterContext, ViewResultBase result, string alternateViewName)
49.       {
50.            var view = ActionContext.FindView(filterContext, result, alternateViewName);
51.            return view != null;
52.       }
53.
54.   }
55.

Some gotchas to note:

• The guard on the cast at the top (lines 7–9) is because not all actions are ViewResults.
• All the work needs to be done in the OnActionExecuted method, when typically the OnActionExecuting method is used. This is to allow the original view resolution process from the controller to take place before we mess around with it.
• Finding the view and checking that it exists got slightly messy (line 17), so we wrapped that stuff into another class called “ActionContext” (see below).

internal class ActionContext
    {
        public string Url { get; private set; }
        public string User { get; private set; }
        public String SessionId { get; private set; }
        public string OriginalViewName { get; private set; }
        public string FullyQualifiedOriginalViewName { get; private set; }

        public ActionContext()
        {
            Url = "";
            User = "";
            SessionId = "";
            OriginalViewName = "";
            FullyQualifiedOriginalViewName = "";
        }

        public ActionContext(ActionExecutedContext filterContext, ViewResult result)
        {
            var httpContext = filterContext.HttpContext.ApplicationInstance.Context;
            User = httpContext.User.Identity.Name;
            SessionId = httpContext.Session.OrNull(s => s.SessionID, "");
            Url = httpContext.Request.Url.SafeToString();
            OriginalViewName = result.ViewName.OrIfEmpty("Index");
            FullyQualifiedOriginalViewName = FullyqualifiedViewName(filterContext, result);
        }

        private string FullyqualifiedViewName(ActionExecutedContext filterContext, ViewResult result)
        {
            var originalView = FindView(filterContext, result, result.ViewName.OrIfEmpty("Index"));
            RazorView razorView = (RazorView) originalView;
            return razorView.ViewPath.Substring(11).Split('.')[0];  //may need to fiddle for particular path structure
        }

        public static IView FindView(ActionExecutedContext filterContext, ViewResultBase result, string alternateViewName)
        {
            var found = result.ViewEngineCollection[0].FindView(filterContext.Controller.ControllerContext, alternateViewName, "", true);
            return found.View;
        }
    }

Putting it into use

Now that the controller’s view routing has been subverted, let’s give our customers something to look at.

AlternateViewName = OriginalViewName + “_” + split

In order to show the customer something new it was as simple as creating a new view file in the same folder as the existing one and calling it something like Index_B.cshtml. In this way we can choose which views we want to appear differently (maybe only one of them, maybe all).

Alternatively, for very minor changes we added a AB_Split variable to the ViewData array (line 25) that can be used for simple swap-out logic or adding a css class to the view. This approach is good for things like testing button colours.

Extensions

Now that we've covered the mechanics, we can extend the process to improve usability:

• The Split() function only returns “A” or “B”. So we could add a constructor with an integer parameter to indicate the number of split values desired, maybe with a default of 2.
• When the customer returns to the page on the next day, he/she should really see the same split as before.
  * Extending the Split() method to add a cookie to the response (and examining the request too) worked fairly well to give consistency. Of course, this doesn't work over multiple devices.
  * If the user is logged in then we could store the split value against the account, but this only works for non-anonymous pages.

Next Time

Does the data support my idea?

In the next part of this 2-part series I’ll discuss the data side of things… running the experiment and getting some results.