Disruption is the new norm. Rolling out changes, understanding mistakes and adjusting to them rapidly is a key factor in a success of any product
“Test fast, fail fast, adjust fast.” — Tom Peters
Talking about the success of a feature in Web Development, how are you currently measuring it?
A trivial approach to measure is of using Google Analytics(GA) to know the impact of a release by comparing it to the time window before it. Given the varying nature of user base with respect to time, this is not a reliable method. Moreover, SEM(Search Engine Marketing) events and festivals can change the game altogether.
Having a reliable method requires that a release is made to a segment of users and the response is compared with respect to other users rather than comparing it with other time windows. We compare the existing version of a website with the new version containing the feature. Existing version is also called the Control Version. This is the very essence of A/B Testing.
A/B Testing is also referred to as A/B/n Testing if you are testing more than one experiment in parallel.
“If you have a question about what you should do, code it up, let people use it, and see which alternative gives you the results you want” — Avinash Kaushik, Digital Marketing Evangelist
Next, what are Phased Releases?
How are you ensuring that a new feature will not disrupt conversions on a stable site? Regression Testing, Automated Testing… Is that the answer?
No matter how smart Product, UX, Developers or QA folks are, one cannot replace a real user… a user who could be vivaciously trying out new things, or could want a laid-back approach like express checkout, or could be continuously comparing prices and experience at your website with competitors…sigh!
Each change has a potential to impact the user behavior, thus, increasing or decreasing the conversions. Here, I am not talking about UI/UX changes only, changes in backend like optimizations and code refactoring can also lead to reduced conversion by causing bugs or by slowing down performance.
It requires that we roll out a release to a segment of users with the flexibility to failover and, if things turn out positive then gradually roll it out to the wild. This process of slowing down a release is called Phased Release also termed as Safe Release.
“Take calculated risks. That is quite different from being rash.” — General George Patton
How about building a process around these?
At FabHotels, we are introducing new features at pace and trivial approaches to measuring the user response for a feature launch did not give confidence over the data. A/B has helped revealing insights about user behavior.
With A/B framework in place, new ideas are welcomed and to test each a version is developed and launched. If the idea succeeds it is merged to the main branch or else discarded. We have set up multiple pipelines for multiple experiments facing different % of users. Traffic Distribution amongst experiments is configurable. It has proved to be an important tool in our arsenal.
“We are willing to go down a bunch of dark passageways, and occasionally we find something that really works.” — Jeffrey Bezoz Founder of Amazon.com
Before all else, know your Data
Being an engineer, the architecture part excites me the most and that is where I wanted to begin. But, in order to have A/B Testing “Practice” in place, the most important thing is knowing what to measure and gathering the data from sources that can be relied upon.
Data should be gathered from below perspectives:
- Impact on User behavior
- Impact on Perceived Performance
- Impact on Server health
- Impact on External Systems
1. Impact on User behavior
Google Analytics(GA) is a popular tool used for Web Analytics. Important metrics here are:
- Ecommerce Conversion Rate
- Bounce Rate
- Session Duration
- Average Time Spent on a page
- Successful Sign Ups
- Click on CTA per Visitor
- Exit Rate
and, it needs to be collected across dimensions like:
- Platforms, eg: Mobile website, Desktop, Tablet, Android/iOS apps
- Landing Pages, eg: Homepage, Listing Page, Details Page, Payment Page, etc.
- Browsers, eg: Chrome, Mozilla, Safari, UC Browser, etc.
- New vs Returning Users
- Organic vs Direct
2. Impact on Perceived Performance
Tools:
- Google Pagespeed
- Lighthouse
- sitespeed.io
- Webpagetest
Metrics:
- Page Load Time
- Page Interactive Time
- DOM Loaded Time
- Number of Network Calls
- Size of Images
- Size of CSS
- Size of JS
- Server Response Time
- Performance for 3G vs 2G User Bandwidth
3. Impact on Server health
Tools:
- Newrelic
- Cloud Watch
- Log monitoring Systems like ELK, Loggly, etc.
Metrics:
- Increase in Error Logs
- CPU Usage
- Memory usage
- iops
- Slow Query Log
- Bandwidth
4. Impact on External Systems
- Count of Calls in Call Centre
- Count of Customer Care Tickets
Though all of these metrics are important and should be watched constantly, it is crucial to know beforehand on how a given change request will move the needle. What metrics are expected to change and by what amount?
“I wish you knew more about what to do with information once you get it, but that is a private wish” — Peter Drucker
Going Agile! — Benefits of A/B Testing
- This model has proved to reduce the market time as it gives you power to take calculated risks.
- You can rollout changes at pace as the control time window to compare the experiment results with is no more required.
- This process allows more pipelines going to production. Multiple experiments running in parallel!
“Learn from small experiments rather than large ones” — John Warnock Co-founder of Adobe Systems
Implementing it
There are three common ways of implementing it:
- Use 3rd Party Tools
- Version Support inside an Application
- Deploy Versions as Multiple Applications
I have discussed insights about the implementation in my next article. Here is the link to that:
Thanks for reading! If you liked this article, hit that clap button below 👏. It means a lot to me and it helps other people see the story.
You may reach me at charanjeet2008@gmail.com
