BE Experimentation with VWO

Why we Integrated VWO ?

We were used to A/B testing on FE. Where two FE changes are A/B tested and the better is taken live to 100% audience.
But we have a lot of info stored on BE which we were handicapped to experiment.

For example

• What if we want to experiment with the discount %, ie the optimal discount that can be offered to the user.
• What if we want to understand the impact of decreasing the minimum payment amount to be able to make transaction using our platform (which is currently INR 3k).
• Monthly transaction limit per user

Basically, we wanted to have the flexibility to A/B test the variables stored on BE.
We needed the functionality to perform multivariate testing (not being limited to bi-variate testing)
A user when once sees variant A, always sees variant A until that experiment is running.

What is VWO ?

VWO FullStack lets you do the enterprise-grade serverside testing of your product by deploying code behind features, experiment with A/B tests, and rollout or rollback features immediately.

With VWO FullStack, you can literally test anything- be it the search algorithms or recommendation engine for your e-commerce store or subscription workflows for your SaaS business. VWO FullStack is built to help you execute even the most complex tests. It enables you to test on any platform and not just limited to devices that have a browser. Think native apps, beacons, and connected devices.

With VWO FullStack, you can perform the testing wherein the variations of a particular test are rendered directly from the webserver and are then sent to the visitors’ device. Implementing directly on the server allows you to run more sophisticated tests that might otherwise hamper the user experience if implemented on the client-side. In addition to that, serverside testing is also opted for in cases when it is simply unfeasible to experiment on the client-side. For example, testing two different product recommendation algorithms for an eCommerce website.

Typical use cases of VWO FullStack are:

• A/B Test deep within your stack
• You can test multiple variations of the feature with a different set of variables
• Manage Features’ Lifecycle With Feature Flag Management
• Deliver Features Confidently with Staged Feature Rollouts

VWO FullStack enables you to test much deeper. Rather than being limited to testing images or buttons on your website, you can test algorithms, architectures, and re-brands.

Consider the following situation. An eCommerce store owner, with an aim to optimize for better conversions, wants to test two different checkout flows on both his/her website and mobile app at the same time. To do so, they need an advanced testing methodology, which is more inherent to the process and integrates with the website to ensure a seamless user experience. Neither of which is possible by running a simple A/B test on the client-side and that is why serverside testing is required.

Integration

Identify Users

User Identifier, also abbreviated as User ID, is a way to uniquely identify a user. Since campaigns are directly associated with users, our SDK relies on the User ID which you have to provide us. User ID is simply a string. You can customize the User ID based on your business requirements.

How SDK Works

Activate API:-

• The activate API is used, and the first action an SDK performs is parameter validation. If the parameters passed are not of the desired data type, the evaluation ends there. Otherwise, we proceed further.

Activate API

Get Variation Name API:-

• The evaluation process is exactly the same as the Activate API mentioned above, except that it does not send an impression event to the VWO server.

Get Variation Name API

Track API:-

• The evaluation process is exactly the same as Activate API mentioned above, except that it tracks conversions instead of visitors corresponding to the goalIdentifier.

Track API

For Feature Rollout:-

Feature Rollout

For Feature Test:-

Feature Test

VWO FullStack Java SDK Usage

import com.vwo.VWO;

String settingsFile = VWO.getSettingsFile(accountId, sdkKey));

VWO vwoClientInstance = VWO.launch(settingsFile).build();

String variationName = vwoClientInstance.activate(campaignKey, userId, options);

vwoClientInstance.track(campaignKey, userId, goalIdentifier, options);

Get Settings File

Each VWO SDK client corresponds to the settings file representing the current state of the campaign settings, that is, a list of FullStack running campaign settings.

The method accepts two parameters:

• accountId — account-id associated with your VWO account.
• sdkKey — generated inside the respective Project under the FullStack Testing.

Usage

import com.vwo.VWO;

String settingsFile = VWO.getSettingsFile(accountId, sdkKey);

Launch

The VWO client class needs to be instantiated as an instance that exposes various API methods like activate, getVariationName and track.

Parameter Definitions

• Below is the list of all parameters that can be used for configuring the VWO SDK.

Returns

• An instance of the VWO class, which can be referenced later for calling out different API methods.

Usage

import com.vwo.VWO;

String settingsFile = VWO.getSettingsFile(accountId, sdkKey);
VWO vwoClientInstance = VWO.launch(settingsFile).build();

Activate

After successfully instantiating a VWO class, Activate API activates a FullStack A/B test for a specified user for a running campaign.

The API method:-

• Validates the parameters passed.
• Checks whether the user is whitelisted.
• Assigns a deterministic variation to the qualified user.
• Sends an impression event to the VWO server for generating reports.

Usage

import com.vwo.VWO;

String settingsFile = VWO.getSettingsFile(accountId, sdkKey);

VWO vwoClientInstance = VWO.launch(settingsFile).build();
 
HashMap customVariablesMap = new HashMap();
map.put("browser","chrome");

VWOAdditionalParams options = new VWOAdditionalParams();
options.setCustomVariables(customVariablesMap);

// Activate API
String variationName = vwoClientInstance.activate(campaignKey, userId, options);

if (variationName.equals("Control")) {
  // Write code for handling 'Control'
} else if (variationName.equals("Variation-1")) {
  // CODE: write code for Variation-1
} else {
  // CODE: When user does not become part of campaign.
}

Unique Visitors are tracked

• If User Storage Service is provided, SDK will not track the same visitor multiple times. Once tracked and stored by the User Storage Service, the next time the same visitor lands, it will check the existence from the storage via User Storage Service. If found, it will not track the same visitor.

VWO only tracks a visitor and its corresponding conversion only once even if the SDK sends multiple calls.

Get Variation Name

Campaign Activation with User Storage Service:-

• If User Storage Service is provided, campaign activation is mandatory before tracking any goal, getting a variation of a campaign, and getting the value of the feature’s variable.
• Correct usage

vwoClientInstance.activate(campaignKey, userId, options);
vwoClientInstance.track(campaignKey, userId, goalIdentifier, options);

Track

Tracks a conversion event for a particular user for a running FullStack campaign.
The API method:

• Validates the parameters passed.
• Checks whether the user is whitelisted.
• Checks if User Storage Service is provided to know whether the user is returning. If yes, show the previously assigned variation always.
• Assigns the consistent variation to the new/returning qualified user.
• Sends an impression event to the VWO server for generating reports.

Usage

// For CUSTOM goal type
boolean isSuccessful = vwoClientInstance.track(campaignKey, userId, goalIdentifier, options);

// For REVENUE goal type
boolean isSuccessful = vwoClientInstance.track(campaignKey, userId, goalIdentifier, options);

Configure Webhooks

• Anytime there is a change in the FullStack campaign settings, VWO sends an HTTP POST call with a payload to the configured URL. Therefore, you know when to fetch the settings-file. It helps you to keep the most up-to-date version of the settings file.

Configuring Webhooks

Enabling Webhooks in VWO

If you wish to get notified whenever there is a change in campaign settings, enable the Webhooks feature.

• Log in to your VWO account.
• From the left panel, go to FULL STACK > Projects and select a project.
• Under the Environment(s) section, select the Enable Webhooks option for those environments that you prefer to subscribe to the changes for.

Enabling Webhooks in VWO

• In the Enter the URL field, enter the URL where you wish to receive the change event notification.
• To finalize your settings, click SAVE.

Payload Format

{
  "timestamp": 1606482285,
  "event": "settings_changed",
  "action": "campaign_settings_changed",
  "triggered_by": "vwo"
}

Usage

// Endpoint to subscribe to changes made in VWO FullStack running 
@PostMapping("/vwo-webhook")
@ResponseStatus(HttpStatus.OK)
public void webhook(
  @RequestHeader("x-vwo-auth") String secretKey,
  @RequestBody String body
) {
  String webhookAuthKey = "SECRET_WEBHOOK_KEY_GENERATED_IN_VWO_APP";
  
  if (webhookAuthKey != null && secretKey != null) {
    if (secretKey.equals(webhookAuthKey)) {
      System.out.println("VWO webhook authenticated successfully.");
    } else {
      System.out.println("VWO webhook authentication failed. Please check.");
      return;
    }
  } else {
    System.out.println("Skipping Webhook Authentication as webhookAuthKey is not provided.");
  }
   
  if (vwoClientInstance != null) {
    vwoClientInstance.getAndUpdateSettingsFile(accountId, sdkKey);
    System.out.println(vwoClientInstance.getSettingFileString());
  }
}

The webhook will be triggered whenever:
- There is a change in the settings of any running FullStack campaign.
- A FullStack campaign is created.

Campaign Settings & Results

Impression Events

• An impression event signifies that you want to track a visitor inside our VWO application.

When to use activate API?
When you want to get variation and track visitors inside the VWO application. It is used for showing various metrics like conversion rate, improvement rate, etc.
When you want to track a visitor even if he visited the site before.

Mutually Exclusive Campaigns

To create a mutually exclusive group at the account level:-

• Click the gear icon ⚙ on the top right, and go to CAMPAIGN SETTINGS.
• Under the Mutually Exclusive Groups section, click Create mutually exclusive group.

• Enter the name for the group, select platform as FullStack, and then select campaigns you want to include in the group.
• Click Create Campaign Group.

Reports

Inside the VWO application, click the FullStack A/B menu option, select your campaign, and then click the detailed report to see the reports of your campaign.

FullStack Campaign Reports

Impact Analysis

VWO SDK makes different types of network calls to VWO servers depending on the API method used.

• getSettingsFile
  To fetch settings before SDK initialization.
• activate
  To send an impression to the VWO server when a user becomes part of a campaign to count a new visitor. Only for A/B Campaign.
• getVariationName
  Does not send any impression call.
• track
  To send an impression to the VWO server when a user converts the goal of a campaign to count a conversion corresponding to a new visitor.
• isFeatureEnabled
  To send an impression to the VWO server when a feature is enabled for a user to count a new visitor. Only for Feature Test Campaigns.
• getFeatureVariableValue
  does not send any impression call.

Benefits of Integrating VWO

Experiment with everything you build. Deliver better experiences.

Experiment throughout the product stack to discover revenue opportunities while building world-class experiences.

Launch Winning Features With Server-Side Testing

• VWO FullStack gives you the server-side flexibility to run omnichannel experiments with deep segmentation capabilities without any performance impact to build consistent, higher-performing experiences.

Run complex experiments and feature changes

• With VWO FullStack, you can literally test anything, be it subscription flows, search algorithms, or any idea that can move your north star metric. Leverage our rich feature set with multi-platform testing, mutually exclusive testing, goal tracking, targeting, scheduling, and more, to optimize product experiences.

Confidently rollout features with minimal risk

• Mitigate risk by safely delivering a new feature to a small group of targeted audience before making it live. This enables the product team to control the feature rollout and measure the impact on users, while the development team can seamlessly deliver the feature and monitor the effect on the backend.

Advantages of server-side testing with VWO

• A/B Testing:
  Run complex experiments rendered directly from your server and premeditatedly send variations to a visitor’s device.
• Feature Flags:
  Decouple deployment from release by creating flags, managing them at runtime, and choosing which user gets to see which feature.
• Feature Rollouts:
  Mitigate risk by safely releasing new features to a small audience before going live.
• Feature Testing:
  Test different feature configurations to determine the variant which delivers the best experience.
• Traffic Allocation:
  Split traffic between multiple versions of your website and choose the percentage of visitors to be included in your experiment.
• Webhooks:
  Get notified whenever there is a change in campaign settings with Webhooks, and receive change event notifications.