TestSync: Revolutionizing Feature Stability Testing with Automation

Introduction
In modern software development practices, feature flags play a crucial role in controlling feature releases. However, ensuring the stability of existing features and preventing conflicts between new and old feature flags can be a challenge. This article explores TestSync, a powerful solution designed to detect production bugs early by simulating a production environment. By leveraging a new testing infrastructure, TestSync aims to provide developers with confidence in launching new features while safeguarding existing ones.

Problem Statement
The utilisation of feature flags in software development poses challenges when it comes to ensuring feature stability and preventing unintended interactions. When developing a new feature, developers often face the following problems:

1. Lack of comprehensive testing: Integration tests for new features usually focus solely on their respective feature flags, neglecting the real-world production flag values of existing features. This omission can lead to incomplete testing and insufficient confidence in the integration test suite.
2. Unawareness of flag interactions: Existing feature test suites are often unaware of new feature flags, as the default flag configuration remains unchanged. Enabling a new feature’s flag in production can inadvertently impact existing features, causing disruptions that may only surface if explicitly tested.

What are the goals of this solution?
The TestSync solution aims to achieve the following goals:

1. Prevention of feature collisions: Running integration tests against production flags helps identify any potential conflicts or collisions between new and existing features. By uncovering these issues early in the development process, you can address them before they impact the overall system.This ensures that the tests reflect real-world scenarios, increasing confidence in the stability of the new feature.
2. Enhanced stability of existing features: By validating the stability of all existing features against the new feature’s flag, TestSync ensures that enabling the new feature does not inadvertently cause disruptions or issues in the established functionality. This proactive approach helps maintain a stable and reliable system.
3. Robust integration test coverage: TestSync facilitates the creation of a robust integration test suite that covers interactions between features. By running tests against production feature flags, you can achieve thorough coverage and gain confidence in the overall system behavior.
4. Reduction in manual testing effort: With TestSync, the need for manual testing of all existing features in response to a new feature’s flag change is eliminated. This saves significant time and resources for both QA and development teams, allowing them to focus on other critical tasks.
5. Improved confidence in feature launches: By ensuring that all existing features are stable and unaffected by the introduction of new feature flags, TestSync instills confidence in developers and stakeholders. This confidence is crucial before launching any new feature, contributing to a smoother and more successful deployment.

What is TestSync solution ?
In summary, TestSync provides a generic solution for automated testing against production feature flags. By leveraging this tool, we can overcome the challenges associated with testing numerous flag combinations and configurations.

At Walmart, we have implemented the TestSync solution and successfully leveraged it across teams to ensure the stability of our features. You can use this solution to integrate with any standard CI tools. Popular options include Jenkins, Travis CI, CircleCI, and GitLab CI/CD. Here’s an overview of how we implemented and utilized TestSync:

We have created a parameterized CI job with the following parameters. While Walmart uses an internal CI tool(built on top of Jenkins), you can adapt this solution to work with any standard CI tool of your choice. The default branch name is typically set as “main,” but you can specify any branch against which you want to run TestSync.

1. branchName: This parameter represents the branch against which you want to execute the TestSync process. By default, it is set to “main,” but you can customize it to match your specific branch naming conventions.
2. testSyncCCM: This parameter represents the new feature flag that you want to test for stability. It is important to note that this flag is not yet present in the production environment. You can specify any number of comma-separated flags in this parameter field to test the stability of both new and existing features against these flags.

This flexibility allows you to run TestSync against different branches and feature flags, providing a comprehensive and adaptable testing solution for your development process.

Once the TestSync regression is triggered, you can follow these steps to create a new temporary branch and commit the production feature flags to a JSON file:

1. Retrieve Current Branch: Obtain the current branch from the provided parameter in the CI job. This branch will serve as the base for creating the temporary branch.
2. Create Temporary Branch: Generate a new temporary branch based on the current branch. The branch name can be constructed by appending a suffix, such as “-testsync,” to the current branch name.
3. Pull Production Feature Flags: Pull the production feature flags from the appropriate source, such as a configuration management system or a dedicated feature flag portal. Retrieve the flag values that are currently active in the production environment.
4. Commit Feature Flags to JSON File: Create a JSON file to store the production feature flags. Write the retrieved feature flags into the JSON file, ensuring that they are in the appropriate format.
5. Commit and Push Changes: Add the JSON file containing the production feature flags to the temporary branch. Commit the changes with a meaningful message indicating that the production flags are being added. Finally, push the temporary branch to the remote repository.
   Also, we are maintaining a configuration of exclusion list. This list would be useful if some feature flags like enableGoogleAutoComplete are not required to be synced with production, because in a testing environment we don’t need autocomplete scripts and all.
   We are using NX on Walmart web repo and using NX affected, we are making a CI plan with only affected integration test cases (We don’t consider Functional tests, build steps, lint etc). Below is the code snippet of the above steps and the command to run test cases in testsync mode.

function createTemporaryBranchWithFlags(baseBranch, tempBranchSuffix, featureFlags, exclusionList) {
  try {
    // Create a new temporary branch from the base branch
    createTempBranch(baseBranch, tempBranchSuffix);

    // Pull production feature flags
    const prodFlags = pullProductionFlags();

    // Apply the exclusion list to remove unwanted feature flags
    const mergedFlags = applyExclusionList(prodFlags, exclusionList);

    // Merge the provided feature flags into the production flags
    Object.assign(mergedFlags, featureFlags);

    // Create a JSON file with the merged feature flags
    createFlagFile(mergedFlags);

    // Commit and push the changes to the temporary branch
    const commitMessage = "Add production feature flags";
    commitAndPushFiles(tempBranchSuffix, commitMessage);
  } catch (error) {
    throw new Error(`Error while creating temporary branch with flags: ${error}`);
  }
}

// Apply the exclusion list to remove unwanted feature flags
function applyExclusionList(flags, exclusionList) {
  const mergedFlags = { ...flags };
  exclusionList.forEach((flag) => {
    if (mergedFlags.hasOwnProperty(flag)) {
      delete mergedFlags[flag];
    }
  });
  return mergedFlags;
}

// Merge the provided feature flags into the production flags
function mergeFlags(prodFlags, featureFlags) {
  return { ...prodFlags, ...featureFlags };
}

// Usage
const baseBranch = "main";
const tempBranchSuffix = "testsync";
const featureFlags = {
  flag1: true,
  flag2: false,
  flag3: true,
};
const exclusionList = ["enableGoogleAutoComplete"];

createTemporaryBranchWithFlags(baseBranch, tempBranchSuffix, featureFlags, exclusionList);

nx affected:test --configuration=ci --env.TestSyncMode=true --env.TestSyncCCM='{"cart":{"enableACC":true}}'

Here’s a breakdown of the modified command that you can run :

• nx affected:test: Runs the affected test cases based on the changes detected in your repository.
• --configuration=ci: Specifies the configuration to use for running the test cases. This configuration can be tailored to your CI environment.
• --env.TestSyncMode=true: Sets the TestSyncMode environment variable to true, enabling the TestSync mode.
• --env.TestSyncCCM='{"cart":{"enableACC":true}}': Sets the TestSyncCCM environment variable to {"cart":{"enableACC":true}}, providing the specific feature flag configuration for TestSync.

Now once the test case runs, we have the override logic to deep merge the production feature flags and the flags that are there in test cases.
To achieve the deep merge of production feature flags and test case flags during the test execution, you can implement the following logic:

1. Load the production feature flags from the JSON file into a dictionary or object.
2. Load the new flags provided in the TestSyncCCM environment variable into a separate dictionary or object.
3. Perform a deep merge of the two dictionaries, giving priority to the test case flags. This ensures that the test case flags override the corresponding production flags.
4. Use the merged dictionary or object as the feature flag configuration during the test execution.

const fs = require('fs');

// Load production feature flags from JSON file
const productionFlags = JSON.parse(fs.readFileSync('production_flags.json', 'utf8'));

// Load new flags from environment variable or other source
const newFlags = {
  cart: {
    enableACC: true
  }
};

// Deep merge the two objects
const mergedFlags = deepMerge(productionFlags, newFlags, testCaseFlags);

// Use the merged flags for the test execution
runTestWithFeatureFlags(mergedFlags);

// Function to deep merge two objects
function deepMerge(...objects) {
  const result = {};

  objects.forEach(obj => {
    Object.entries(obj).forEach(([key, value]) => {
      if (typeof value === 'object' && !Array.isArray(value)) {
        result[key] = deepMerge(result[key] || {}, value);
      } else {
        result[key] = value;
      }
    });
  });

  return result;
}

// Function to run test with feature flags
function runTestWithFeatureFlags(flags) {
  // Test execution logic using the merged feature flags
  // ...
}

How to skip a lib running from TestSync mode ?
Sometimes it is often required to skip some library to not run in TestSync mode, i.e. a test doesn’t need to be synced with production feature flags. To make the exclusion of libraries in TestSync mode more generic, you can define a separate configuration file (e.g., testSyncConfig.json) where you specify the list of libraries to be excluded. This allows you to have a flexible and customizable approach to excluding libraries without directly modifying the project configuration files.

Here’s an example of a generic approach:
1. Create a testSyncConfig.json file with the list of libraries to be excluded:

{
  "excludedLibraries": ["cart"]
}

During the CI plan creation, read the testSyncConfig.json file and retrieve the excluded libraries:

const fs = require('fs');

// Read the excluded libraries from testSyncConfig.json
const configData = JSON.parse(fs.readFileSync('testSyncConfig.json', 'utf8'));
const excludedLibraries = configData.excludedLibraries || [];

// Retrieve the list of affected libraries
const affectedLibraries = ["cart", "product", "user"]; // Replace with your own logic or command to get affected libraries

// Create the CI plan with only the opted-in libraries
affectedLibraries.forEach(library => {
  if (excludedLibraries.includes(library)) {
    console.log(`Skipping ${library} from TestSync mode`);
  } else {
    console.log(`Adding ${library} to CI plan`);
    // Add the library to the CI plan
    // ...
  }
});

The code reads the excludedLibraries from the testSyncConfig.json file and stores them in the excluded_libraries list. The affected libraries can be obtained using your preferred logic or command and are stored in the affected_libraries list for demonstration purposes.

The code then iterates over the affected libraries, checking if each library is in the excluded list. If a library is in the excluded list, it will be skipped from TestSync mode. Otherwise, it will be added to the CI plan.

Architecture Diagram

Here is one snapshot from our CI job in TestSync mode.

Future road map:
We are working on some future plan to make TestSync more advanced, which anyone can adapt to.

1. Pull Request Integration: TestSync can be triggered from the Pull Request (PR) workflow. Developers can add a specific comment to their PR, such as /TestSync <feature flags>, to initiate the TestSync flow for that particular PR. The TestSync process, as described earlier, including creating temporary branches, merging feature flags, and running integration tests, will be executed for the PR.
2. Scheduled Cron Job: A scheduled cron job can be set up to trigger the TestSync flow periodically before production launches. This allows for regular and automated TestSync executions to validate the stability of existing features and detect potential bugs introduced by new feature flags.
3. Dashboard and Notifications: A dashboard can be developed to display the results of TestSync integration tests, indicating success or failure. The dashboard serves as a centralized platform to view the test outcomes across different features. Additionally, notifications can be sent via communication channels like Slack to notify relevant users or teams about the test results. This facilitates efficient communication and ensures that stakeholders are promptly informed.

By implementing these generic features, you enhance the usability and automation of TestSync, enabling early bug detection during the development phase, scheduled testing before production launches, and streamlined communication through customisable dashboards and notifications.

Please note that the implementation details, such as the choice of CI/CD tool, scheduling mechanism, dashboarding system, and notification channels, may vary based on your specific requirements and infrastructure.

Conclusion
This solution has become an integral part of our development workflow, providing developers and teams with increased confidence and reducing the risk of breaking changes in production.