From Code to Prod: the lessons I learned the hard way
Every group or even a single developer wants to move their code and features to production, and at Torq, we strive to continuously and safely deploy code to production as quickly as possible.
This post will present our strategies and processes to achieve this goal, along with the decisions we made in each part of this journey, like using LaunchDarkly, how you test with Playwright, etc
The common problem everyone faces when deploying
When we started building our app, we needed to address the “problem” of finding the right balance between deploying code fast and deploying it safely.
We detected 4 points to address in order to reach our main goal:
- The ability to put some code in prod that is limited to specific users
- The ability to test our code to identify potential bugs as early as we could
- The ability to deploy our code automatically
- The ability to find the root cause of bugs that reached prod (‘cos, let’s face it, everyone has bugs)
We love solving common problems
First, we had to limit our code. We looked for an approach that would help us achieve one of our main goals: to keep deploying code and features that were still raw, while also keeping the PRs small and easy to review.
This basically screams “feature flags”. In the past, we used Flipt — a free, open-source feature flagging solution, but during our work we found it limited in segmentation limitation, and it also doesn’t have in-house auditing. We did some quick research and found LaunchDarkly, which gives us the ability to open any feature or group of code lines to any user we want, or more correctly, hide it from any user that shouldn’t see it. It also lets us create dependencies between flags and restrict access to the edit flags state between groups in RnD.
Once a user is logged in to the app, there is a request that gets all the open flags for this specific user. It is saved in the state management, and we can hide any block section we want. It’s also good for A/B testing. LaunchDarkly also has push notifications along with the flags saved in the state management, so any change can be reactive and affected in real time.
// src/helpers/featureFlags/featureFlags.ts
import { LDClient, LDUser, LDFlagSet, LDOptions, initialize } from "launchdarkly-js-client-sdk";
export default {
async evaluateFlags(): Promise<LDFlagSet> {
const user: LDUser = {
key: `${userEmail}:${accountName}`,
email: store.getters.getActiveUserEmail,
custom: {
"account-id": accountId,
"requested-service": "app",
},
};
const options: LDOptions = { hash: userFlagsHash };
const client: LDClient = initialize(window.env.LAUNCH_DARKLY_CLIENT_ID, user, options);
await client.waitUntilReady();
return client.allFlags();
},
};// src/main.ts
import { FeatureFlags } from "@/helpers/featureFlags";
const evaluateFeatureFlags = async () => {
const flags = await FeatureFlags.evaluateFlags();
store.dispatch(SET_FEATURE_FLAGS, { flags });
};// src/store/modules/featureFlags/index.ts
import { Module } from "vuex";
import { UPDATE_FEATURE_FLAGS } from "@/store/mutationTypes";
import { SET_FEATURE_FLAGS } from "@/store/actionTypes";
import { Flags } from "@/helpers/featureFlags/flagTypes";
import { LDFlagSet } from "launchdarkly-js-client-sdk";
interface FeatureFlagsState {
flags: LDFlagSet;
}
export const featureFlags: Module<FeatureFlagsState, unknown> = {
state: {
flags: {},
},
mutations: {
[UPDATE_FEATURE_FLAGS]: (state, payload: { flags: LDFlagSet }) => {
state.flags = payload.flags;
},
},
actions: {
[SET_FEATURE_FLAGS]: ({ commit }, payload: { flags: LDFlagSet }) => {
commit(UPDATE_FEATURE_FLAGS, payload);
},
},
getters: {
getFlags: (state): LDFlagSet => {
return state.flags || [];
},
isFlagOn: (state) => (flag: Flags): boolean => {
return state.flags[flag];
},
},
};When it comes to testing code, we sort of detach unit testing in terms of UI tests and put our trust in integration tests. We test our logic functions in unit tests and table tests, using Playwright as our testing framework. We researched other tools like Puppeteer and Cypress, but found that Playwright was the best fit as it supports several languages other than JS. This gave us the ability to use the same tool both in integration tests that we wrote in JS, and in E2E tests that we wrote in Python.
Playwright comes with a set of testing tools, “Jest alike”, with all the assertions you may be familiar with from other testing tools, along with strong features to reach any element in the dom and simulate actions.
We split our tests into 2 sections:
- Classic integration tests
- Snapshot testing
When testing a UI application, it’s a must to mock the backend. We took a really cool approach to this. We used Sinon, which gave us the ability to mock functions inside the code. We used it to basically mock the layer that fires the API requests and returns the response.
// clientMocks.ts
// when initializing the tests we save sinon on the window object
// and also we save all our api files on the window object under __INTEGRATION_TEST__
// will return the same result each time the mock function called
const mockClient = async (page: Page, service: string, fn_name: string, res: any, args?: any) => {
return await page.evaluate(([service, fn_name, res, args]) => {
const mockObj = window.sinon.stub(window.__INTEGRATION_TEST__[service], fn_name).returns(res);
args && mockObj.withArgs(...args);
(window as any).sinonSpies = {
...window.sinonSpies,
[`${window.__INTEGRATION_TEST__[service]}-${fn_name}`]: mockObj,
};
return mockObj;
}, [service, fn_name, res, args]);
};
// will return results by order from responses
// if the mock function will be called more times than responses length
// result will be undefined
const mockClientEx = async (page: Page, service: string, fnName: string, ...responses: any) => {
return await page.evaluate(([service, fnName, ...responses]) => {
let i = 0;
const stub = window.sinon.stub(window.__INTEGRATION_TEST__[service], fnName);
for (i = 0; i < responses.length; i++) {
stub.onCall(i).returns(responses[i]);
}
return stub;
}, [service, fnName, ...responses]);
};
// will return a promise of the mocked result
const mockClientPromise = async (page: Page, service: string, fn_name: string, res: any) => {
return await page.evaluate(([service, fn_name, res]) => {
return window.sinon.stub(window.__INTEGRATION_TEST__[service], fn_name).resolves(res);
}, [service, fn_name, res]);
};Once Playwright runs, it tests code on an actual browser and simulates the real behavior of an application.
// test.spec.ts
// this is a snapshot test that take snapshot of the page bothe in dark and light mode
import { beforeEach, describe, it } from "@/integration/fixtures";
import SettingsPageV2 from "@/integration/pages/settingsV2";
import {
mockClientPromise,
mockInitCalls,
ready,
waitForInitialize, mockClient, mockCustomFlags,
} from "@/integration/helpers/clientsMock";
import { assertSnapshot, switchThemeMode } from "@/integration/helpers/common";
describe("settings general tab snapshot testing", () => {
let settingsPage: SettingsPageV2;
describe("happy flow", () => {
beforeEach(async ({ page }) => {
settingsPage = new SettingsPageV2(page);
await settingsPage.navigate();
await waitForInitialize(page);
await mockCustomFlags(page, [Flags.SETTINGS_GENERAL_TAB]);
await mockInitCalls(page);
await mockClientPromise(page, "MyService", "MyMethod", mockResponse);
await mockClientPromise(page, "JournalService", "listOnDemandAlerts", listAlerts);
await ready(page);
});
const color_modes: Array<"light" | "dark"> = ["light", "dark"];
color_modes.forEach((color: "light" | "dark") => {
describe(`${color} mode`, () => {
beforeEach(async ({ page }) => {
await switchThemeMode(page, color);
});
it("upload logo and than delete", async ({ browserName }) => {
await settingsPage.generalTabClick();
await settingsPage.uploadLogo();
await assertSnapshot(settingsPage.page, `general-tab-with-logo-${browserName}-${color}.png`, 1000, 8);
await settingsPage.deleteLogo();
await assertSnapshot(settingsPage.page, `general-tab-no-logo-${browserName}-${color}.png`, 1000, 8);
});
});
});
});
});
// this is an example of regular test
import { beforeEach, describe, it, expect } from "@/integration/fixtures";
import SettingsPageV2 from "@/integration/pages/settingsV2";
import {
mockClientPromise,
mockInitCalls,
ready,
restoreAndMock,
waitForInitialize,
} from "@/integration/helpers/clientsMock";
describe("settings plan tab testing", () => {
let settingsPage: SettingsPageV2;
beforeEach(async ({ page }) => {
settingsPage = new SettingsPageV2(page);
await settingsPage.navigate();
await waitForInitialize(page);
});
describe("paid customer", function () {
beforeEach(async ({ page }) => {
await mockInitCalls(page);
await mockClientPromise(page, "MyService", "MyMethod", MockResponse);
await ready(page);
await settingsPage.planTabClick();
});
it("should display organization workflows count", async () => {
const tabContent = await settingsPage.getPlanTabContent();
await expect(tabContent).toBeDefined();
await settingsPage.usersTabVisible();
await expect(await settingsPage.ssoTabVisible()).toBeTruthy();
await expect(await settingsPage.planTabVisible()).toBeTruthy();
});
});
describe("trial customer", function () {
beforeEach(async ({ page }) => {
await mockInitCalls(page);
await mockClientPromise(page, "MyService", "MyMethod", MockResponse);
await ready(page);
});
it("should not display plan tab", async () => {
await settingsPage.usersTabVisible();
await expect(await settingsPage.ssoTabVisible()).toBeTruthy();
await expect(await settingsPage.planTabVisible()).toBeFalsy();
});
});
describe("failing requset", function () {
beforeEach(async ({ page }) => {
await mockInitCalls(page);
await mockClientPromise(page, "MyService", "MyMethod", MockResponse);
await ready(page);
});
it("should not display plan tab", async () => {
await settingsPage.usersTabVisible();
await expect(await settingsPage.ssoTabVisible()).toBeTruthy();
await expect(await settingsPage.planTabVisible()).toBeFalsy();
});
});
});As I mentioned before, Playwright gave us the ability to create snapshot testing, which let us take a real image snapshot of the application and run an image comparison to alert us about changes.
One thing that we learned was to take snapshots only of the relevant component in each test and not a snapshot of the entire screen (Playwright can do both). This makes tests less sensitive to unrelated changes.
Another thing we implemented (that I recommend everyone implement too!) is running the process that updates the snapshots at the same time as a PR is building. This can cost time if some of the snapshots fail. I recommend taking the updated snapshots and going over them to verify the changes and committing them again.
We improved that also by creating an npm command that finds the relevant build by branch name, downloads the updated snapshots from our CI tool, and puts them inside a local branch ready to review and commit.
We deploy continuously. For every PR a developer opens, our tests (integration + unit) will run. Once they pass, the code merges with the main branch, and an automatic process detects that there is a new build. Then it starts to run our E2E tests in the staging environment. After it passes, the developer gets a notice asking whether they want to move their code to prod. If they do, the E2E runs with the new code in prod.
In the past, the movement to prod was automatic, but as part of finding the balance between fast and safe, we moved it to manual. This was an easy decision in order to give the developer more time to play with their own feature and also share it with Product team, and decrease the number of bugs that reach production.
When it comes to handling bugs, our code in production is minified and uglified using Sentry. Sentry collects all the errors from the app and lets us deploy source maps. Once we invested in an issue, we had a specific place in the code and the complete track to lead to this issue. Also, it integrates with Jira and FullStory. Each issue contains a link to a specific FullStory session where the issue occurred.
And to emphasize the connection between each part of this solution, each bug that is found and solved should be covered with a test that checks this scenario.
What I learned from my experience
I learned that sometimes the best solution is to choose the tools that orchestrate together in the best way, instead of the “best” product for each part of your Solution.
I believe we found the balance that suits us to achieve the main goal.
Reading this post will give you an almost complete set of tools to deploy your code in a way that addresses the main points, with the ability to adjust the balance according to your needs.
One last thing: finding the right balance is an ongoing process, and you always need to adjust accordingly to your current needs.
