Building reusable front-end toolkit at Tokopedia

Hi everyone, today I’d like to share my experience about building front-end toolkit at Tokopedia.

Background

If you followed the latest trend on front-end technologies (especially React), you might’ve heard about the famous toolkit: create-react-app. So why it’s famous (53k+ ⭐️)? Here is its tagline:

Create React apps with no build configuration.

So, before create-react-app existed, it’s been painful just to get started with React development. This applies to any front-end library/frameworks without official CLI support. One might build a starter/boilerplate from scratch and some will just clone existing popular starters.

After having the starter, the next problem arises:

“How do I get update from the upstream? I want that new shiny feature!”

“I have many apps, how to keep them updated?”

You will have a hard time keeping your starter-based app to be up-to-date with the upstream starter, all the more if you have many apps. FYI, my team at Tokopedia have more than 10 (and counting) front-end applications and of course you need to keep maintaining them.

“So what is the better solution for this? “

I recommend you to watch Dan Abramov’s talk on Youtube. So he’s one of React team at Facebook and he’s building create-react-app. In his talk, he recommended us to build a reusable toolkit/toolboxes. So toolkit comes in form of CLI, it abstracts away the painful configuration so you can just focus on code and the project just need to contains your application code.

There are many zero config toolkit these days, like create-react-app, cra-universal (mine 😂), Jest, Razzle, Nextjs, Docusaurus, Webpack 4, Parcel, the list goes on and each with their own purposes.

In this case, we want a toolkit that’s able to start a dev server, build a bundle, testing, generate project, and other common dev tools.

By using a reusable toolkit, just bump your toolkit’s version in your package.json and your app is updated. Since it’s versioned, you can keep compatibility between apps without worry, and one improvement can be enjoyed by the rest of the apps.

So we already have create-react-app (or other framework CLI), so why bother creating another toolkit?

• It may not suite your needs
• You already have existing legacy apps and find it hard to migrate them into third-party toolkit since you’ll face many incompatibilities. It’s better to just pull out their common config and build a toolkit for them, they’ll be compatible no matter what and you can migrate them incrementally
• You have opinionated and high customization
• Everyone have their own toolkit for their own purpose, example: kcd-scripts, paypal-scripts (private toolkit by Kent C Dodds)

Building the toolkit

Initially, we have some legacy front-end apps, they are starter-based app. Since we’re growing so fast and keep launching new products, we lack agility to bootstrap a new front-end app, and worse, we’re having a hard time maintaining them because each app have their own tooling, and not standardized. Based in this problem, the toolkit development comes in mind.

So, we use isomorphic React starter in Tokopedia, initially the project consists of the following components:

starter-based app

The orange ones are considered unique per app, while the blue ones are considered similar per app. Let me explain the components:

• The client/server codes are simply your application code
• The client/server bundler contains Webpack, Babel, SASS, and many other configuration. We use this to run local dev-server and build production-ready bundle
• We also have Express SSR server to do server-side rendering and its implementation is same across apps
• Shared tools are chore tooling that are not related with bundling system, but useful to boost DX (e.g.: automation), and you usually copy-and-paste them across apps.
  Example: rimraf, eslint, Storybook config, puppeteer config for E2E tests or pre-rendering

The main idea is to move the blue ones into centralized space, the toolkit.

toolkit-based app

Now, it’s looking good! Your app now only contains your application code and depends on your toolkit as development dependency.

Actually it feels not worth the effort if you only have one app, but if you have many apps to maintain, you will greatly feel the benefits. The picture below is example how we can maintain multiple apps with different versioning:

multiple toolkit-based apps with different versions

Toolkit is versioned so you can maintain compatibility and easy updates across your apps. When your product owner wanted to launch a new product, you can quickly generate a new project using the toolkit instead of cloning a starter again. This way you have more reusable, scalable, and maintainable front-end applications.

Note:

- The generated project is also a starter, but it only contains your app code and it depends on your toolkit

- You may build multiple toolkit(s) instead of one, remember “S” of SOLID 😄

Our front-end toolkit tech stack:

What did we use to build the toolkit? Here is our tech stack:

• yargs (for CLI engine)
• Webpack (for module bundler)
• Lerna (for monorepo management)
• SASS (for styling)

You can find out more at https://github.com/antonybudianto/react-kits.

It’s based on this starter

P.S.: We have separate version for Tokopedia 😆

I used Webpack Node API to watch and bundle the app via CLI script. So you might be wondering:

If you put Webpack and its config on the toolkit, then it won’t work because Webpack will be resolving files on the toolkit directory, not the app one right?

Yes, by default Webpack will resolve on the current directory, which in this case, your current directory is your toolkit directory, not your app directory even though you called the toolkit command on your app directory. It’s because Webpack is called by the script on your toolkit, not on your app directory.

The solution is to use context option to make sure Webpack running on app working directory. In order to do this, you need to get the absolute path to your app working directory. Here’s how to do it:

const path = require('path');
const cwd = process.cwd();
const pcwd = path.resolve(cwd);
___

{
  context: pcwd
}

Okay, now Webpack finally able to resolve the entry file in your application code.

The next one is when trying to resolve Webpack loaders, you’ll get errors caused by missing loaders. You can solve this by adding the toolkit’s absolute path into resolveLoaders option:

function resolveDir(...args) {
  return path.resolve(__dirname, ...args)
}
___

{
  resolveLoader: { 
    modules: [resolveDir('../../node_modules'), 'node_modules']
  },
}

Now, your webpack should work correctly just like when it was in starter-form.

Challenges

I believe that there’s no silver bullet to everything, so here’s what we faced during development of the toolkit that’s worth to note:

• Determine the orange and blue on current apps.
  Deciding whether some stuff should be put on the toolkit or app-side. There are things that are better be put at app-side like public path config, since we have different CDN for each app. Think carefully about this
• Research many Webpack API(s) to enable code+config separation
• Wrong abstraction can be more fatal than duplication
• Allowing customization is actually tricky, you need to define what can be customized without exposing too much. Many customization on app side usually means you do a wrong abstraction
• Managing Peer dependency. React and React DOM usually installed on app-side, not on toolkit-side. You must define whether some packages should be installed on toolkit or app side.
  Tips: Core framework like React, React DOM, Redux, GraphQL, usually installed on app side so they can be updated anytime. Having them installed on toolkit means you locked their versions to the toolkit version
• Keeping the toolkit clean from over-bloated configuration
• Keeping compatibility between new and legacy apps

I think that’s it for my experience of developing front-end toolkit at Tokopedia. There is no silver bullet, our approach might not be the best but we already feel the benefit by doing this and will keep improving it. I hope you find it useful too.

If you have any ideas or questions, please post a response below, and if you like this article, please share with your friends and claps on! Thank you so much for reading.

P.S.: We’re hiring! If you’re curious and wanna be a part of amazing team, apply now!