Efficient, reusable application templates and scaffolding

Do you want to start a new project and need to re-write the code again? Or you recently joined the team which needs to build something from the ground up and wants to start building the business logic asap? I think it sounds familiar to anyone who has ever developed a fin-tech project.

In this article, I will propose a few approaches to organizing and delivering your project faster.

The question often arises of how to build a system that can easily scale. Logically, it leads to the problem of creating a scaffolding utility for automating project creation from the specified template with given data. For example, the approach to build scalable apps for particular microservices as features is called Bedrock within BlueHarvest terminology.

Okay, let’s get started.

Let’s consider the following 2 steps.

1. Create a scaffold or generate the initial application
2. Create a pipeline how to add new dependencies in the application

Why Generators?

Having a generator the developer can easily install the new project with basic templates and run it without having prior knowledge about specific technologies.

But how to create a scaffold? What is it? What solutions are in place? What are the best practices?

Having a single solution to generate a boilerplate is important because saving your team (or yourself) 5–15 minutes when creating every repeatable route, component, controller, helper, test, view, etc… really adds up. Also, context switching is expensive, and saving time is not the only benefit of automating workflows. I will give an overview of existing generators written with Javascript which you can extend with your own features/methods.

The must-have requirements for generator are:

1. Developer-friendly interface
2. Reasonable defaults can be configured via the in-project config file
3. Interactive CLI
4. Extendable functionality (adding, upgrading, removing)
5. Control over application dependencies

There are a lot of issues that should be solved within the team when using or creating the generator. The following are certain dependencies in the project and initial configuration, keeping them in an up-to-date state and compatible with each other at the same time, etc.

Here are some of the good javascript examples of application generators.

1. Create-react-app is a React scaffold for frontend applications. The codebase is here. Creating and running a project with it is as simple as Create-react-app app_name — template {custom_template}. The idea of a micro-generator framework can be used for any backend project as well. It’s an npm package that you can install locally or globally that provides functionality via interactive project scaffolding.
2. Vue CLI aims to be the standard tooling baseline for the Vue ecosystem. It is roughly the equivalent of react-scripts, although the feature set is different. This section on CLI Service covers its detailed usage. It ensures that the various build tools work smoothly together with reasonable defaults so you can focus on writing your app instead of spending days wrangling with configurations.
3. Webpack solution to create javascript-agnostic apps. For instance, here’s a solution provided by Yeoman to extend basic functionality with your own features(https://yeoman.io/authoring/).
4. Bedrock generator is a lightweight solution created by Blue Harvest that essentially is a shell script. сustomize.sh script reads an application.properties file and changes necessary settings (maven artifact name, project name, Kubernetes manifests, etc.).

CLIs such as @vue/cli and create-react-app provide the ability to quickly scaffold a new project or instantly prototype new ideas.

CLI Plugins for JS(NodeJS) development are npm packages that provide optional features to your projects, such as Babel/TypeScript transpilation, ESLint integration, unit testing, and end-to-end testing. They are delivered as plugins included as part of your project creation process or added into the project later. They can also be grouped into reusable presets.

Micro-framework generator will have your “best practice” method of creating any given pattern in CODE. Code that can easily be run from the terminal by typing for example “npm run start” or “java -jar JarFileName.jar”. You might have noticed that creating such boilerplate generators is a very time-consuming job that requires experience and needs to have a team of maintainers. Bedrock’s generator, on the contrary, is very lightweight. The key feature is language-agnostic. it’s stored inside an application and therefore highly customizable.

Bedrock concept

Before writing a scaffold for any bedrock, think about the goal of the boilerplate.

What are you trying to achieve and who is going to use it?

• Do you want a generic scaffold that could be used by a wide variety of applications and projects?
• Do you want something specific, like a scaffold that creates both your configuration file(package.json, pom.xml, etc.) and your framework code?
• Who are potential users and what user experience will look like for the users of your scaffold?

Ideally, bedrock should adhere to such principles as MVC, Unix, KISS, DRY, YAGNI, Twelve-Factor, Occam’s razor, and dogfooding.

Each application has many levels of abstractions. And only developers need to decide what should be included and what’s not in your project.

For example, every conventional Javascript project (be it a node.js application or a frontend app that is built with react, webpack etc.) — that uses dependencies in some form — has a package.json file that specifies packages the project makes use of. And the approach is about the way to fill the package.json with certain packages.

We can visualize bedrock pattern by using this picture:

Bedrock concept

The level 1 or base bedrocks contains an empty project with framework and language details.

The level 2 bedrocks are extending the base. A level 2 bedrock contains boilerplate code and additional configuration. In our example, “h2” bedrock contains everything from the “base” bedrock and generic code and configuration for an h2 database.

The level 3 bedrocks are extending one level 2 bedrock. It contains additional boilerplate code and additional configuration on top of the level 2 bedrock. In our example, “jwt” bedrock contains everything from the “rest” bedrock and generic code and configuration for JWT authentication for a RESTful application.

This was the example of the vertical extending of base bedrock.

However, sometimes we have to extend the bedrock horizontally. For example, we created the base bedrock on Nodejs(node-base), but we can create a node-base bedrock on typescript(node-base-typescript). Same with eslint(node-base-eslint, node-typescript-eslint) or jest(node-base-jest, node-typescript-jest) which require not only new dependency in package.json but also some setup and configurations around the boilerplate. Such examples represent the horizontal scaling of bedrock.

Another interesting thing is deciding which packages are supposed to be a part of the new bedrock and which should be taken out. We think that such configurations that perform the layer of your infrastructure around the code should optionally be a bedrock. For example, there are a lot of solutions to deploy the code. Most popular are Azure, CircleCI, Github, Bitbucket, AWS, Netlify, and many other cloud solutions or any on-premise system. We think that the range of these options is almost endless. That’s why it takes a lot of effort to implement and support a big range of solutions aimed to solve the same problem.

Ways to implement the bedrock

Bluedev bedrock

Recently, BlueHarvest developers have started to create initial versions of bedrock patterns. We were inspired by the article Bedrock pattern: a better way of sharing code?

We realized that there’s no way to develop one generator for different languages (Java, NodeJS, Python, etc). Only platform-agnostic approach can provide this option. For example, bash can be utilized as a tool to gather the data from a user’s input.

Our BlueDev bedrock doesn’t contain any logic in the project and install dependencies, default configuration, and basic architecture that can be changed from user input.

Our bedrocks prepare a minimum setup the configuration for the project but don’t dictate the way how to use them. Some people prefer to install dependencies locally on virtual machines, some keep all services in docker files, others like Kubernetes very much. Each bedrock is performed as Git repo. So, each git project implements a feature that can be extended to any project. The usage looks like picking up required bedrocks in your custom project and customizing it afterward by an execution shell script. The details usage is explained here.

What’s included in the project?

In the case of Javascript project, there is only package.json with the base dependencies and scripts required to build, test and run the application.

In the case of Java, there is a pom.xml. Here is the list of published bedrocks in BlueHarvest.

The technical implementation of any bedrock is written in HOW_TO_USE_BEDROCKS.md in each repo with the bedrock.

Microframework/plugin-based bedrock

Again, the frontend provides a lot of ideas for building solutions. Microframework is based on the Plugins system. In the context of NodeJS, we build the base project with index.js and all the rest features are provided with a system of plugins. All plugins aim to extend the main plugin with some additional logic. Any values you add for “dependencies” and “scripts” will be merged with the node-base defaults. Values for any other keys will be used as-is, replacing any matching defaults.

Example of target package.json:

Dependencies:

• Node-base-plugin: core plugin
• node-base-graphql-plugin : graphql over nodejs application

Or

• Node-base-plugin: core plugin
• Node-base-mongodb-plugin: set up and simple usage of MongoDB in-app
• node-base-express-plugin(or node-base-graphql-plugin) simple REST or GraphQL server

If the team decides to support the typescript in the application then each of our plugins should support typescript out of the box by having plugin-name.d.ts in the file structure. In the principal, bedrock covers the opportunity to use the technology in the project but does not contain any logic inside. Sets of technologies allow us to provide solutions. But the solution is the responsibility of the application/product team.

For example, we are going to build a simple NodeJS application with 2 major bedrocks: typescript and javascript. Just simply run: npx create-node-app app_name. Command create-node-app should be an npm package. But for simple usage it is just a GitHub repo with the console command utility. Then all child features/bedrocks should be installed from this directory. Using Git pull, users will fetch the latest version of node-base to create your app and set up an app.

A template must have the following structure:

The template folder is copied to the user’s app directory during create-node-app installation. You can add whatever files you want in here, but you must have at least the files specified above. The template.json is the configuration file for your template. As this is a new feature, more options will be added over time. For now, only a package key is supported. The package key lets you provide any keys/values that you want to add to the new project’s package.json, such as dependencies and any custom scripts that your template relies on. The example of the implementation logic above was created in the repo.

Video of creating node-base bedrock with typescript template

Conclusion

There are some common things to keep in mind before building the bedrock system. It requires a lot of work on the initial stage of creating a project generator and a plugins/bedrocks system. And especially if you have an intention to provide a nice tool for beginners with a nice user experience. Also, a rich collection of plugins integrating the best tools for the backend ecosystem should be built. Another big challenge in the world of rapidly changing tech is to support and update bedrocks in a long-term perspective. Maintainers have to oversee the code changes and check for sub-projects not following standards

Pros and cons of BlueDev bedrock approach

Pros and cons of Micro Framework bedrock approach

Useful links:

Bedrock Pattern: A better way of sharing code?