Building a production grade Javascript npm package

Published in

Government Digital Services, Singapore

9 min readMar 16, 2024

Steps and processes for delivering a simple production grade Javascript npm package.

At local scale, we write modular codes within applications to reduce duplication, provide abstraction, improve readability and testability.

As we progress, we could make Node Package Manager (NPM) modules of code that could be shared across projects.

But to be effective, the modules need to be reliable. The following are the qualities that makes them reliable.

Version controlled.
Equipped with tests which specifies behavior and a safety net for refactoring.
Having Static Analysis Security Tests (SAST) performed on its code.
Consistent quality through use of Continuous Integration/ Continuous (CI/CD) pipeline and workflow.
Have ownership, responsive to changes and issues.
Have documentation on its usage.

I have created an opinionated basic NPM package (Refer to [1]) that could be used to start creating and publishing NPM modules.

The intent of this package is to have basic qualities required for production, yet without complex solutioning and toolchains.

This package is meant for individual and small team contributions. As such, complexity makes it harder for people to get started and to learn how various production requirements are addressed.

If things start to scale, we will upgrade and swap out with more relevant tooling as required.

Initial parts of this are based on the work shared in a Snyk blog (Refer to [2]). Let me know through the issues page (Refer to [3]) if you encounter problems or you could give feedback on how we can make it better.

In this article, I will explain how the basic structure works. It relies on the following suite of tools.

Gitlab repository for version control.
Gitlab CI (Refer to [4]) for pipeline, workflow, container registry and issue tracking.
Security scans provided by Gitlab CI templates.
Jest (Refer to [5]) for unit testing.
ESLint (Refer to [6]) for formatting and code convention consistency.
NPM for package management and publication (Refer to [7]).
Semantic Versioning convention (Refer to [8]) for version numbering.
Using Conventional Commits (Refer to [9]) for commit messages.
Commitlint (Refer to [10]) to enforce commit message conventions.
Husky (Refer to [11]) to implement Git hooks to lint code and comments before allowing commits to be made.
Docker container (Refer to [12]) to build custom runner images which speeds up jobs running in Gitlab CI pipeline.
Markdown (Refer to [13]) for documentation.
Semantic Release tool (Refer to [14]) to help with bumping version based on commit comments, changelog generation and publication to NPM package registry.

The reasons why I choose them are as follows:

Gitlab CI is an integrated SaaS (Software as a Service) tool for Software Configuration Management (SCM), Continuous Integration/ Continuous Deployment (CI/CD) and issue management. It allows custom containers for task execution. It also provides us with basic security scans.
Jest for unit tests. The reference by Snyk blog uses Mocha. I will take the opportunity here to setup a Jest variant.
NPM’s registry is the default registry where our NPM command line tool will perform look up when installing modules. Publishing at NPM will make installing our module easier.
ESLint. We just need to use the most common one. The intent is consistency and not comparing styles. More important to achieve consistency with automation, improve readability and free our attention at more important things.
We will use Semantic Versioning conventions for our version numbering. The numbering quickly reflects the impact of changes like breaking, features and fixes. It is commonly used and hence many tools supports it. This brings us to our next decision.
Using Conventional Commits automate our version bumping and publication all in one with the Semantic Release tool.
To ensure that our commit messages will be compliant, we will use Husky to install Git commit hooks. This approach enable us to update the hook from scripts in package.json when we decide to change out our tools.
Markdown formatted text documents like README.md and CHANGELOG.md as documentation. Markdown allows ease of reading and writing with text editor independent of rendering. There is no need to fine adjust visual alignments. We just focus on content, organisation and readability. Let the renderer handle the rest.
Docker containers are built locally where we pre-install the npm modules once and use everywhere in the pipeline. This is faster than doing it every time the runner starts for each job. The caveat is that we need to update the docker image whenever we add new npm modules.

To use this basic npm package, let us create a NPM Access Token of automation type from your NPM dashboard.

From the Gitlab CI web console, we create empty Gitlab project with your desired namespace and project name. Hit the plus button, select New project/repository from the menu and press the Create blank project button.

Press the Plus button and select New project/repository

Remember to un-check the Initialize repository with a README to keep it empty. We will use the one provided by basic-gitlab-npm-package repository.

Subsequently, create a Gitlab Personal Access Token by navigating to your Preferences and Access Token menu.

Add a new token with an recognizable name and reasonable expiry date. Select api, read_repository and write_repository as the scope for the token. Semantic Release will use the Application Programming Interface (API) offered by Gitlab CI to create tags and releases.

Dialog for adding new Personal Access Token

Now head over to the environment variables located at project Settings and CI/CD.

Add the environment variables for the following as a protected and masked variable.

GL_TOKEN — Enter your Gitlab Personal Access Token here
NPM_TOKEN — Enter your NPM Access Token here.

Add the following as normal variables.

DRYRUN — Set this to true during test run to prevent publication to NPM registry. Set to false when you feel it is ready.
COMMIT_LINT_DEPTH — This determines the depth of commits to lint for compliance to Conventional Commits. Set to 1 at the start. This is to cover the case where code is pushed in as a bunch of commits and we want to cover them.

Once these are done, we can head over to our local repository. Clone from basic-gitlab-npm-package repository found here (Refer to [1]).

git clone git@gitlab.com:kaikokchew/basic-gitlab-npm-package.git your_projectname
cd your_projectname

On your local repository clone, update all references to kaikokchew namespace and basic-gitlab-npm-package project name with your own.

We will update NPM cache to local folder. This will be part of optimisation to speed up the building of our docker container.

npm config set cache .npm

Then we perform an NPM install to add the various tools needed.

npm install

Not only will this install the NPM modules we need, it will also configure your git repository’s commit hooks for Husky.

Next, we build our container for the CI/CD tasks.

docker build -t registry.gitlab.com/your_namespace/your_projectname/runner:latest --build-arg GL_PROJ_GROUP=your_namespace --build-arg GL_PROJ_NAME=your_projectname --progress=plain .

If all is well, we proceed to login to our Gitlab CI registry and push our container image into it.

docker login registry.gitlab.com
docker push registry.gitlab.com/your_namespace/your_projectname/runner:latest

We are building out own runner container image so that the NPM modules could be pre-installed to save time and resource from duplicative work in the pipeline.

The tradeoff to this is that we need to remember to build and update a new container image whenever we add, remove or update modules that we use.

Once done, let’s change the origin of the our local repository from basic-gitlab-npm-package to your own namespace and project.

git remote delete origin
git remote add origin git@gitlab.com:your-namespace/your-projectname.git
git push --set-upstream origin --all

Once pushed, it should set off a pipeline execution.

This workflow is simply described using a .gitlab-ci.yml file in your repository. Gitlab CI will pick it up and execute. Refer to the excerpt below.

stages:
- static-check
- unit-test
- build
- release

sast:
  stage: static-check
include:
  - template: Security/SAST.gitlab-ci.yml

linter:
  image: $CI_REGISTRY_IMAGE/runner:latest
  stage: static-check
  script:
    - ln -s /cache/node_modules node_modules
    - npm run lint

lint-commit:
  image: $CI_REGISTRY_IMAGE/runner:latest
  stage: static-check
  script:
    - ln -s /cache/node_modules node_modules
    - npx commitlint --from HEAD~$COMMIT_LINT_DEPTH

unit:
  image: $CI_REGISTRY_IMAGE/runner:latest
  stage: unit-test
  script:
    - ln -s /cache/node_modules node_modules
    - npm run test:ci
  coverage: /All files[^|]*\|[^|]*\s+([\d\.]+)/
  artifacts:
    when: always
    paths:
      - coverage/cobertura-coverage.xml
      - junit.xml
    reports:
      junit:
        - junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

build:
  image: $CI_REGISTRY_IMAGE/runner:latest
  stage: build
  script:
    - ln -s /cache/node_modules node_modules
    - npm run build
  dependencies:
    - sast
    - unit
    - linter
    - lint-commit

publish:
  image: $CI_REGISTRY_IMAGE/runner:latest
  stage: release
  variables:
    GIT_STRATEGY: "clone" # https://github.com/semantic-release/gitlab/issues/259
  script:
    - ln -s /cache/node_modules node_modules
    - ([[ ${DRYRUN} == "false" ]]) && npm run semantic-release
    - ([[ ${DRYRUN} == "true" ]]) && npm run semantic-release -- --dry-run
  dependencies:
    - build

The following describe how the Gitlab CI config file works.

First, we describe the available Stages that exist in our pipeline. Stages are executed in series.
Stage do not perform the actual tasks but sequence the tasks instead.
Each task is described by a Job which are declared by their names. For the above they are sast, linter, lint-commit, unit, build and publish.
Hence each Job need to indicate which Stage they belong to.
Jobs within a Stage executes in parallel when there are enough Runners to work on them.
The image property inside the Job description indicates which container to be used when the Runner performs the task. We are mostly using the container image that we pushed into the Gitlab CI container registry earlier.
The exception being the sast Job which uses a template from Gitlab CI to invoke the free SAST scanning tools. We will not cover templating here for now.

Some gotchas to take note here.

We need to use clone strategy for Gitlab CI checkout in the publish job for Semantic Release tool to work correctly (Refer to [15]).
Semantic Release uses NPM publish command. This will in turn invoke test and build step before publishing to NPM repository (Refer to [16]). It seems like we could use this build step instead of our own to save time. However it will not work because version bumping happens before we build. If we encounter a build error, version will be bumped and tagged before failure is encountered.
Semantic Release create commits with release notes included. I have disabled length checks (Refer to [17]) in the rules for now using commitlint.config.mjs file.

With most things up, we could now start developing code for re-use in other projects!