Automation for Typescript libraries

First, a disclaimer: this might not be the ready-to-use recipe. It is rather a story of me progressing through the world of Typescript and NodeJS, and the result of my experiments. Nevertheless, at the end of this text there will be a link to GitLab repository, which you might want to inspect and may be even get something for yourself out of it. Maybe you’re even going to create your own fully automated solution, following my example.

Reasoning

So, why would I want to create a library, or NPM package in a first place?

1. Reuse code between projects.

I started noticing that while working in business projects I tend to create /tools folder in each of them. And then, I also happen to copy some of those tools from one project to another with me when I move. I asked myself — why not create a package and reference it instead?

2. Different lifecycle. In one of the projects we had a big corporate shared components dependency packaged into one big library. Several other projects in the company also depended on it. And we had to update whole package even if only one of components was updated. Changes in other components could break something and we might not have enough time reserved to test it all. This model is quite inconvenient. When each package serves its own purpose (or a fairly small set of purposes) and is separately published, you only update it when you really need it. And it gets released only when there are some changes in it.

3. Separate support code from business logic. In DDD, there is a principle called Domain Distillation. It suggests to identify parts of the application that do not belong in the core domain and to isolate from them. And how to better isolate than to move those parts out into another project. By the way, Domain Distillation looks much like SRP, but on a different level.

4. Its own code coverage. In our business project the coverage is about 30%, and library that I extracted from it is almost at 100%. Main project is down one percent, but it was red before extraction and is still red, while library is shiny green. And it stays that way through the year and 4 major versions.

5. Can make this code open source. Code not containing business logic is the first candidate for extraction, therefore such a code could be made publicly available.

Starting new library is expensive

There is a problem though — to create a library, aside from creating git repository, you must create tasks to build, lint and test. And, to additionally benefit from testing, you will need code coverage. Also, there is going to be a manual publishing process. And readme. Can’t help with readme though.

So, what might one do with all those menial tasks?

First Step: Seed

First, I created the seed project. It has the same structure as the first Typescript library project that I had. I populated it with gulp tasks and scenarios that would build, test and collect coverage for me in one action. To create another project, I needed to clone the seed to a new folder and change the origin remote to point to new repository on GitHub (I used GitHub then). Starting off cloned seed bears another benefit — when I need to make some change to those tasks, I apply it to the seed. No need to copy and paste was files to another project. Instead, next time I touch one of thos projects, I add second remote called “seed” and merge latest changes from it.

And it worked for me for quite some time. Until I used the seed for a project to be supported by several people. I did write a 3-step instruction, which basically consisted of pulling master, building and publishing. And one of developers, for reasons unknown, forgot the second step. What are the chances of that?

Second step: Automatic publish

Even though it was but a single mistake, the process of manually publishing the library is tedious. So i thought automatic publish is necessary. CI was also needed to avoid merging a branch with failed build or tests. So, I investigated Travis CI on GitHub in order to do such automation. It did not work for me since it had a limitation — for Travis, pull request to master is the same as a commit already in master. One of my colleagues pointed at GitLab and recommended its CI, and it worked for me perfectly.

I implemented following flow to implement a change in my projects, be it bugfix, new feature or new version with breaking changes:

1. I create a branch off master, implement feature and tests there.
2. I create a merge request
3. Tests are automatically run by GitLab inside “node:latest” container.
4. Request is approved and merged.
5. Gitlab runs second set of scripts in another container. Script creates a tag at that latest master commit with next version. To calculate version number, it checks version in package.json and if is higher than last published version, package version is used, otherwise previous publish version is auto-incremented.
6. Script builds and tests the project again.
7. Script pushes created tag back to repository and publishes the package at that version.

This way tag on master branch always corresponds to version of published package. For it to work, to have access rights, environment variables have to be set in GitLab project settings.

Last step: Automate everything

I automated a lot of things already, but there were still a lot of manual steps to go through while creating a project. It was still an improvement though, since these steps only needed once per project, as opposed to once per version. But even then, the instruction of how to create new project was 11 steps long. And I failed to follow my own instruction couple of times. So i figured: why stop there? I want to automate it all.

There is but one prerequisite for it to work, I need to put 3 files to .ssh folder. I figured that this folder is as secure as it gets. First file is “id_rsa”, it is needed to push tags to repository. And it is supposed to already be there. Second file is “gitlab”, it contains GitLab API token, needed to create and configure a project. And last file is “npm”, contains token to publish the package.

And then the magic begins. To create new package, all I need is to run following command in the seed folder: “gulp startNewLib -n [npmName]/[libName]”. And it is there. Ready to go, to implement and auto-publish.

For example “vlr/validity” was one of the packages, created using this command .

That task does the following for me:

1. Creates a project on GitLab
2. Clones seed to a local folder next to the seed the command is executed on
3. Changes the origin remote to a project created in step 1
4. Pushes master branch
5. Sets environment variables from files in .ssh folder
6. Creates “first implementation” branch
7. Changes library name in package.json, commits and pushes it to new branch.

There, all I need is to put the code in and create a merge request. Which is going to be published upon merge.

As a result, that I am quite proud of, when i decide to extract some code from business project, the time it takes from decision to first published version is about 5 minutes. 4 minutes of those are taken by 2 GitLab pipelines, 2 minutes each. The rest is to run startNewLib command and then copy the extracted code from main project.

There are few limitations: Npm login here has to be the same as gitlab nickname. And unlike all other features in the seed, this task only works on Windows.

If you want to investigate the seed, here is the URL: https://gitlab.com/vlr/tslib-seed