Hopefully that doesn’t look like our code ¯\_(ツ)_/¯

Continuously Delivering Full Stack JavaScript: Part 1

Crafting applications destined for Agile deployment

This post is the first of two parts, with this part covering deployment principles and tools, and concepts relevant to the software development lifecycle. The second part covers implementation details and practical guidelines regarding the deployment of JavaScript applications and can be found at this link!

Roughly nine months ago, a particular team at the Agile Consulting & Engineering tribe went head-on into the land of JavaScript with the advent of a new product opportunity. Today, we live to tell the tale with not only a product, but a delivery pipeline which matches that of our other projects, including automated unit/system/integration tests and automated deployment processes; and here is our story.

Why JavaScript?

Like how any good story starts, we start with why — why the apparently risky move to an ecosystem that has only gained popularity a few years ago?

1. Established specification and evolution process: The TC39 process governs the evolution of the ECMAScript specification and if you care to check it out, it’s looking pretty decent.
2. Established tooling/support: What JavaScript lacks in terms of language semantics, its supporting software ecosystem makes up for. There’s ESLint for static code analysis, TypeScript if you get the jitters over lé wild let, Babel to allow for syntactic sugar, and code optimisers such as Uglify and the Closure Compiler. Node.js also comes out of the box with a debugger and performance profiler.
3. C-like Syntax: Most Computer Science types across all ages have had their altercations with C. Having a C-like syntax eases on-boarding for more experienced developers and recent graduates.
4. Package/Open-source ecosystem: NPM is the largest package ecosystem with 546,720 packages as of time of publication. This means that there’d be a high chance someone else has already built something we needed. Reduce, reuse, recycle y’all!
5. Assistance ecosystem: As of time of publishing, there are 1,510,498 questions tagged with JavaScript on Stack Overflow. This is in comparison with 4,913 for Elixir, 26,004 for Golang, 189,214 for Ruby, 852,595 for Python, and 1,143,031 for PHP and 1,156,411 for C#, all of which are up-and-coming/commonly used languages for web development. Lots of questions means lots of potential help when your code decides to spontaneously combust.
6. Developer ecosystem: The presence of JSConf, monthly JavaScript meetups like talk.js, and presence of JavaScript wielding companies with engineering capabilities in Singapore such as Microsoft and PayPal, and the growing community of tech startups meant that we would be spoilt for choice if the need ever came to quickly upsize our team. With quantity comes a higher chance of quality.

Today, JavaScript is one of the languages/runtimes that through its inception and evolution as a first class tool for web applications, has the potential to impact the most people through the Internet. It does not elegantly solve every type of problem out there, but its ecosystem and developer community enables it to handle almost any type of problem. As Atwood’s Law goes:

Any application that can be written in JavaScript, will eventually be written in JavaScript

Product Architecture

How a product is deployed often depends on its architecture and this section focuses on how to think about the different services that compose your product when planning for deployments. This section covers the mental model of what an atomic deployment of one or more services should look like in relation to a product.

Thinking about Deployments

Services within a product can be categorised into three broad types:

1. Application services are services that code is being written for and require high availability such as websites and their backing web Application Programming Interfaces (APIs) — anything you write code for and where the logic is created by developers. These services often expose behaviours that users interact with directly (websites) and indirectly (APIs). Each deployment should contain at most one application service.
2. Backing services are services that your application services depend on such as databases and caches. Since the application depends on them, they need to be highly available too. These services usually require declarative configurations but not logical development. Each deployment can specify multiple backing services.
3. Administrative services can be thought of as scheduled tasks, need not always be available for the application to work. Examples of these include garbage cleaners or data synchronisation jobs that run once a week/day. Behaviour of these services are dependent on the application’s needs. Each deployment can specify multiple administrative services that share configurations with the application service.

How we did it

Our product consists of two application services for which deployments are designed around:

1. A front-end React.js based website served from an Express server
2. A back-end Node.js application which runs our web API

For our back-end, we include backing services for data retrieval and state persistence:

1. A relational database to store persistent data and state
2. A key-value cache to store frequently requested data and data which requires relatively more processing power to retrieve such as complex SQL queries on large data sets

And lastly, multiple administrative services for updating and synchronising data to/from external systems. These scripts are scheduled for running when certain events are detected and should be limited to one instance running at a time:

1. Script to update database schemas iteratively (aka database migrations)
2. Script to seed database with static content (aka database seeding)
3. Script to pull updated data dumps from various sources
4. Script to synchronise our database data with the updated data dumps

These services are packaged as Docker Images, pushed to a private Docker Registry, and finally aggregated and deployed using Kubernetes specfiles in the sprit of infrastructure as code.

Development Lifecycle

Each deployment should serve a purpose — i.e. for a particular type of testing, for a specific group of users et cetera — and thus, it is important to consider the product’s development lifecycle before designing deployments. This section covers how to come up with the necessary deployments for continuous delivery.

Our Definition of Done

We starting rolling with Agile Scrum through User Stories that developers pick up and work on. Our development lifecycle is best expressed through an Agile artefact, the Definition of Done, which specifies the different states that a User Story goes through before being released:

Un-started: A User Story in question has been approved by the Product Owner (PO), prioritised in the Product Backlog, and has been committed to during the current Sprint (an iteration of development in Agile Scrum)

Started: Developers work on writing code and unit/system tests for the User Story in question. Work done here resides in feature branches.

Finished: Code has passed unit/system tests and the corresponding feature branch has been merged into the development branch. Code in this state is available in a Quality Assurance (QA) deployment.

Delivered: Quality Engineers have written end-to-end integration tests which the code passes in QA. Code in this state is made available in a User Acceptance Testing (UAT) deployment for POs to review.

Accepted: POs have reviewed the code in UAT and has declared that the code meets the requirements. Accepted User Stories are scheduled for the next deployment.

Released: Ops engineers have deployed the code into the production copy of the application.

When planning for appropriate deployments, one useful consideration to make is the roles assigned to each state change in the development lifecycle:

• Developers: Un-Started to Started to Finished states
• Quality Engineers: Finished to Delivered states
• Product Owners: Delivered to Accepted states
• Ops Engineers: Accepted to Released states.

From the above understanding, we can see that we require four deployments minimally.

How we did it

We define two environments for our code to run in:

1. Development has many convenience capabilities that help with developer productivity such as hot/live reloading together with more detailed and specific exception messages.
2. Production on the other hand is a no-frills, optimised for speed and size packaged version of the application, removing critical debugging mechanisms like meaningful error messages which may pose security vulnerabilities through exposing system parameters and inner workings.

We defined five deployment configurations (one more for staging to reduce the risk of failing to deploy in production):

1. Continuous Integration (CI). Code here runs in the development environment and is deployed immediately after merging with the development branch before automated tests are run. Started code resides here.
2. Quality Assurance (QA). Code here has passed unit and system tests, and is run in a production environment for E2E integration tests to run against. We run QA in production to avoid errors caused by environment disparity. Finished code resides here.
3. User Acceptance Testing (UAT). Code here has passed all automated tests and checks, and is run in a production environment ready for user review by Product Owners. Delivered code resides here together with code that has been Accepted and is pending deployment.
4. Staging. Code here has been Accepted, meaning it has passed all checks and is considered release-ready. This deployment mimics the production deployment closely to minimise risk of misconfigurations when deploying to production.
5. Production. Released code resides here.

Defining Deployment Requirements

Deployments are essentially about fulfilling the non-functional requirements of a product. As the deployment of an application does not provide business value directly, it cannot be a User Story, and its requirements are largely left to DevOps/Ops engineers to come up with based on their understanding of delivering the application which delivers business value.

This results in the need to consider the non-functional requirements of the application — more simply, what does the application require in order to provide business value? This section covers what configurations should be done for each deployment we defined in the above section.

Non-Functional Requirements

A product is usually composed of one or more applications working in synergy, with each application having its own corresponding codebase. An application deployed from the same codebase should necessarily:

1. Execute successfully. This is obviously critical, so we need to ensure that relevant system and application dependencies are available for the deployment in question.
2. Be available. The availability of a deployment affects whether the application can be accessed by its target group of users. Availability is primarily affected by expected load and network ingress/egress rules.
3. Be accessible via a unique URL. For ease of understanding which deployment type we are accessing.
4. Connect to different backing services. So that deployments are independent of each other, allowing for appropriate tests and checks to be run with pre-defined data that validates the correct behaviour. Think sandboxes to avoid cross-deployment data corruption.
5. Maintain similar low level behaviour. A functionality once defined and exposed to consumers/users, should be the same in all environments so that tests can run against consistent behaviour, confirming that more abstract lower level code exposes the correct high level behaviour.
6. Expose different high level features. The deployment should expose an accordingly defined set of features that reflect the state of the User Stories in the development lifecycle.

Correspondingly, a deployment is essentially a configuration that provides for:

1. Standardised base system — to allow for development to be done in a known system environment
2. Infrastructure scaling parameters — to allow the application to be available
3. A deployment resource locator — to allow users to access the application
4. Configurations to various backing services — to allow for sandboxed consumption of features
5. Application level configurations — to specify an environment, allow for features to be toggled, et cetera

Let’s take a look at how we defined these requirements in Docker and Kubernetes.

How we did it with Docker

Developers always have their preferred operating system to develop on, which creates problems that result in the ‘it works on my machine’ syndrome. Docker solves this by abstracting away the operating system layer, providing an encapsulated Virtual Machine (VM) like environment with a standard system that includes base software installations (think node, yarn git, jq, openssh, curl et cetera) and user permissions.

We use Docker Images to:

1. Define a base system that we would use across all deployments. This included software such as Node, Yarn, curl et cetera and allowed us to test in development if things would break in production.
2. Define development/production system from the base system that we could use for development and production environments. Difference in Node dependencies were done by carefully listing production/development dependencies in the dependencies and devDependencies property of the package.json file. This provided us two base systems for use in builds, tests and releases.
3. Define a build environment from the development base system that we could use to reliably build binaries which deployments depended on. This reduced the risk of errornous ELF header messages which arise when binaries compiled on one system are loaded in another.
4. Define a test environment from the development base system where we could run tests in similar environments. This helped to resolve situations where developers would run working tests on their machines which broke in the CI.
5. Create immutable releases from the production base system by copying our code into a Docker Image which is then pushed to a private Docker Registry. We ensure its immutability by coupling this with automated version tagging of each successfully built and tested commit. This allowed us to easily pick versions for deployment to production depending on what version is currently in UAT.

How we did it with Kubernetes

Like Docker, Kubernetes adheres to the ‘infrastructure as code’ ideology and we specify our deployment configurations in specfiles which could be easily modified and applied to a cluster of VM instances.

We used the following artefact types of Kubernetes:

1. Deployments to specify scaling and rollout parameters. This allows us to detail how many instances of our services should be available at any point in time. Being able to specify a rollout strategy also allows us to ensure zero downtime by automating the blue-green deployment process.
2. Deployments to specify version, environment and global configurations. This allows us to pick a version of an image and deploy to the correct environment by attaching ConfigMaps and mounting Secrets.
3. Services to specify port bindings which affect how deployed services refer to each other within the same cluster and for allowing an Ingress resource to access these services.
4. ConfigMaps to define environment variables and global configurations which is made available to application and backing services to customise behaviour and specify sandbox resource locators.
5. Secrets to mount volumes over application directories. This allows us to define a set of development keys that can be stored in the codebase since they would be overwritten by a Secrets volume during deployment.
6. Ingresses to specify how deployments could be accessed. This allows us to deploy services, route to them depending on the host and path and load balance these requests via an Nginx controller.

✊🏽 The Call to Action

We’ve come to the end of the first part on continuously delivering full-stack JavaScript applications, the second part covering implementation details and guidelines using JavaScript and can be found at this link! Once again, if you’ve enjoyed and found this post helpful, do consider applauding 👏🏽 a few times (or more!) so that others like yourself may discover this post and benefit from it too(:

To keep up with similar content, subscribe to our humble Government Digital Services — Singapore publication where we frequently document our journey towards product design & engineering excellence. You could also follow me to keep updated on my journey in impacting Singapore and the world through software engineering.

Lastly, if you’re interested in working with us to create citizen-centric applications in Singapore and believe you have the technical chops, our team is hiring! Ping me at joseph_goh@tech.gov.sg and lets work something out(:

Cheers!