Building a Scalable API in Node

Published in

The Startup

12 min readNov 12, 2020

This article covers the topic of creating a scalable and configurable REST API, served with the help of express. The code is written in TypeScript. The API is database-agnostic, but in the final example, I will use MongoDB.

TL;DR: Here is the repository with the complete template.

Building a greenfield project is exciting. We think about the problem we face, technologies that can simplify the solution and many other factors we believe may be beneficial for the design. Nevertheless thinking about API we want it scalable and configurable. During my career, I had several situations where I was responsible for building the core of an API, and I believe I can share with you some hints that I found useful. It doesn’t matter if you are starting a brand new project in node, or you just want to learn something about building an API — hopefully, you can find this article relevant in both cases.

Scalability

Scalability begins with defining the proper architectural boundaries. I saw many startups that have started with the ‘Yolo’ approach, building the monolithic backend for the whole app, without giving it a second thought. As time passes such a backend becomes harder and harder to maintain. And if (hopefully) the startup gain popularity, the only way to scale it is by setting up the whole application multiple times, on multiple nodes. It’s costly and not as effective as it could be. Such a solution often needs to be re-written.

Think about your product first. Write down requirements. Discover the architectural boundaries — the places where the system can be divided. DDD (Domain Driven Design) is a great way to teach you how to do it. What for? To let every part of the system work separately. If there is a need to scale our system up, you don’t have to do it for the whole. You can discover some most overloaded parts. It gives you much better maintainability of the system.

Building the system in such a way is preparing it for becoming microservice-based one day. But we never start with microservices. We start with the monolith that has well-defined architectural boundaries and then, someday, we can easily extract microservices from it. Listen to what Martin Fowler says about it:

Don’t even consider microservices unless you have a system that’s too complex to manage as a monolith
Martin Fowler

Configurability

How sure you are that you always gonna use this one particular database? How sure you are that the tool that is good for now, will be good in the future as well? You cant be sure at all. Architectural drivers can change as the times passes, your tools will as well. So firstly — postpone the important architectural decisions as long as you can — until you know a lot about the system. And secondly — write the system in a way that allows you to change the decisions later on.

But how to do it? Configurability is key. And that’s where the Dependency Inversion Principle comes to play (the D principle from the well-established SOLID rules). Invert the flow of dependencies. Don’t let your core logic to rely on the concrete classes/modules. Always create a layer of abstraction. It’s easier than you think. For instance, talking about the database — try not to call DB directly, always use interfaces. You can implement this interface any way you please, using any database you want. That will allow you to change your decision about the database in the future.

Let’s write some code!

In the example below I will show you my idea for an API template. We will write some logic responsible for user creation. We can start with one of two approaches. Either we create a true monolith, where architectural boundaries are kept by separating the code into the modules, or we can create a template for multiple services in the single git repository and keep boundaries more strict. Let’s start with the second approach and see where we can get.

mkdir api-template & cd api-template
mkdir user-service & cd user-service & yarn init
yarn add express
yarn add typescript @types/express ts-node --dev

Then please add tsconfig.json, tslint.json and .gitignore files to finalize the project configuration. Then create the following directory structure:

api keeps the implementation of all our API endpoints
application is responsible for Application Services. We want our api to know as little as possible, and delegate all the job to application services
common keeps the implementation of Value Objects — objects that don’t change in time and represent the same arbitrary value
db handles all database-related implementations
events are responsible for notifying the app about some changes that happened recently
exceptions keeps the custom exceptions we can throw to be more informative
middlewares handles all the behavior that we want to inject before the call reaches the API
models is responsible for all the domain models our application work with

User Model

Let’s say we want to know the user’s first and last names, email, and password. Create an appropriate model inside the models directory. Let’s start with the naive implementation including simple email and password validation:

export class UserModel {
    firstName: string;
    lastName: string;
    email: string;
    password: string;

    constructor(firstName: string, lastName: string, email: string, password: string) {
        const re = /^(([^<>()\[\]\\.,;:\s@"]+(\.[^<>()\[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;        if (!re.test((email).toLowerCase())) {
            // todo throw some exception
        } else if(password.length < 6) {
            // todo throw some exception
        } else {
            this.firstName = firstName;
            this.lastName = lastName;
            this.email = email;
        }
    }
}

It doesn’t look pretty, does it? Email and passwords are a perfect candidates for a value objects. So let’s create the proper exceptions first:

export class PasswordNotValidException extends Error {
    constructor() {
        super("Password not valid");
    }
}export class EmailNotValidException extends Error {
    constructor() {
        super("Email is not valid");
    }
}

Then its time to create value objects:

import {EmailNotValidException} from "../exceptions/email-not-valid.exception";export class Email {
    private readonly value: string;    public static of(email: string): Email {
        if (Email.isValidEmail(email)) {
            return new Email(email);
        }        throw new EmailNotValidException();
    }    private static isValidEmail(email: string) {
        const re = /^(([^<>()\[\]\\.,;:\s@"]+(\.[^<>()\[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
        return re.test(String(email).toLowerCase());
    }    private constructor(email: string) {
        this.value = email;
    }    public getValue() {
        return this.value;
    }
}

For a password, it may be a little bit more tricky, as we probably don’t want to keep and store plain passwords in our system. Addbcrypt and @types/bcrypt libraries using yarn, and then implement the Password class:

import {PasswordNotValidException} from "../exceptions/password-not-valid.exception";
import {hash} from "bcrypt";export class Password {
    // todo move them to env
    private static SALT_ROUNDS: number = 10;
    private static MIN_PASS_LENGTH: number = 6;    private readonly value: string;    public static async of(pass: string): Promise<Password> {
        if (Password.isValidPassword(pass)) {
            return await Password.create(pass);
        }        throw new PasswordNotValidException();
    }    public static ofHash(hash: string): Password {
        return new Password(hash);
    }    private static isValidPassword(pass: string): boolean {
        return pass.length >= Password.MIN_PASS_LENGTH;
    }    private static async create(pass: string) {
        const passHash = await hash(pass, Password.SALT_ROUNDS);
        return new Password(passHash)
    }    private constructor(hash: string = '') {
        this.value = hash;
    }    public getHash() {
        return this.value;
    }
}

And now our implementation of the UserModel simplifies drastically:

import {Email} from "../common/email";
import {Password} from "../common/password";export class User {
    firstName: string;
    lastName: string;
    email: Email;
    password: Password;    constructor(firstName: string, lastName: string, email: Email, password: Password) {
        this.firstName = firstName;
        this.lastName = lastName;
        this.email = email;
        this.password = password;
    }
}

We don’t have to validate anything here, as we know email and password have to be valid to be created in the first place.

User Database Abstraction

As I mentioned earlier, we want to postpone important decisions as long as we can. So, for now, let’s use the simple abstraction for the database. Create db/user/interfaces directory and then define an interface:

import {Email} from "../../../common/email";
import {User} from "../../../models/user";

export interface UserDatabase {
    isEmailUnique(email: Email): Promise<boolean>
    create(companyUser: User): Promise<User>
}

As you see, I’ve added the method isEmailUnique because we need to check it before creating the user. Of course, we could set the constraint on the database, but if we want to keep control of the application and not rely on specific features of external tools, let’s stick to my proposition.

It’s time to store the data somewhere. As we have an interface, we can go with any database or any implementation we want. Here is an example of the implementation of MongoDB. Let’s create an in-memory database for our case.

import {UserDatabase} from "./interfaces/user.database";
import {User} from "../../models/user";
import {Email} from "../../common/email";

export class UserInMemoryDb implements UserDatabase {
    private users: User[] = [];

    create(user: User): Promise<User> {
        this.users.push(user);
        return Promise.resolve(user);
    }

    isEmailUnique(email: Email): Promise<boolean> {
        return Promise.resolve(!this.users.find(x => x.email.getValue() === email.getValue()));
    }
}

UserService

We don’t want our api to act directly on the database. We want to have a separate logic responsible for any action that touches the database. That’s where Application Services helps. Create a new class inside the application directory:

import {UserDatabase} from "../db/user/interfaces/user.database";
import {User} from "../models/user";
import {EmailNotUniqueException} from "../exceptions/email-not-unique.exception";export interface ApplicationService {

}

export class UserService implements ApplicationService {
    private readonly userDatabase: UserDatabase;

    public constructor(userDatabase: UserDatabase) {
        this.userDatabase = userDatabase;
    }

    public async create(user: User): Promise<User> {
        if (!await this.userDatabase.isEmailUnique(user.email)) {
            throw new EmailNotUniqueException();
        }

        return await this.userDatabase.create(user);
    }

    static getType(): string {
        return "UserService";
    }
}

As you see we are using a new exception here. Please add it to the exceptions directory by yourself. The other thing worth mentioning is that we are passing the database interface to the service constructor. It enables us to easily inject there any DB we want.

getType method is also added for a reason — we will be able to determine the service using its type.

Notice in this step we only do create a user. In such a situation some other actions should happen — possibly email should be sent, etc. We don’t want to wait for sending an email, it’s the action that can be run asynchronously. And it can be one of many actions that we want to execute afterward. We will get back to this topic soon.

User Controller

Now, install the express-validator library, open the api directory, and create index.ts file:

import {Router} from 'express'
import {check} from "express-validator";

const router: Router = Router()
const userController = new UserController();

router.post('/',
    [
        check('name', 'Field required').exists(),
        check('nip', 'Field required').exists(),
        check('email', 'Field required').exists(),
        check('password', 'Field required').exists()
    ],
    userController.create)export default router;

Here we validate the existence of the required request parameters and we pass the request for the UserController to handle. Let’s create the UserController then:

import {User} from "../models/user";
import {UserService} from "../application/user.service";
import {Request, Response} from 'express';

export class UserController {
    public create = async (req: Request, res: Response): Promise<any> => {
        const user: User = await User.of(req)
        const result: User = await this.getService().create(user)

        res.status(201).send(new ResponseBuilder(result).setMessage('User created'))
    }

    private getService(): UserService {
        // todo implement it!
    }
}

You can notice a few errors here. First of all, we didn’t pass UserService yet, as we want to configure it later on. The second thing is we don’t have the of method on the User object. Let’s fix it right now by adding the following method to the User’s object:

static async of(req: Request) {
    const body = req.body;
    return new User(body.firstName, body.lastName, Email.of(body.email), await Password.of(body.password));
}

The third problem is the ResponseBuilder class. You can send the response in any way it pleases you, but you are free to use some ResponseBuilder sample I did here.

Service config

And finally, we need to configure our service. Install cors, helmet and morgan and create the service class inside src directory:

import express from 'express'
import apiV1 from './api/index'
import bodyParser from 'body-parser'
import cors from 'cors'
import helmet from 'helmet'
import morgan from 'morgan'
import {UserService} from "./application/user.service";
import {UserInMemoryDb} from "./db/user/user.in-memory-db";
import {ApplicationService} from "./application/interfaces/application.service";

class Service {
    private readonly _express: express.Application
    private readonly _appServices: Map<string, ApplicationService>

    get express(): express.Application {
        return this._express;
    }

    get appServices(): Map<string, ApplicationService> {
        return this._appServices;
    }

    constructor() {
        this._express = express();
        this._appServices = new Map<string, ApplicationService>();
        this.setUp();
    }

    public setUp(): void {
        this.setApplicationServices()
        this.setMiddlewares()
        this.setRoutes();
    }

    protected setApplicationServices() {
        this.appServices.set(UserService.getType(), new UserService(new UserInMemoryDb()))
    }

    public setRoutes(): void {
        this._express.use('/api/v1', apiV1)
    }

    private setMiddlewares(): void {
        this._express.use(cors())
        this._express.use(morgan('dev'))
        this._express.use(bodyParser.json())
        this._express.use(bodyParser.urlencoded({extended: false}))
        this._express.use(helmet())
    }
}

export default new Service();

A few important things are going on. We:

create the express app
register the application service by its type, configuring a database it uses.
set middlewares
set routes we’ve already created

Now, as we already have UserService registered, we need to get back to the UserController and implement thegetService method:

private getService(): UserService {
    return service.appServices.get(UserService.getType()) as UserService;
}

Then the last step is to create anindex.ts file in the user-service directory:

import service from './src/service'

const PORT = 8080

service.express.listen(PORT, () => {
    console.log(`Server is listening on ${PORT}`)
})

Then add a new script to package.json

"scripts": {
  "dev": "nodemon --watch src --exec ts-node ./index.ts"
},

Voilà! We have an initial logic to run the service. You can run it and send some data to the endpoint http://localhost:8080/api/v1/ Nevertheless there are still some steps we need to make to ensure it’s working properly.

Response data

After sending a valid request to the endpoint, you might have noticed that we get the whole user model as a response. We are also getting a password hash. It’s unacceptable. To fix it add the following method to UserModel:

/* method called while sending the model as API response */
public toJSON(): any {
    return {
        email: this.email.getValue(),
        firstName: this.firstName,
        lastName: this.lastName
    }
}

Error Handling

You’ve probably already noticed that sending a request with the wrong input throws an exception and puts the request on hold. That’s because we don’t have a proper error handling middleware yet. Let’s create one:

import {Request, Response} from 'express'
import {validationResult} from 'express-validator'
import {ResponseBuilder} from "../response/response.builder";

export const route = (func: any) => {
    return (req: Request, res: Response, next: () => void) => {
        const errors: any = validationResult(req)
        
        /* validate all generic errors */
        if (!errors.isEmpty()) {
            return res
                .status(422)
                .send(
                    new ResponseBuilder().err('Validation failed', errors.array())
                )
        }

        /* process function and catch internal server errors */
        func(req, res, next).catch((err: any) => {
            res
                .status(err.ERROR_CODE ? err.ERROR_CODE : 500)
                .send(new ResponseBuilder().err(err.toString()))
        })
    }
}

As you can notice it catches all errors that our system can generate and returns it in an elegant response. From now on, adding an ERROR_CODE property to the exception class can change the status code sent back to the client — and I strongly advise you to change it to 422 for all custom errors we’ve created.

Executing actions on success

Sometimes we want to react somehow when the action completes. We may need to send w Welcome Email, log something asynchronously or do anything else after the request is completed. On the other hand, we don’t want to make the client wait for the response. One of the elegant solutions for this problem is an event handler. Please take a look at the code below:

export interface DomainEvent {
}

export interface DomainEventSubscriber {
    handle(event: DomainEvent): void
    canHandle(event: DomainEvent): boolean
}


export interface DomainEventPublisher {
    publish(event: DomainEvent): void
    subscribe(subscriber: DomainEventSubscriber): void
}

export class ForwardDomainEventPublisher implements DomainEventPublisher {
    private subscribers: DomainEventSubscriber[] = [];

    subscribe(subscriber: DomainEventSubscriber) {
        this.subscribers.push(subscriber);
    }

    publish(event: DomainEvent): void {
        this.subscribers.forEach(async (subscriber) => subscriber.canHandle(event) ? await subscriber.handle(event) : null)
    }
}

Here we create an abstraction for Event, EventPublisher and EventSubscriber. There is also a sample implementation of EventPublisher. Let’s move on with our implementation:

export class UserCreated implements DomainEvent {
    email: Email

    constructor(email: Email) {
        this.email = email;
    }
}export class WelcomeEmailSubscriber implements DomainEventSubscriber {
    handle(event: UserCreated): void {
        // todo send an welcome email to the user
    }

    canHandle(event: DomainEvent): boolean {
        return event instanceof UserCreated;
    }
}

And register this Subscriber in the service.ts

class Service {
    ...
    private readonly _publisher: DomainEventPublisher
    ...

    constructor() {
        ...
        this._publisher = new ForwardDomainEventPublisher(); 
        ...
    }

    public setUp(): void {
        ...
        this.registerEventsSubscribers()
        ...
    }

    protected registerEventsSubscribers() {
        this._publisher.subscribe(new WelcomeEmailSubscriber());
    }    protected setApplicationServices() {
        this.appServices.set(UserService.getType(), new UserService(new UserInMemoryDb(), this._publisher))
}
}

As you may notice, we’ve passed the publisher to the application service. Now let’s fix the application service:

export class UserService implements ApplicationService {
    private readonly userDatabase: UserDatabase;
    private readonly publisher: DomainEventPublisher;

    public constructor(userDatabase: UserDatabase, publisher: DomainEventPublisher) {
        this.userDatabase = userDatabase;
        this.publisher = publisher;
    }

    public async create(user: User): Promise<User> {
        if (!await this.userDatabase.isEmailUnique(user.email)) {
            throw new EmailNotUniqueException();
        }

        return await this.createNewUser(user);
    }

    private async createNewUser(user: User) {
        const newUser: User = await this.userDatabase.create(user);
        this.publisher.publish(new UserCreated(newUser.email));
        return newUser;
    }

    static getType(): string {
        return "UserService";
    }
}

That’s it! From now on, we will be publishing the event UserCreated, then all the subscribers that can handle that type of event will react.

The template on GitHub

If you want to see the full code, feel free to download it here. Please note, that I did some improvements in this repo such as:

I’ve added the swagger configuration. From now on you can see the endpoint’s documentation opening the URL: http://localhost:8080/api/v1/docs
I’ve added the MongoDB and I’ve changed the default DB to Mongo (from in-memory database)
I’ve used yarn-workspaces to keep the code more structured and reusable
I’ve added the docker-compose for setting up the MongoDB easily
I’ve added the authentication mechanism (logging in / out)
I’ve moved some of the configuration logic to the .env file
I’ve added tests to make sure our API works properly

Wrapping it up

It took some time, but we did it! We’ve implemented an elegant template for a scalable and configurable node-based API. From now on, you can use it to create new services in your application and to keep control of the complexity of your codebase.