Building a scalable Node.js Express app

This publication was initially a response to how to properly isolate a controller using Node.js Express application. Since the community response was so positive I thought I might as well adapt my answer to make it my first publication on Medium. :-)

It is assumed that you have already worked with Express 4 and Mongoose, and have general knowledge about JavaScript and Node.js.

Neither folder structure nor Express setup are included. But I will consider writing about my whole Node.js setup if you enjoyed this article! let me know in the comments below

The real problem behind starter Node.js tutorials

Most Node.js tutorials, which none of them are to blame for, focus on the content: getting certain libraries / frameworks working. To make it easier for the readers who are new to them they are often presented the most simple solutions.

Whilst these solutions suit most of the times for small to medium projects, large applications tend to require far more abstraction, which is what we’ll be analyzing next.

Let’s start with a small hypothetical case:

• I want to have an API that serves an user search through AJAX.
• I want to have an API that also serves the same user search through Socket.io.

Let’s start with Express. That’s easy peasy, isn’t it?

routes.js

import * as userControllers from 'controllers/users';
router.get('/users/:username', userControllers.getUser);

controllers/user.js

import User from '../models/User';
function getUser(req, res, next) {
  const username = req.params.username;
  if (username === '') {
    return res.status(500).json({ error: 'Username can\'t be blank' });
  }
  try {
    const user = await User.find({ username }).exec();
    return res.status(200).json(user);
  } catch (error) {
    return res.status(500).json(error);
  }
}

Now let’s do the Socket.io part:

Since we’re not here to talk about socket.io, I’ll skip the boilerplate.

import User from '../models/User';
socket.on('RequestUser', (data, ack) => {
  const username = data.username;
  if (username === '') {
    ack ({ error: 'Username can\'t be blank' });
  }
  try {
    const user = User.find({ username }).exec();
    return ack(user);
  } catch (error) {
    return ack(error);
  }
});

We’re done! However, something smells here…

• if (username === ''). We had to write the controller validator twice. What if there were ncontroller validators? Would we have to keep two (or more) copies of each up to date?
• User.find({ username }) is repeated twice. That could possibly be a service.

We have just written two controllers that are attached to the exact definitions of Express and Socket.io respectively. They will most likely never break during their lifetime because both Express and Socket.io tend to have backwards compatibility. BUT, they are not reusable. Changing Express for Hapi? You will have to redo all your controllers.

Another bad smell that might not be so obvious…

The controller response is handcrafted. .json({ error: whatever })

APIs in RL are constantly changing. In the future you might want your response to be { err: whatever } or maybe something more complex (and useful) like: { error: whatever, status: 500 }

Let’s get started (a possible solution)

I can’t call it the solution because there is an endless amount of solutions out there. It is up to your creativity, and your needs. The following is a decent solution; I’m using it in a relatively large project and it seems to be working well, and it fixes everything I pointed out before.

I’ll go Model -> Service -> Controller -> Router, to keep it interesting until the end.

Model

Although I won’t go into details about the Model, you should be having a similar Mongoose Model structure.

models/User/validate.js

export function validateUsername(username) {
  return true;
}

You can read more about the appropriate structure for mongoose 4.x validators here.

models/User/index.js

import { validateUsername } from './validate';
const userSchema = new Schema({
  username: { 
    type: String, 
    unique: true,
    validate: [{ validator: validateUsername, msg: 'Invalid username' }],
  },
}, { timestamps: true });
const User = mongoose.model('User', userSchema);
export default User;

This is just a basic User Schema with an username field and created updated mongoose-controlled fields.

The reason why I included the validate field here is for you to notice that you should be doing most model validation in here, not in the controller.

Mongoose Schema is the last step before reaching the database, unless someone queries MongoDB directly you will always rest assured that everyone goes through your model validations, which gives you more security than placing them on your controller. Not to say that unit testing validators as they are in the previous example is trivial.

Read more about this here and here.

Service

The service will act as the processor. Given acceptable parameters, it’ll process them and return a value.

Most of the times (including this one), it’ll make use of Mongoose Models and return a Promise (or a callback; but I would definitely use ES6 with Promises if you are not doing so already).

services/user.js

function getUser(username) {
  return User.find({ username}).exec(); // Just as a mongoose 
                                        // reminder, .exec() on find
                                        // returns a Promise instead
                                        // of the default callback.
}

At this point you might be wondering, no catch block? Nope, because we're going to do a cool trick later and we don't need a custom one for this case.

Other times, a trivial sync service will suffice. Make sure your sync service never includes I/O, otherwise you will be blocking the whole Node.js thread.

services/user.js

function isChucknorris(username) {
  return username === 'Chuck Norris';
}

Controller

We want to avoid duplicated controllers, so we’ll only have a controller for each action.

controllers/user.js

export function getUser(username) {
}

How does this signature look like now? Pretty, right? Because we’re only interested in the username parameter, we don’t need to take useless stuff such as req, res, next.

Let’s add in the missing validators and service:

controllers/user.js

import { getUser as getUserService } from '../services/user.js'
function getUser(username) {
  if (username === '') {
    throw new Error('Username can\'t be blank');
  }
  return getUserService(username);
}

Still looks neat, but… what about the throw new Error, won't that make my application crash? - Shh, wait. We're not done yet.

So at this point, our controller documentation would look sort of:

/**
 * Get a user by username.
 * @param username a string value that represents user's username.
 * @returns A Promise, an exception or a value.
 */

What’s the “value” stated in the returns? Remember that earlier we said that our services can be both sync or async (using Promise)? getUserService is async in this case, but isChucknorris service wouldn't, so it would simply return a value instead of a Promise.

Hopefully everyone will read the docs. Because they will need to treat some controllers different than others, and some of them will require a try-catch block.

Since we can’t trust developers (this includes me) reading the docs before trying first, at this point we have to make a decision:

• Controllers to force a Promise return
• Service to always return a Promise

⬑ This will solve the inconsistent controller return (not the fact that we can omit our try-catch block).

IMO, I prefer the first option. Because controllers are the ones which will chain the most Promises most of the times.

return findUserByUsername
         .then((user) => getChat(user))
         .then((chat) => doSomethingElse(chat))

If we are using ES6 Promise we can make use of a nice property of Promise to do so: Promisecan handle non-promises during their lifespan and still keep returning a Promise:

return promise
         .then(() => nonPromise)
         .then(() => // I can keep on with a Promise.

If the only service we call doesn’t use Promise, we can make one ourselves.

return Promise.resolve() // Initialize Promise for the first time.
  .then(() => isChucknorris('someone'));

Going back to our example it would result in:

...
return Promise.resolve()
  .then(() => getUserService(username));

We don’t actually need Promise.resolve() in this case as getUserService already returns a Promise, but we want to be consistent.

If you are wondering about the catch block: we don't want to use it in our controller unless we want to do it a custom treatment. This way we can make use of the two already built-in communication channels (the exception for errors and return for success messages) to deliver our messages through individual channels.

Instead of ES6 Promise .then, we can alternatively make use of the newer ES2017 async / await (now official) in our controllers:

async function myController() { 
  const user = await findUserByUsername(); 
  const chat = await getChat(user);
  const somethingElse = doSomethingElse(chat); 
  return somethingElse; 
}

Notice async in front of the function.

Router

Finally the router, yay!

So we haven’t responded anything to the user yet, all we have is a controller that we know that it ALWAYS returns a Promise (hopefully with data). Oh!, and that can possibly throw an exception if throw new Error is called or some service Promise breaks.

The router will be the one that will, in an uniform way, control petitions and return data to clients, be it some existing data, null or undefined data or an error.

Router will be the ONLY one that will have multiple definitions. The number of which will depend on our interceptors. In the hypothetical case these were API (with Express) and Socket (with Socket.io).

Let’s review what we have to do:

We want our router to convert (req, res, next) into (username). A naive version would be something like this:

router.get('users/:username', (req, res, next) => {
  try {
    const result = await getUser(req.params.username); // Remember: getUser is the controller.
    return res.status(200).json(result);
  } catch (error) {
    return res.status(500).json(error);
  }
});

Although it would work well, that would result in a huge amount of code duplication if we copy-pasted this snippet in all our routes. So we have to make a better abstraction.

In this case, we can create a sort of fake router client that takes a promise and n parameters and does its routing and return tasks, just like it would do in each of the routes.

/**
 * Handles controller execution and responds to user (API Express version).
 * Web socket has a similar handler implementation.
 * @param promise Controller Promise. I.e. getUser.
 * @param params A function (req, res, next), all of which are optional
 * that maps our desired controller parameters. I.e. (req) => [req.params.username, ...].
 */
const controllerHandler = (promise, params) => async (req, res, next) => {
  const boundParams = params ? params(req, res, next) : [];
  try {
    const result = await promise(...boundParams);
    return res.json(result || { message: 'OK' });
  } catch (error) {
    return res.status(500).json(error);
  }
};
const c = controllerHandler; // Just a name shortener.

If you are interested in knowing more about this trick, you can read about the full version of this in my other reply in React-Redux and Websockets with socket.io (“SocketClient.js” section).

How would your route look like with the controllerHandler?

router.get('users/:username', c(getUser, (req, res, next) => [req.params.username]));

A clean one line, just like in the beginning.

Further optional steps

Controller Promises

It only applies to those who use ES6 Promises. ES2017 async / await version already looks good to me.

For some reason, I dislike having to use Promise.resolve() name to build the initialize Promise. It's just not a clear what's going on there.

I’d rather replace them for something more understandable:

const chain = Promise.resolve(); // Write this as an external imported variable or a global.
chain
  .then(() => ...)
  .then(() => ...)

Now you know that chain marks the start of a chain of Promises. So does everyone who reads your code, or if not, they at least assume it's a chain a service functions.

Express error handler

Express does have a default error handler which you should be using to capture at least the most unexpected errors.

router.use((err, req, res, next) => {
  // Expected errors always throw Error.
  // Unexpected errors will either throw unexpected stuff or crash the application.
  if (Object.prototype.isPrototypeOf.call(Error.prototype, err)) {
    return res.status(err.status || 500).json({ error: err.message });
  }
  console.error('~~~ Unexpected error exception start ~~~');
  console.error(req);
  console.error(err);
  console.error('~~~ Unexpected error exception end ~~~');

  return res.status(500).json({ error: '⁽ƈ ͡ (ुŏ̥̥̥̥םŏ̥̥̥̥) ु' });
});

What’s more, you should probably be using something like debug or winston instead of console.error, which are more professional ways to handle logs.

And that’s is how we plug this into the controllerHandler:

...
  } catch (error) {
    return res.status(500) && next(error);
  }

We are simply redirecting any captured error to Express’ error handler.

Error as ApiError

Error is considered the default class to encapsulate errors in when throwing an exception in Javascript. If you really only want to track your own controlled errors, I'd probably change the throw Error and the Express error handler from Error to ApiError, and you can even make it fit your needs better by adding it the status field.

export class ApiError {
  constructor(message, status = 500) {
    this.message = message;
    this.status = status;
  }
}

Additional information

Custom exceptions

You can throw any custom exception at any point by throw new Error('whatever') or by using new Promise((resolve, reject) => reject('whatever')). You just have to play with Promise.

ES6 ES2017

That’s very opinionated point. IMO ES6 (or even ES2017, now having an official set of features) is the appropriate way to work on big projects based on Node.

If you aren’t using it already, try looking at ES6 features and ES2017 and Babel transpiler.

Result

The complete code (shown before step-by-step), with no comments or annotations. You can check everything regarding this code by scrolling up to the appropriate section.

router.js

const controllerHandler = (promise, params) => async (req, res, next) => {
  const boundParams = params ? params(req, res, next) : [];
  try {
    const result = await promise(...boundParams);
    return res.json(result || { message: 'OK' });
  } catch (error) {
    return res.status(500) && next(error);
  }
};
const c = controllerHandler;
router.get('/users/:username', c(getUser, (req, res, next) => [req.params.username]));

controllers/user.js

import { serviceFunction } from service/user.js
export async function getUser(username) {
  const user = await findUserByUsername();
  const chat = await getChat(user);
  const somethingElse = doSomethingElse(chat);
  return somethingElse;
}

services/user.js

import User from '../models/User';
export function getUser(username) {
  return User.find({}).exec();
}

models/User/index.js

import { validateUsername } from './validate';
const userSchema = new Schema({
  username: { 
    type: String, 
    unique: true,
    validate: [{ validator: validateUsername, msg: 'Invalid username' }],
  },
}, { timestamps: true });
const User = mongoose.model('User', userSchema);
export default User;

models/User/validate.js

export function validateUsername(username) {
  return true;
}

Have any questions? Please, let me know in the comments.

You can also get in touch with me via Twitter, LinkedIn, GitHub, StackOverflow or Mail.

Mail: bWFpbHRvOnp1cmZ5eEBnbWFpbC5jb20=