The Modern API, Part 1

At decent labs we know a thing or two about building software. In our pursuit of client happiness paired with using solid software fundamentals to build resilient, scalable systems, we’ve iterated through the process of building APIs maaaany times.

I took the time yesterday to (start to) take all of my current best practices around API design and put them into one “boilerplate” repo.

TL;DR: https://github.com/decentorganization/decent-api

This post kicks off a series of blog posts aimed at walking through and explaining this repository in detail.

To let readers know which technologies, libraries, and concepts we’ll be going over (and to help with SEO):

• highly-supported runtimes via nodejs
• simple servers via express
• postgres via docker-compose
• programmatic sql via knex
• extreme access via cors
• es6 support via babel
• testing via mocha and chai
• logging via debug and morgan
• error handling via custom errors
• linting via eslint
• hot reloading via nodemon

Let’s go!

Development Dependencies

We need to install some tools on our system to support the project and tests actually running locally. Then we’ll need to install some packages into our project to support the various repetitive tasks of developing code.

System Development Dependencies

We’ll be using a couple of tools to execute our development environment, namely git, nodejs, yarn (instead of npm), and docker. I'll leave this up to the reader.

Tip: for nodejs and yarn, use some sort of version manager, like nvm and yvm. It's a good idea for when you're working with different projects with different runtime requirements on your computer. This isn't necessary though, installing from your favorite package manager, or by downloading from their websites, will work just fine.

Project Development Dependencies

If you’re creating a new project, go ahead and yarn init your way into it. If you're adding the contents of this guide into an existing project, good luck!

Now we’re getting into it! Let’s install a bunch of packages to aid with development. Run this in your terminal:

$ yarn add \
    @babel/cli \
    @babel/core \
    @babel/node \
    @babel/plugin-syntax-dynamic-import \
    @babel/plugin-transform-runtime \
    @babel/preset-env \
    @babel/runtime \
    babel-plugin-dynamic-import-node \
    chai \
    chai-http \
    dotenv \
    dotenv-cli \
    eslint \
    eslint-config-prettier \
    eslint-config-standard \
    eslint-plugin-import \
    eslint-plugin-mocha \
    eslint-plugin-node \
    eslint-plugin-promise \
    eslint-plugin-standard \
    mocha \
    nodemon \
    rimraf \
  --dev

Quickly:

• @babel/* packages are for transpiling our fancy modern Javascript into a more vanilla Javascript
• mocha and chai are for the test suite
• dotenv is for our environment variables
• eslint-* is for linting
• nodemon is for watching for file changes on the system
• rimraf is for deleting shit, like when we're creating a production build and want to clean the project first

Now we’ve got a node project with some tools installed. Neat.

Configuration

I like to build projects in a way that use as much “default” configuration options from third party tools as possible. This just keeps things as simple as possible.

Sometimes, though, that’s just not possible. Let’s create some new config files for some of our tools.

git

If you’ve ever worked in a git repository before, this should be of no surprise. Need that .gitignore. I like to use https://gitignore.io for my .gitignore needs. Here's what gitignore.io has for the standard node project: https://gitignore.io/api/node.

There’s one more thing to add to our .gitignore, which will be the directory that production builds get transpiled into. We'll be calling that directory dist, so add an entry for dist in your .gitignore, too.

Here’s the .gitignore from the final repo: https://github.com/decentorganization/decent-api/blob/master/.gitignore

I’ve also got and entry for .vscode in there, too, because I like to tell VSCode to auto-format on save, for certain projects, and I don't want those options part of the repo: they're really just my preference.

dotenv

To manage environment variables on our development machines, we’ll use dotenv. This package relies on a file called .env to read environment variables from, and inject them as actual system environment variables into the running node process.

.env is .gitignored, so I like to create a file called .env.example and commit that into the repository. In the README, instruct developers to copy the contents from .env.example into a new (ignored) file named .env.

For our project, the .env(.example) file will hold a port number on which the API should be exposed (API_PORT), database connection information (DB_*) and a "base" logger string (LOGGING_BASE), which can be anything. It's used to prefix logs.

https://github.com/decentorganization/decent-api/blob/master/.env.example

All of the values in this .env.example are defaulted to in the actual codebase. A developer really only needs to create their own .env file if they want to override anything here.

babel

We’ll use babel to transpile our nice ES6-style Javascript into vanilla Javascript. To configure babel, we'll use a .babelrc file.

Honestly, I haven’t taken much time trying to understand the nuances of babel, so I'll just show you what works for our project. This works for our needs:

https://github.com/decentorganization/decent-api/blob/master/.babelrc

eslint

We’re using ESLint to lint our code, and because we’re writing new style Javascript with ECMAScript 9 (2018), we’ll need to configure ESLint appropriately with an .eslintrc file.

https://github.com/decentorganization/decent-api/blob/master/.eslintrc

nodemon

nodemon is a simple package that monitors for changes on your filesystem, and kicks of scripts when they do. We'll use nodemon to watch for changes to our source code and test code, then to run the linter and run the tests and restart the development server.

We will ignore the specific directory that holds migration files, because things get weird if those get executed before we’re done writing them. (The app executes migrations automatically upon startup, so if we’re restarting it in the middle of writing a migration via nodemon, a half-formed migration will be executed against the development database and sucks.)

https://github.com/decentorganization/decent-api/blob/master/nodemon.json

docker

We’re using docker in this project for one very specific purpose: to provide us with a database. We’re not dockerizing our project (yet lol).

We’ll use docker-compose.yml to specify that we just need a basic postgres service.

https://github.com/decentorganization/decent-api/blob/master/docker-compose.yml

The Non-Development-Specific Dependencies

Now we’ll need to install the packages which will support and enable the actually running API process. Do this in your terminal:

$ yarn add \
    cors \
    debug \
    express \
    express-list-endpoints \
    knex \
    knex-db-manager \
    morgan \
    pg \
    pg-escape

Quickly:

• express is the minimalist web framework
• cors is a little express middleware package that enables CORS for our API
• express-list-endpoints is a nice package that enables us to output a list of all the endpoints of our API. Good for self-documentation.
• debug and morgan are for logging

Now we’ve got all of the tools and packages installed that we’ll need to support writing some actual code.

Development Scripts

One more thing to do before we can start writing code. We’ll add a bunch of scripts to package.json to help us do some common tasks. Add the following scripts object into your package.json:

"scripts": {
  "build": "babel ./src --out-dir dist",
  "clean": "rimraf dist",
  "dev": "NODE_ENV=development dotenv yarn dev:logs",
  "dev:logs": "DEBUG=$LOGGING_BASE:* yarn server",
  "lint": "eslint . --ignore-pattern '/dist/*'",
  "migration": "knex migrate:make --migrations-directory ./src/database/migrations",
  "prod": "yarn clean && yarn build && yarn server:prod",
  "server": "babel-node ./src",
  "server:prod": "NODE_ENV=production node ./dist",
  "start": "yarn prod",
  "test": "NODE_ENV=test mocha --recursive --require @babel/register",
  "watch": "nodemon"
 },

• clean: deletes the dist directory
• lint: lints the source code
• migration: creates a new knex migration and sticks it in the ./src/database/migrations directory
• prod: cleans and builds the project using existing scripts then starts the server via server:prod script
• server: uses babel-node to kick off a development server using the src directory
• server:prod: sets the NODE_ENV environment variables to production then starts node by pointing at the dist directory, which is where our built production code lives
• start: simply an alias for the prod script

See the final package.json here: https://github.com/decentorganization/decent-api/blob/master/package.json

PHEW

Without yet writing a single line of code, we’ve:

• installed some system development dependency tools
• installed some project development dependency packages
• configured our development tools
• installed project dependency tools
• created some scripts to help us do common development tasks

Join me in the next post and we’ll start diving into some real code!

Originally published at https://blog.decentlabs.io on October 17, 2019.