How to Write Applications that Scale, or Writing Your Code as a Service, Part I.

Scalability is one of the hardest problems you’ll have to solve as a developer. It’s difficult to maintain a product when it has a long or indeterminate lifespan and you have to account for growth in users, code base, the development team, and the systematic outdating of tools and frameworks that is commonplace in web development now. The solution to the problem can be conveyed in one succinct sentence, “Write good code.” That’s a gross oversimplification, so I should explain. The goal is to be able to write software that can be maintained by ever changing sets of eyes, deployed on changing platforms, and so on, so the only thing you can do is write code that people understand. Whether the people who need to understand it is a new set of developers, or you 3 weeks in the future, it is imperative to write code that people can understand without any question, any additional thought. this is how you maintain productivity, alone or on a team.

Style is obviously a subjective matter, but there are plenty of rules that can be implemented; the key to making something widely and easily understandable is to enforce simple rules uniformly. I’m going to discuss four categories that are particularly difficult to map together, largely because of the wide variety of options to choose from.

  1. Uniform editor configuration.
  2. Programmatic safety nets, like linting and static typing.
  3. Build processes that can scale with technology.
  4. An architecture that is based on the idea of services and modules, where pieces can be swapped out without breaking everything else.

The first two points I’ve mentioned are processes you run in the background and don’t require building out any tooling for the project. Today, I’m going to discuss these two items, and will write individual articles about build processes and service based architecture as they are lengthy topics that warrant their own discussions.


Editor configuration

The simplest point to discuss is editor configuration, so we’ll start there. You’ve probably never really thought about your editor settings before; how many spaces is the size of your tab, or is your tab writing actual tabs, or spaces, and how is a linefeed written? This is one of the more basic and immediate problems of working on a team, and is particularly important at a time when open source and community built projects are so prevalent, thanks to Github. It may not sound like much, but when you have one file where four or five people have made changes, and they all use different width tabs, represented by either tabs or spaces, you’ll quickly find that the resulting code looks pretty rough. The solution couldn’t be more simple; thankfully, there’s a dotfile for it.” If you drop an .editorconfig file in the root of your project or any child folder, it will define how the editor writes characters to that file, overriding their original settings. All modern browsers respect these configurations, including Sublime Text, Atom, Vim, and so on. Let’s take a look at an example:

[*]
root = true
indent_style = space
indent_size = 2
tab_width = 2
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true

That’s the editor config that I drop in every project, the rules that I find to be the most comfortable to work with. Yours might be different, and that’s fine. All that matters is that the rules are always the same amongst the team members of any given project. You can configure the group of rules — everything that follows the bracketed asterisk [*] — to target globs. Taken from the editorconfig site, all of the following glob types work:

# Target JavaScript & Python files 
[*.{js,py}]
# Target just Python files
[*.py]
# Only the Makefile
[Makefile]
# All of the JavaScript files in the lib folder
[lib/**.js]
# Either package.json or travis.yml
[{package.json,.travis.yml}]

While all of these are possible, I find it best to just maintain the same settings across all filetypes for the sake of consistency.


Lint and type: programs that help you program your programs

This is a tough sell for some, as it takes extra time to write code that polices your other code. And the honest truth is, you probably don’t have to do all of these on every project you work on. Timelines, budgets, and other resources often get in the way of linting, testing, and statically typing your code on every project, but I will argue that at the very least, you should be linting your code. Since I’m so adamant about that one, we’ll start there.

Linting is incredibly easy, and also really confusing. It’s easy because, in reality, it’s as simple as picking some rules and installing a tool that tells you when you break those rules. This is hard because there are seemingly infinite rules, infinite tools, and infinite ways of rendering those tools into a readable, digestable format. As a developer who works largely in JavaScript (namely with React and Babel), I find the best tool to be ESLint. ESLint works like any other JavaScript linter, but the set of rules it comes with are fairly minimal, and it’s all about adding packages that flesh out its feature set. My favorite combination of ESLint plugins is:

The best way to set up ESLint is to install it through NPM. I like to install all of my packages locally to avoid maintaining multiple global versions, which ensures that everyone who clones your repository receives the same version of the tools you used to originally write the application. When it comes to writing applications that can be maintained across a wide set of platforms and developers, this is a pretty obvious precaution to take. Let’s install these tools in a project through the command line now.

npm install --save-development eslint eslint-config-airbnb eslint-plugin-babel

And quickly, my .eslintrc file looks like this:

{
“extends”: “airbnb”,
“env”: {
“browser”: true,
“node”: true,
“mocha”: true
},
“rules”: {
“spaced-comment”: 0,
“new-cap”: 0,
“babel/generator-star-spacing”: [2, “after”],
“babel/new-cap”: 2,
“babel/object-curly-spacing”: [3, “always”],
“babel/object-shorthand”: 2,
“babel/arrow-parens”: [2, “as-needed”],
“max-len”: [2, 80, 2, {
"ignoreComments": true,
"ignoreUrls": true,
}],
},
“plugins”: [
“babel”
]
}

I want to extend the Airbnb config file, as their developers have come up with what I believe to be the most sane and least abrasive set of rules for modern JavaScript including ES6 functionality and React standards. If you’re interested, you should check out their Github repo, because there isn’t much value in me spoon-feeding you those rules right now. Additionally, I add all of the Babel ESLint plugin settings, which add strict rules around new ES7+ functionality found in Babel.

You can use any of these plugins or not, I don’t very much care, and no one else will either… unless you write lists of variables, beginning each line with a comma, you people make me sick, bunch of animals. The important thing is that you enforce linting rules, and anyone that contributes to the project has to live by these rules. To protect software as it scales, we need to make sure that things break in a very bad and severe ways when the rules aren’t followed. When we discuss the build process at a later point, I’ll explain how to force ESlint to fail your build and refuse to compile until all linting, testing, and so on all pass successfully.

Oh, and about static typing…

Statically typing variables is kind of a new concept in the world of JavaScript. Microsoft originally made this widely popular with its superset of JavaScript, called TypeScript. The goal of static typing is to apply rules around your methods and variables that dictate what types of data are acceptable. Somewhat unrelated, but one nice side effect of this is that it starts to enable intelligent code hinting if you have the tooling to support it — Visual Studio Code for TypeScript developers, or Nuclide for Babel / Flow developers. If I’m going to be honest, I fall into the latter camp, because I love Babel and Flow.

As I have already bought into a static typing language, I’m going to describe how to work with those tools. These are what I recommend, but obviously, if you choose to go with TypeScript, then you’ll see similar benefits, and if you choose neither, then whatever, I just can’t even. To install Flow, just turn to Homebrew:

# Update homebrew, you should always do this before installing
brew update
# Install flow
brew install flow
# Upgrade flow periodically, and as always, update homebrew first
brew update
brew upgrade flow

Like every other tool, flow starts with a dotfile. In the root of your project, type “flow init” to create an empty flowtype config file. Here’s a quick look at mine.

[ignore]
.*/node_modules/invariant/.*
.*/node_modules/koa/.*
.*/node_modules/koa-static/.*
.*/node_modules/express/.*
.*/node_modules/fbjs/.*
.*/node_modules/babylon/.*
.*/node_modules/babel-.*
.*/node_modules/babel-core/.*
[include]
./app/
./config/
[libs]
./interfaces/
[options]
esproposal.class_instance_fields=ignore
esproposal.class_static_fields=ignore
esproposal.decorators=ignore
module.system=haste
munge_underscores=true
strip_root=true
suppress_type=$FlowIssue
suppress_type=$FlowFixMe
suppress_type=$FixMe
suppress_comment=\\(.\\|\n\\)*\\$FlowFixMe\\($\\|[^(]\\|(\\(>=0\\.\\(1[0–6]\\|[0–9]\\).[0–9]\\)? *\\(site=[a-z,_]*www[a-z,_]*\\)?)\\)
suppress_comment=\\(.\\|\n\\)*\\$FlowIssue\\((\\(>=0\\.\\(1[0–6]\\|[0–9]\\).[0–9]\\)? *\\(site=[a-z,_]*www[a-z,_]*\\)?)\\)? #[0–9]+
suppress_comment=\\(.\\|\n\\)*\\$FlowFixedInNextDeploy
[version]
0.18.1

What does all of that mean? There are a few important things to note:

  1. I’m explicitly including the folders that I won’t flow to scan through. Flow will look at files in these folders that open with the /* @flow */ comment block.
  2. Flow is in its infancy, and there are certain things (like duplicate module definitions and code that contain new ES2015+ proposal features that it doesn’t understand) that cause it to more or less freak out.. I’m telling it to not bother looking at those packages.
  3. I’m telling flow that I need exactly version 0.18.1.
  4. The [options] block is particular to how I work. You’ll notice that I’m telling flow to ignore some esproposal features it doesn’t understand (this only works for some features), and I’m also declaring a comment regex pattern that will instruct flow to ignore the following lines. This was largely stolen from code at Facebook.

So what does that give us? We can now run flow check in the root of our project and it will walk through our code and ensure there’s nothing weird going on with data passing through. I’ll show you some examples of how the flow checker might fail:

const age: number = 'Thirty'; // Fails
function getStringLength(str: string): number {
return str.length;
}
debug(getStringLength(true)); // Fails
type PluginConfig = {
name: string;
value: any;
author: string;
company?: string;
private?: boolean;
listeners?: {[key: string]: Function};
};
function Plugin(config: PluginConfig): void {
const { name, ...rest } = config;
this.name = name;
debug(rest);
}
new Plugin({
Name: 'My Plugin', // fails, missing required prop `name`,
value: 2,
author: 'Christian de Botton',
company: 'Brooklyn United',
private: 1, // fails, expecting `boolean`
listeners: 'yes please', // Fails, needs obj of key/function pairs
});

You strictly define what datatypes the values can be. It’s a pretty simple concept, and if any code changes in such a way that you could potentially pass the wrong value types on through, resulting in a situation where data won’t flow correctly (Get it? Puns are fun), you’re immediately notified. Ever see the cryptic error ‘Undefined is not a function’? Probably all the time, well, this will prevent that from happening again. And, as a pretty cool bonus, if you use Nuclide as an IDE, you get this for free:

Hover to see type hinting definitions. Image property of Facebook, http://nuclide.io/docs/flow/
Cmd/Ctrl + Click to jump to code definitions. Image property of Facebook, http://nuclide.io/docs/flow/

That’s all for today

The pattern we’re working toward should be pretty apparent at this point. We want to make it as difficult as possible to break our code, and when that does happen, we want to make it very apparent to everyone, immediately. As applications scale they become more frail, where one small change can have a ripple effect and break many other things. These tools that enforce very strict rules on us as developers reduce this fragility.