React, IoT, Bots, APIs — Why Web Development Needs a Change

The web ecosystem is growing at an unprecedented rate. From single page application and dynamic UI frameworks like Angular, React and Vue to an overabundance of mobile apps, IoT devices and now chat bots, we’re beginning to learn that the basic assumptions developers have been making for years about how to build and architect web applications are tearing at the seams. We’re seeing a major shift in application requirements. As we rush to hack together solutions, developer toolchains are exploding in complexity as we try to hold everything together with ad hoc “duct tape” fixes.

There needs to be a change. It’s happening, but the process needs a stronger push forward. The first step is education: what’s the problem, why is it a problem, and how are we going to craft the next generation of products and services? It all starts with changing the way we think about web development.


The Browser as a Platform, Not a Client

The main assumption we need to reexamine is that the web browser is the de facto standard user-facing client of the web. It’s simply not the case anymore. The browser is now a platform through which applications are downloaded, compiled and consumed. When we think about development through this context we begin to see that our “single page applications” have become the true client layer, the term “page” is an anachronism, and these clients are now first-class citizens of the web.

Peering through this lens we begin to see that our development ecosystem is actually poorly equipped to deal with the creation, deployment, management and distribution of these sorts of apps. The majority of websites still rely on a single web monolith responsible for compilation and delivery of static assets, server-side HTML template rendering and SEO, and some sort of computational API layer with data storage.

Our old assumptions about the browser, which have supported the development ecosystem for the past decade, led to a simple single client / single server architecture you should be familiar with:

This setup is just no longer tenable. The amount of development tooling focused around each specific client (JavaScript app) is immense. For every new dev tool, there’s a new build chain associated with each and every server-side solution. Rails packages for React. Django packages for Angular. Developers are frustrated and fatigued. We’re stuck because we still think we’re building for the browser.

Let’s take a step back.


The Next Generation of Web Architecture

Once we reframe our development target as a JavaScript app instead of the browser, we begin thinking of constructing these applications in the same context as mobile, IoT (connected hardware), and even chat bots. These are all just different clients that exist as first-class citizens of the web, each consuming data from an online service.

With a bit of rearranging, we can decouple a lot of functionality. This avoids duplication of effort, follows DRY (don’t repeat yourself) principles and allows us to optimize for service (deployment, distribution and management) requirements. The architecture looks like this:

There’s something important to note here. I’ve separated out “Discovery and Branding” (old school, monolithic apps) into their own category of client. Make note that we’re not throwing the baby out with the bathwater. Server-rendered HTML pages still have their place, but they should communicate with your database via your API. They should not have direct database access. These old “servers” are actually part of a client, a bridge layer between basic browser functionality and our API servers. They’re just like modern fully-JavaScript web apps, with the rendering layer living on another machine we own.

Rethinking Development

In order to properly decouple these application layers and clients, it is extremely important that we give each client its own development, deployment and delivery pipelines. Why? Because each of these layers has completely different request load, scaling and technical requirements. The best API server, best React app server and best traditional web server are unlikely to ever be built using the same technologies. They’re just too different, and their individual infrastructure requirements vary greatly.

This means that we must build individual application servers for all of our React, Angular or Vue apps. While it might seem like a lot of effort, we want to keep our concerns separate. For example, an API server needs a live database connection. An application server does not. Conversely, an application server needs an asset compilation pipeline where an API server does not.

The Emperor’s New Dev Tools

Looking at this decoupled ecosystem (an abundance of clients and a single API server) we begin to immediately see a problem — developer tools are generally poorly equipped to deal with the problems that pop up around multiple application environments, deployments and app distribution. The recent obsession with realtime connections everywhere poses scaling and synchronization issues between machines. “Universal JavaScript” — tightly coupling your server logic and client logic — places an artificial binding between technologies growing at different rates, sometimes orthogonally to each other. It also naively continues to treat the browser as the client, not your application.

We have a ton of tools that let us quickly run a server in our console, but we’re left utterly clueless when it comes to how to manage hardware behind the scenes. When does your deployment process involve uploading to a CDN? How do you set that up? It’s no wonder we’re so slow to adopt a completely decoupled architecture — running everything in one place is just easier. If we run a web monolith, we don’t need DevOps engineer to manage every piece of our application. This is a naive approach to development, and we need developer tools that help to remove these sorts of headaches, specifically focused around decoupled architectures serving multiple types of clients.

Without understanding these problems, much of the community is simply looking at a naked Emperor —we see the tools we’re using as sufficient, and we don’t disagree for fear of seeming left out. Many toolchains we utilize aren’t adequately solving the most expensive problems we all have, instead opting to create more abstraction layers on top the code we write — not the architecture we build. Many have had the courage to say, “this doesn’t look quite right,” but we need to act together. The development community must collectively put aside their egos and biases to agree that, hey, the Emperor isn’t wearing any clothes. Let’s give the guy some pants.

Building the Future of Dev Tooling

What does the next generation of web tooling look like? David Heinemeier Hansson began us on the “path of enlightenment” with Rails. Hand-picked design and architectural patterns, a curated technology stack. Open source. The popularity of Rails speaks volumes to its utility. What does the future look like? We need to start thinking of curation in terms of more layers of the stack, outside of just writing code — if only so that we can make sure developers are focused on what they’re best at, working at the layer of computation.

This means that developers should be expected to be developers, not a miniature IT department. We write code. We build product. Every minute we’re not taking advantage of our core competency is a minute of productivity loss. Let’s try to figure out how to abstract away the metaphor of hardware completely so we can focus on writing more code and shipping products people love.

Where Next?

These sorts of problems are the ones I’m focused on solving, and I’m excited to work with others to create the best solutions to new and emerging problems in architecture and infrastructure. I built Nodal, an open source framework for creating, deploying and distributing decoupled API services to focus on providing the best API layer possible and remove the headaches associated with infrastructure management. In adjacent spaces there are sole developer projects like Vue tackling giant ecosystems (React), generating alternatives that seem like good targets for delivering promising results.

This is a space we can make better, together. I recently founded a startup, Polybit (literally “many bits”), to craft the best open source software to tackle these evolving problems. This is only the beginning of many new, creative and thoughtful solutions to help support a changing ecosystem and the newest frontiers of web development.

Let’s build the future. E-mail me if you’d like to help — keith@polybit.com.


Keith Horwood is an avid open source contributor on GitHub, a Toronto-born Canadian, and lives in San Francisco. Previously the Engineering Lead at Storefront, he is now the Founder of Polybit Inc., where he’s focused on providing open source solutions to emerging web technology, specifically architecture and infrastructure problems.

You can read some of his previous articles, Hello, Nodal — Building Node.js Servers for Everybody, Single-Page Applications: Building a Web Stack That Works and for something a bit more technical, Using Graph Theory to Build a Simple Recommendation Engine in JavaScript.

You can follow him on Twitter, @keithwhor.