The RealWorld project is an ambitious effort to gather practical examples of a single web application, written in a wide variety of languages and frameworks. The application is called Conduit, which is essentially a clone of some of the core features of this very application (Medium!). The idea is this: RealWorld provides an interface spec and an API spec (i.e. frontend and backend), developers implement one spec without worrying about the implementation of the other, and then we find ourselves with several hundred unique “stacks” by pairing any frontend example with any backend example. Ever curious to see a Vue frontend talk to a Rust backend? Aurelia and Rails? Dojo and Go? You get the picture: there are a ton of rich examples to explore.
We’ve recently added a new such example to the list by implementing the RealWorld API with hapi pal: a combination of nodejs, the hapi web framework, Objection ORM, and SQLite, all under hapi pal’s methodology/toolset for building APIs. This article takes an in-depth look at the stack and the codebase used to implement the RealWorld API spec, which includes features such as:
- Users can sign-up and authenticate with JSON Web Tokens (JWTs).
- Articles can be created by users, and later edited or deleted only by their owning user.
- Articles may be favorited by users.
- Tags may be attributed to articles, which can later be used in paginated article searches. These searches can additionally filter by articles favorited by or written by a given user.
- Users may follow other users, and each user has a paginated feed of articles by their follows.
- Users may comment on articles.
- Articles must have unique slugs generated from their titles, while titles need not be unique.
- There are particular requirements around response payload formatting and errors.
And a closer look at the stack:
- hapi, our nodejs web framework of choice for its stability, high-quality ecosystem and architecture, bright community, and security-mindedness.
- Objection, our favorite ORM for its SQL-savviness, emphasis on queries over records, flexibility, top-notch maintenance, and helpful community.
- SQLite, an easy-to-install (via npm!), transactional, and relational SQL database—perfect for a widely-shareable project like this.
- And of course hapi pal, an ecosystem of tools and best practices for the hapi web framework. Pal primarily provides us our project structure and application architecture while letting hapi, Objection, and SQLite do what they do best. Beyond that it also gives us some great scaffolding and debugging tools.
Let’s begin!
With all that said, let’s take a look under the hood. We suggest following along with the codebase, which you can find right about… here. You can also find it deployed as a tweakable, hackable live demo hosted on Glitch. If you’re looking for a lighter, more guided introduction to hapi and hapi pal, please feel free to hop on over to our Getting Started tutorial, which utilizes the exact same stack as this project.
Directory structure
The codebase is based upon the pal boilerplate, which splits-up projects into two top-level directories: lib/ and server/.
The lib/ directory contains all the core functionality of the application. Strictly speaking, it constitutes a hapi plugin, which is a portable and well-encapsulated way to articulate a web service. The sub-directories under lib/ each define some pieces of the application: routes, models, services, other hapi plugins, etc. Most of the contents of lib/are picked-up automatically by pal's file-based hapi plugin composer haute-couture, and were scaffolded using the hpal CLI. Without haute-couture we would instead make many imperative calls to the hapi server interface; for example, we would call server.route() rather than creating a file in lib/routes/, server.auth.strategy() rather than a file in lib/auth/strategies/, server.register() rather than a file in lib/plugins/, etc.
The server/ directory contains all configuration and code required to deploy the application. Given that lib/ exports a hapi plugin, server/ is primarily responsible to create a hapi server and register the app's plugin with some configuration provided by server/.env.
The reasoning behind this separation of lib/ and server/ is explained in detail in an article: The joys of server / plugin separation.
The model layer
This application’s model is based upon Objection ORM, which is integrated into hapi using the schwifty plugin. Each model lives in lib/models/ and corresponds to a particular table in the SQLite database: Users, Articles, Comments, and Tags.
Our model layer is very light. It represents a thin mapping between the application and the database, and enforces some basic rules related to data integrity: setting createdAt and updatedAt fields, computing an article's slug from its title, and validating column values when they are persisted to the database. The models are used to interface with the database via Objection's wonderfully expressive SQL query builder which extends knexjs.
You will find that models are used exclusively within the service layer, which is detailed below.
The service layer
The service layer represents a sort of “headless” interface to all the actions and means of fetching data within the application. You’ll find a service method that causes one user to follow another, another to fetch a user’s articles feed, etc. In this way our route handlers/controllers have a means of sharing common logic (“how do I get an article by its id?”) while hiding away the implementation details (e.g. details of the model) in a common library. The service layer is actually generic enough that it could also be re-used to write a different interface to the exact same data and actions, such as a CLI.
We endow our application with a service layer using the schmervice hapi plugin. Alongside the plugin, schmervice also ships with a base service class that provides some useful and convenient functionality, such as access to the hapi server and application configuration (plugin options), integration with the server’s start/stop lifecycle, and the ability to leverage hapi’s robust system for persistent caching.
This application has three services: the ArticleService, the UserService, and the DisplayService, all in the lib/services/ directory. Each service is a class that extends schmervice's base class. The ArticleService comes with methods such as create(), findBySlug(), and addComment(); it provides an interface to articles, comments, tags, and favorites. The UserService comes with methods such as signup(), findByUsername(), and login(); it provides an interface to users, following, and authentication.
Lastly, the DisplayService is responsible for enriching and transforming user, article, comment, and tag models into objects transferred by the API endpoints. This allows us to defer to the ArticleService and UserService to worry about the details of fetching/searching articles and users in various ways—which are complex in their own right—without having to also be concerned with composing the data in these equally complex API responses. For example, the articles list (GET /articles) must be able to paginate while filtering by tag, author, or favorited status; then the API response must additionally include specially-formatted information about whether the author of each article is followed by the current user (if there's a logged-in user), and whether each article is favorited by the current user. In the RealWorld specification there are also multiple representations of some models; for example, a user presents differently when the current user is acting on or asking for information about themselves, versus a separate user (a.k.a. a "profile"). That's a lot of responsibility, so we decided to decouple fetching from enriching/formatting! Luckily, as you will see in the DisplayService, Objection's loadRelated() feature is especially well-suited to this approach.
The final point of interest in the service layer is its convention for transactions. Objection’s handling of SQL transactions is very ergonomic. We take advantage of the fact that you may optionally specify a knex transaction object at query-time to any Objection query. By convention, each of our database-backed service methods take a transaction object as an optional final argument. That transaction object is simply passed down to any queries inside the method, and commits/rollbacks of a transaction are handled by the caller of the service method. In this way any database-backed service method may be composed into arbitrary transactions with other service methods without the caller having to understand the underlying queries being made. More on this in the next section on routes!
Routes
At the end of the day, we do all this work so that we can create some routes, or API endpoints. Each route consists of a hapi route configuration placed as a file in lib/routes/. These configurations provide information about the matching HTTP method and path; validation of incoming query, path, and payload parameters; authentication; and a handler or controller implementing the logic behind the route.
Validation is specified using hapi’s robust joi validation library, which is the same means of validation used by our model layer. Since the routes and models use the same means of validation, routes are able to refer to the model’s validation. For example, when a user logs-in the payload contains an email parameter that must be a valid email; in the route configuration we defer to the User model's definition of a valid email and mark it as a required field: User.field('email').required().
The route handlers themselves are relatively light. They generally compose payload, query, and path parameters, and the user’s authentication status into one or many calls into the service layer, then return a response. Handlers are also responsible for the transactional integrity of their calls into the service layer. For example, if a user makes requests in quick succession to favorite then unfavorite an article, each of those requests must come back reflecting the proper state: there should be no way for the request to unfavorite the article sneak its way in so that the request to favorite the article responds with favorited: false, or vice-versa. So, handlers will often generate a transaction object using a thin helper around Objection.transaction() (defined in lib/bind.js), then pass that transaction to the various service methods that it uses. As mentioned in the previous section, handlers typically end with a call to the DisplayService, whose sole purpose is to format and enrich information about the model (users, articles, comments, and tags) for API responses.
Authentication
Per the RealWorld API spec, authentication occurs via signed JSON Web Tokens (JWTs). There are essentially two sides to this form of authentication:
- The application must hand-out a JWT to a user when that user provides a matching email and password.
- The application must verify the authenticity and contents of the JWT when it is passed with future requests.
In order to hand-out a JWT, we have a login endpoint that performs the process described above by calling into the service layer. In particular, the UserService has a login() method to lookup a user by their email and password, and a createToken() method to create a JWT containing the user's id. Aside from the user id, createToken() also needs a "secret key" in order to sign the token. In our case, we obtain the secret from our application's plugin options (this.options.jwtKey), which the UserService has access to because it extends the schmervice base class. The jwtKey plugin option is set using the APP_SECRET environment variable inside our app's deployment, configured within server/.
In order to verify the authenticity and contents of the JWTs passed with future requests, we utilize the hapi-auth-jwt2 (registered via haute-couture in lib/plugins). This plugin creates an "auth scheme" for JWTs which we configure into an auth strategy in lib/auth/strategies/jwt.js. The auth strategy determines the details underlying our JWT auth: tokens should be signed using a certain hashing algorithm, with a certain secret key (as described above); the token is further validated by looking-up the user whose id is stored on the token; etc. One the auth strategy is created in this way, it's trivial to protect an API endpoint with JWT authentication using hapi's auth route configuration, as can be seen on the route for article deletion.
Error handling
The RealWorld API Spec is particular about the format and HTTP codes that our application responds with. In order to meet those requirements we wrote a centralized hapi request extension, which can be found in lib/extensions/error.js. This request extension is a hook into hapi's request lifecycle to process all responses right before the server responds. In hapi parlance this request extension point is called "onPreResponse".
There are a few different types of errors that are encountered in the app and pass through this request extension. Whenever a route needs to express a standard HTTP error, its handler will throw a boom error, which is standard in the hapi ecosystem. Other errors also may come from within the model layer (e.g. when a record is not found) or from a route’s request validation (these are already considered “400 Bad Request” boom errors). We interpret errors from the model with help from Objection’s objection-db-errors plugin — which normalizes database errors across the various flavors of SQL — and avocat which further transforms them into hapi’s preferred boom HTTP error objects; for example, a uniqueness violation may be transformed into a “409 Conflict” boom error. Once the error is interpreted as an HTTP error, the final step is to simply format them into the shape preferred by the RealWorld specification.
Conclusion
Well, that’s just about all we have for you for now! We hope that the deep-dive proved useful, and that you have some idea what a hapi pal codebase might look like out in the wild. We work on APIs like this every day, which is why we insist that pal is for the practicing, working developer looking for practical tooling. If you’re looking for more info, there’s a community of us who would love to chat with you! Please join us in the #hapipal channel in the hapi hour slack and say hello 👋
Your pals,
Devin (@devinivy) and the pal team
