☝Above, a pal takes a stand. Photo by Paolo Nicolello

A stand against the “easy” nodejs backend

On the homepage of hapipal.com we try to make a memorable first impression when we actively broach the topic we know you’re wondering about: “is this tool going to make things easy for me?”

Writing scalable web services is hard work, and we don’t think there’s any way around that.

“Easy” isn’t exactly what we’re going for.

Now, that’s not to say that we think pal is hard to learn—quite the opposite! (As far as we can tell, folks find it delightful.) We just acknowledge that we have probably not solved all the hardest problems that you have in front of you as you embark on authoring your next web service. You’re going to have to deal with countless issues—sometimes massive cross-cutting concerns of your entire application—that the JS framework/tooling scene has a habit of casually grouping into broad terms: “performance” “security” “realtime” “business requirements”. And we take issue with calling any of this stuff easy.

In fact, we designed hapi pal in such a way that acknowledges the difficulties that lay ahead for you: basically, pal is prepared to get out of your way when it comes time for heavy-lifting. Let us explain!

“Build a production-ready realtime backend in days”

We’re used to seeing claims like this (👆) as theses for blog posts and tutorials, as hooks on framework homepages, in Twitter posts, in contentious Hacker News comments, etc. Whether intentionally or not, it is designed to play into our anxieties about choosing the right tool for the job, as we swirl around in a sometimes dizzying sea of JS tooling and frameworks.

If I try this framework out and it’s taking me over a week to build my application, is it because I’m doing something wrong? 😓

Rest assured, if your application is taking more than days to complete then it doesn’t necessarily mean there’s anything wrong with your technology choices or your approach. Depending on the inherent complexities of the problem you’re servicing, data consistency guarantees, security requirements of your business/country/industry, service-level agreements your API must uphold in the way of availability and responsiveness, your Ops budget, the number of anticipated concurrent users and magnitude of traffic influxes, the size of your team, etc. it would not be at all weird for a project to take months or longer under any framework. And if the term “production-ready” doesn’t encapsulate the laundry list of concerns above, what are we even talking about?

Now, if we all agree that your production-ready application oughtn’t be assumed to be easy to build, what are the technical ramifications for framework authors and tool builders? It all comes down to the abstractions that these libraries choose to introduce their users.

Abstraction leaks

The only thing worse than memory leaks are abstraction leaks. You know that feeling when you feel cornered by a “missing feature” of one of your project’s dependencies? When some library or reusable module feels like it carries existential weight with it wherever it shows-up? When the code that you write using a library seems so distant from its performance characteristics? That’s what a bad leaky abstraction feels like.

A common definition for this you’ll find on wikipedia: “a leaky abstraction is an abstraction that exposes details and limitations of its underlying implementation to its users that should ideally be hidden away.” I would opt to amend that, because I think these leaks go both directions. If a library requires changes to its implementation in order for application code to circumvent its limitations, that’s a leak too. For example, how many times have you tried to use some particular MySQL or PostgreSQL feature only to find that you were completely barred from it until it’s implemented in your ORM of choice? That’s not an ORM with missing features—it’s an abstraction leak.

Yes, abstractions are beautiful. They’re also bold and sometimes fraught with hubris. After all, creating an abstraction is an assertion that you can solve multiple different problems at the same time. In a world of development where we all share problems that are similar but not quite identical, what does a healthy abstraction look like? Just how far ought framework and library authors go in attempting to solve others people’s problems?

There’s a balance to be struck here between having a tight, minimally-leaky abstraction that doesn’t offer much value, and a massive-but-leaky abstraction that provides great value only within its limits. Libraries that claim to make your life “easy” or “quick” have a tendency to end in one of these “massive-but-leaky” abstractions. And the reasoning there isn’t so controversial: optimizing for easiness implies feature richness over the ability to customize and fine-tune. Needless to say, we’ve sort of obsessed over this balancing act in authoring hapi pal, and we’ve learned a thing or two along the way.

How pal takes a stand

We designed hapi pal with a hyper-sensitivity to all that we acknowledge as “hard.” Sensitivity to all the problems that we could never anticipate, that you are inevitably going to encounter. As such, we have spent a massive amount of effort to ensure pal can get out of your way when the going gets tough. In the end it’s a bunch of small design decisions that we hope add-up to a flexible, minimally leaky experience. Here are a few examples of the many “escape hatches” built into pal,

• haute-couture isn’t a hapi plugin itself. We want you to be able to use haute-couture to compose your hapi plugins from files, but also want you to feel free to write and run any other plugin code you may need—for example, if hapi introduces a new feature that haute-couture doesn’t immediately support. So we made sure that haute-couture can be used as a sub-routine of any plugin, and you can write all the custom code you need!
• Objection ORM is really just a glorious SQL query builder. When deciding on a data access layer for ourselves, we wanted to break away from the massive abstractions ORMs traditionally introduce. Objection strikes a wonderful balance, where it focuses on top notch support for features of SQL data stores specifically (composing transparently with the knex query builder) rather than attempting to create a generic interface for working with “records.” This allows us to drop down to the metal and work closely with all the hard stuff that we need like transactions, column types with specialized indexes (e.g. tsvectors, JSONB, and geospatial types), aggregations, join semantics, etc. Objection makes working with model/table relationships and these critical features of our RDBMS feel organic.
• schwifty hooks-up models to db connections for you… unless you do it first. Schwifty is a hapi plugin integrating Objection ORM as a model layer for your application. It respects hapi plugin boundaries by letting you configure different db connections for your models on a per-plugin basis, e.g. to handle the case that you’re composing multiple pluginized hapi apps into a single deployment. We find this to be very flexible, but we also acknowledge that there may be more complex use-cases out there: what if someone has a single legacy users table that lives in a different db from all their other models? In that case, no sweat—schwifty is designed to get out of your way if it sees you manually hooking-up db connections to a given model.
• the contents of scaffolding files made by the hpal CLI is totally customizable. The hpal CLI has a command hpal make that allows you to scaffold files for new routes, models, services, auth strategies, etc. We have reasonable default contents of those files, but perhaps you’d like to add a little flair or make it match your coding style more closely. The file contents can be customized by simply creating a haute-couture amendment in your .hc.js file with an example property, which means you’re free to make hpal make behave however you like! In a similar vein, you can use haute-couture amendments to change the meanings of directories in your project: perhaps you want models/ to contain Mongoose models rather than Objection models, for example. We’re happy to get out of your way so that you can work within your own requirements.

There are a bunch more examples of this in the pal universe, but these are a few important ones that jump to mind. Several modules/libraries are mentioned above. All these libraries are also designed to function and “make sense” entirely on their own, but if you’re looking for a more integrated experience, we highly suggest checking out the pal boilerplate, which comes setup to author hapi applications with top notch hapi debugging tools, a test suite and linter, a configuration and deployment strategy, and a bunch of “flavors” e.g. to integrate schwifty and Objection ORM, Swagger UI, templated views, and more. Not to mention a very welcoming community on the hapi hour slack (find us in #hapipal!). We’d love to have you.

Your pals,

Devin (@devinivy) and the pal team

Important P.S.!

We want to make it clear that hapi pal is not here to point fingers! No content of this post should be used to shame open-source maintainers for their “leaky abstractions” or shameless promotion of “the quick and easy.” It’s always your responsibility to understand the abstractions, approaches, limitations of the dependencies you introduce into your application. The greater JavaScript community has a long tradition of touting libraries as quick/easy solutions to complex problems, and we are simultaneously subject to hype, tooling anxieties, “JS fatigue” as much as we are instruments of it. We’re just hoping to acknowledge these issues, counter them systemically, and write some delightful hapijs tooling while we’re at it. 💞