Potential Pitfalls of Using NodeJS in Production

Do take note that Koa 2 is making big waves in the NodeJS space. The article links to details on Promises, Generators & async/await approaches.

As a disclaimer, I would like to mention that for the past 5–6 years I’ve run many production apps to eCommerce platforms based on Rails, with automation ranging from Chef, Ansible to AWS OpsWorks (to just name a few); I’ve worked on solutions for Raden.com to Autoglym.com plus many more.

The bulk of these systems relied on Spree (now Solidus) for the eCommerce aspect, with bespoke tailoring of the checkout flow and payment integration processes (usually via ActiveMerchant) and API integrations with the clients ERP of choice — from Microsoft Dynamics NAV to SAP etc.

Almost 6-months ago, I was approached by a startup to assist with their undertaking of assisting with their MVP — and like most startups, this code-base had “CHURN” written all over it. The proverbial cart was truly before the horse then, and still the same even today.

The battle between a short-financial runway, bootstrapping an ROI funnel and pragmatic business pressures, paint a similar picture for most startups.

Since local talent comprise of a much larger pool of ‘Javascript’ developers — the sort who know a bit of PHP & use jQuery as a crutch (or mix vanilla JS & jQuery ad-hoc!) — the project continues in NodeJS.

This is a brief look at some of the largest hurdles that have presented themselves.

1. NPM: The Package buffet — free for all.

I share much of the sentiment shared in ‘After a Year of Using NodeJS in Production’ by Gavin Vickery — as a lot of the pain-points I have experienced, have already been succinctly articulated by him.

Dependency management is exacerbated by the magnitude of choice provided via the NPM eco-system. Whilst choice can be a good thing, I feel too much of it can be a hindrance too. As a sniff-test — I like to go through each package’s Github page, issue-tracker, just to see how well they are being supported and maintained, as well as glean through internals.

However, it’s impossible to master (Node) in a realistic sense given that every code-base you come across in the future, has a high-degree of having a very different make-up of dependencies. The same could be said of other eco-systems, but none are of similar magnitude.

For new comers though, it is advisable to get to grips with the most popular dependencies, such as PassportJS, which has been excellent.

NPM has also lead to dependency bloat, where authors of libraries tend to build upon even 10–20 line ‘libraries’, just because it’s more convenient (really?) to add a 10–20 line package to their library rather than include that same code as a utility function.

And this leads me to Yarn, which is the defacto package manager that we are using — for better or worse, since it is quick at resolving dependencies.

2. NodeJS/Express: Bare Bones & Bring Your Own Security.

This would be the largest concern, especially when NodeJS is waved around by those of little experience.

The code-base I mentioned earlier, started out without any ‘security’ packages included, and even though I added lusca as a dependency, due to reasons out of my control, this app runs in production without any CSRF or CORS protection (skipping other levels of basic security as well).

I normally do not care too much for opinionated frameworks, but at least when it comes to aspects such as basic security, it wouldn’t be such a bad thing for this to at least be included in by default, plus provide means for overriding/replacing.

I’ve come to appreciate the degree of security offered by Rails ‘out of the box’ and reminds me of an excellent talk by Yehuda Katz ‘Why Rails is Hard’, where he explains that there’s a level of unknown security hazards that are continuously being patched and handled in Rails (and maintained too!) by the community — this is of immense value to those leveraging Rails, as it means those implementing Rails solutions, do not necessarily need to spend as much time diving into the security space to protect their apps (although, I’d urge you to do so, if you can spare the time).

3. Callback Hell Antipatterns

This is a fun one, and to get around this, our code-base is a cluttered mix of old-school query('SELECT * FROM ...', function(err, rows) { ... }) based calls.

I made sure to leverage Promises early using Bluebird, as this is the only saving grace till Node 7 can be adopted for async/await. Its quite good, fast and does a nice job of making things “just work”

There are other options such as co generators though as well as the async package.

Of course Promises themselves aren’t a silver-bullet especially when it comes to error-handling.

4. Exception (or Error) Handling. LOLWUT?

Good luck with that. Yes, seriously. Here’s quote from Gavin —

Coming from other languages such as Python, Ruby or PHP you’d expect throwing and catching errors, or even returning an error from a function would be a straightforward way of handling errors. Not so with Node. Instead, you get to pass your errors around in your callbacks (or promises) — thats right, no throwing of exceptions. This works until you’re more than a few callbacks deep and trying to follow a stack trace. Not to mention if you forget to return your callback on an error, it continues to run and triggers another set of errors after you returned the initial one. You’ll need to double your client invoices to makeup for debug time.

For the time being, I’ve tried to couple use of Honeybadger wherever there’s a try/catch block or a promise.then().catch() block for a Promise, so as to provide further context along with a stack-trace. It’s still a struggle, and boy, do I miss Rails exceptions about now!

5. Migrations — Do You Even Need ‘Em?

New comers to Node will not have much of an idea of what it takes to run robust systems at scale in Production or be aware of Continuous Integration practices.

Here’s a quick summary of the value of migrations —

Shorter release cycles often require database changes. Having those changes under version control allows us to make changes safely and in a more consistent way. A crucial part of following such methodologies is the evolutionary design which can help us keep track of how the database evolved along with our application. That way our data can rollback at any point of the history hand in hand with our app.
Evolving our database design along with our core functioning data during the application development, and doing that in a seamless way, requires the use of more sophisticated processes like Database migrations, which essentially are the management of incremental, reversible changes to our databases.
Database migrations are all about treating our database as another piece of code while applying core principles of Continuous Integration. Publishing new features along with the database changes, without requiring manual Ops work and removing any hands-on session from the DB is the way to move forward and create robust apps.

The platforms that I normally craft for my clients are typically subjected to the twelve-factor methodology, however, the last code-base I spoke about does not have any migrations.

As such, our current workflow is one that hails back to 1980s (or older) approach of copying tables and data from a development database to production — this is an absolutely painful process, and data-integrity… well, there is none at the moment. Good thing it’s a startup right? #shithappens #movefast

6. Data Access Objects (DAO) vs. Object Relational Mappers (ORMs)

In hindsight, I should have moved this higher up the list, but this is another crucial area in which I personally found options lacking in the Node space.

However, I have heard that Kraken.js has options for DAO, so whilst I haven’t checked it thoroughly, I’d suggest you do so along with Knex.js and Sequelize.

Node in Production

The approach that I have taken is to build a stack in AWS that implements TLS/SSL offloading and running the NodeJS app instance within a Docker container, with Nginx routing traffic to the app container and serving/caching static assets.

The Node instance itself is managed by PM2.

Thoughts on Node/Express

Whilst Node has its warts, as do all languages/frameworks/stacks do — there are some aspects that are great. For example, we needed to generate some PDFs and PDFkit has been great at that particular task.

However, I would not recommend it for large-scale products, unless you are able to —

  • Dedicate time to plan the base code-base, including dependencies etc.
  • Curate dependencies, security solutions from scratch.
  • Take your time in crafting your solution.
  • Leverage unit testing and BDD practices as well (Selenium/PhantomJS + user stories).
  • Test and test again (and automate everything).
  • CI/CD/Zero-downtime deploys are a must. It is 2017 after all.

Whilst Rails has its demons as well, I would consider having Rails run the backend as an API based app, and architect the frontend with NodeJS making API calls to the Rails backend. ‘Why the Hell Would You Use Node.js’ proposes that NodeJS should not be used for web-apps with a relational backend.

It’s possible and not uncommon to use Node.js solely as a public-facing facade, while keeping your Rails back-end and its easy-access to a relational DB.

ActiveRecord would be considered a time-tested ORM given its exposure in production at Basecamp, and potentially is more feature complete than ORM options in NPM-land.