Software Architecture for large-scale NodeJS applications

This article consider the large-scale Javascript NodeJS/Meteor applications architecture and treat the case study of the framework KeplerJs.

This article is work in progress…
UPDATES:
19 March 2018 (Plugins dependecies section)
24 March 2018 (Templates section)
4 February 2019 (Templates section)
31 January 2020 (Package.js)

KeplerJs

I will to explain how to structure scalable applications designed to greatly increase their complexity, facilitating contributions and organizing the source code in a sustainable way from today until the end of the universe.

arrows are the dependencies, green are the base packages and the plugin packages are yellow.

Kepler is a open source geosocial framework that allow to aggregate different geospatial data sources in a single application, the data sources are many and varied then the structuring of the code requires a care in the organization of the logic that implements each of them to give space to personalization and special cases of work. It is also an open source project then is need this logic is simple to understand, any developer who wants to contribute to the project should be able to do it in a few minutes without having to study long implementation and reference manuals.
One of the problems with using new frameworks (especially in javascript) lies in the fact that their learning curve is often too steep for the new contributors. In Kepler we tried to avoid this problem by avoiding adding too many new concepts specific of the framework, but working with some simple concepts already present in the minds of javascript programmers, both novices and experts.

Basically it use a package-based architecture meaning that the entirety of its codebase resides in /packages path, then each complex module or a group of modules in Kepler are organized in Meteor packages which can be added or removed at runtime.

In addition to this standard design adopted by many famouse Meteor applications Kepler implements a plugins mechanism that allows you to split different pluggable features into different packages, this option facilitates the work of developers that will be able to connect their services or include their customizations writing a minimum amount of code and using the common API and environment from base packages and plugins.

Package.js

In Meteor and KeplerJs a package is a directory containing a package.js file, which contains roughly two major sections: a basic description and a package definition including dependecies among other packages. By default, the directory name is the name of the package. The following is an example of a plugin Osm having Edit and Core dependecies:

// Information about this package:
Package.describe({
  version: '1.7.0',
  name: 'keplerjs:osm',
  summary: 'Keplerjs Openstreetmap and Overpass API'
});
// This lets you use npm packages in your package:
Npm.depends({
  'bootstrap-tagsinput':'0.7.1'
});
// This defines your actual package:
Package.onUse(function(api) {
  // This lets what Meteor version support
  api.versionsFrom("1.5.1");
// This define KeplerJs plugins dependecies
  api.use([
    'keplerjs:core@1.7.0',
    'keplerjs:edit@1.7.0',
  ]);
...

In the building phase Meteor takes care of rebuilding the javascript code in a coherent way with respect to the dependency relationships between the packages. So if you add in your project the OSM plugin will also automatically insert the Edit and Core plugins and the npm package bootstrap-tagsinput.

Base packages

These packages are mandatory for a basic presentation of data and behavior of platform:

• Lib includes 3rd party libraries and external meteor packages
• i18n contains all languages for base packages
• Core implement the base business logic
• CoreUI define the structure of User Interface

All this set of base packages can be included in a single hit by installing the meta-package Base probably your starting case study will need only this package.

In addition to the Base Packages these plugin packages includeed in the Kepler Application show content with a better look, allow the data entry by users and a centralized application management by the admin users.

• Theme include custom CSS styles icons and images
• Edit plugin to edit/remove places’s data
• Admin plugin to administer the platform’s data/users

Plugins packages

The Kepler plugins packages provide useful pluggable features for your Kepler application. A plugin package only need to make your own package depend on keplerjs:core or others plugins if needed, it is simply a standard Meteor package that contains a file plugin.js that defines the UI structure, custom settings and others configurations for the Kepler environment.
The dependencies between plugins are defined in the classic package.js file, as is the case for Meteor packages.

Plugins dependecies

In Kepler each plugin should implement not redundant features without replicating it in other plugins, but favoring dependencies.

some of the main Kepler plugins and dependency relationships between them

In the diagram above we can observe some dependency relationships:

• (Edit→Admin) the plugin Admin use Edit to editing and removing places by admins users
• (Edit→OSM) the plugin OSM use Edit for creation of new places (imports) from OpenStreetmap
• (OSM→Pois) the plugin Pois(Point Of Interests) use OSM to imports and filter OpenStreetmap and transform these into new Pois
• (Routing →Pois) the plugin Pois use Routing to create the best paths that connect places to its neighboring Pois
• (OSM→Tracks) the plugin Tracks use OSM to imports and filter OpenStreetmap Highways data
• (Geoinfo→Tracks) the plugin Tracks use Geoinfo to imports and update tracks meta-data with geospatial info(Elevations from services)

Plugin.js

A plugin.js file efines the UI structure, the templates and their placement rendered inside the User Interface, custom settings and others Configurations in the environment, similar in concept to package.js file of Meteor. The following is an example of a plugin definition:

K.Plugin({
  name: 'pluginName',
  templates: {  /* where render the plugin templates */
    navSidebar: 'navSidebar_pluginName',
    panelProfile: 'panelProfile_pluginName',
    panelUser: 'panelUser_pluginName',
    popupPlace: 'popupPlace_pluginName'
  },
  schemas: {
    place: { /* extend base model place with additional fields */
      fieldName: []
    },
    pluginModel: { /* define new model of data */
      updatedAt: '',  
      fieldId: ''
    }    
  },
  filters: {
    placePanel: { /* extend a default filter */
      fields: {
        fieldName: 1
     }
    },
    pluginFilter: { /* define new filter */
      fields: {
        fieldName: 1
      }
    }
  },
  settings: {
    public: {
      pluginName: {
        /* here any custom settings */
      }
    }
  } 
});

Configurations

Any basic configuration can be extended by plugins inside their plugin.js file. There is a list of them and their meaning:

• Schemas defines structures for documents in the collections, can be extended by *Kepler plugins* to host the plugin fields
• Filters defines the fields exposed in the queries for pubblications and methods, the structure of this file is deliberately aligned to enhance the different levels of data privacy
• Templates defines the templates in the User Interface where the plugins can extend the content with others templates/views
• Settings contains the main default settings extended from Meteor.settings

Templates

Kepler implements a convenient mechanism to give plugins the ability to extend the platform’s basic UI structure.
Using the dynamic template pluginsTemplate and register the plugin’s templates inside the plugin.js in the section templates.

Here an example of templates defined for the plugin keplerjs:share

K.Plugin({
 name: 'share',
 templates: {
   panelPlace: {
      'panelPlace_share': {order: -5, show:true }
   },
   popupCursor: 'popupCursor_share'
 }
});

Any plugin can be define one or many templates to includes inside default template placeholders: panelPlace, popupCursor.

Any plugin can be define one or many templates to include in default template placeholders panelPlace, popupCursor, popupUser.
At the property panelPlace is also specified the priority(order) of inserting the template respect to the other plugins templates, this value can be from -10 to +10. An additional option is the property show that allow by the settings.json to hide or show any templates of KeplerJs instance.

The templates positioned inside the templates:
panelPlace(core-ui/client/views/panels/place.html)
popupCursor(core-ui/client/views/popups.html)
Through a helper called pluginsTemplate which takes care of rendering in a single position all templates registered in each plugins, related to a certain placeholder, in this case panelPlace:

<template name="panelPlace">
...
 {{> pluginsTemplate name="panelPlace" sep="<br />"}}
...
</template>

the parameter name set which placeholder must be rendered in this position in the html, the parameter sep include this html string among each plugin rendering.

This is the complete list of default templates placeholders is defined in the Kepler module K.templates:
navSidebar, tabLocation, pageHome, panelSettings, panelProfile, panelPlace, panelUser, tabPlace, tabUser, popupPlace, popupUser, popupCursor, markerCursor, markerPlace, markerClusterPlace,
markerUser, itemPlace, itemUser, footer, attribution