Practical Web Development and Architecture
The basic purpose of any web site, like all software products, is to facilitate the processing and presentation of data. As the volume and intricacy of data grows, the challenge becomes how to best present the data in a meaningful and engaging manner. Quality software architecture is often perceived by the resulting scalability and how adaptable the system is to an expansion of capabilities without the need for constant refactoring. The efficiency with which developers of varying skill levels can meet these challenges is dependent on how quickly and precisely they can intuit optimal places within the code to implement various types of logic on sets of data.
In my experiences, adhering to a few basic concepts has served me well in my development endeavors both as individual and as a leader of a team:
- Advocate a code structure emphasizing a separation of concerns between a view-layer, service-layer, business-layer and data-layer
- As much as possible keep business logic away from the UI
- Store and maintain data, especially global State data, in the data-layer no matter where it originates.
- Prevent direct references from the view-layer to the data-layer
Good fences make good neighbors
Robert Frost’s poem, “Mending Wall”, can be interpreted as a contemplation between two neighbors on the rationale for maintaining a fence between their properties. Applying this adage ‘Good fences make good neighbors’, as a metaphor for software development is quite appropriate. Like any successful building effort, the first step begins with creating a strong foundation. In my professional experiences as software developer, building a infrastructure for optimal growth always begins by organizing the initial division of the code into a “separation of concerns”.
Frequently during a project’s inception, distinguishing concerns are primarily focused on the most broad aspects of bringing a product to market. The development and planning regarding technology is centered around how devices will present the product, where and how the services will be delivered to the devices, and how artifacts and data produced by the application will be stored and retrieved.
While these ‘macro’ structure concerns are usually the domain of a development operations team, the ‘micro’ decision regarding how the code will be structured for optimal scalability are made by developers who create and maintain the code base.
He is all pine and I am apple orchard.
Keeping business logic separate from the visual aspects of the user interface is essential to building a scalable, nimble application, loosely coupled to each of its major parts. In fact, from my experiences the number one reason many application development efforts fall into a continual state of refactoring, becoming big monolithic waste lands, is due to global State data and business processes becoming conflated with the code used to implement the visual aspects of components.
Most seasoned developers are familiar with the concept of Model View Control ( MVC ). Perhaps the proliferation of this pattern is due to its seemingly simplistic approach of separating abstract concerns into a perceptible structure of directories of each concern, often labeled Model(data-layer), Views(presentation-layer), and Controller (service-layer). However, the use of MVC as a universal methodology for structuring code is often the cause of quite a few foundational mishaps. If the service-layer is the sole mechanism for bridging a view-layer with a data-layer, it often begins to inherit the responsibility of implementing business rules as well. Problems arise when logic used for processing cross-cutting concerns or business rules in a service-layer become intrinsic to a particular service. It is often the case that the logic inevitably needs to be shared with other services. Usually this means a direct references between the services or logic in one services is duplicated in other services as a short cut for sharing. These compromises will eventually detract or even prevent scalability.
The first and best choice for the placement of business logic is in the back-end where it is closest to the origination of the data. Here, all subscribers to the data, no matter how they intend on consuming the data, will consume the same results of a business process. If placement of business logic in the back-end is not possible, then processing these concerns in a business-layer in the front-end is the next best option.
Most modern back-end application frameworks, especially ones providing asynchronous RESTful api services do not have to contend with the same nuances of State as browser based applications. Back-end applications have a singular responsibility when handling a change request to the current State. Often these responsibilities, do not included ensuring all connected consumers are aware of the change. Even when this is so and a back-end application has been extended to encompass the responsibility of broadcasting State, it handles the duties in a disinterested manner, unconcerned for how or even if the changes are applied by the consumers. Maintaining a division of concerns in applications on the back-end is important. However, the challenge is often based upon separating the implementation of business rules for re-use by multiple consumers. Keeping the service-layer DRY (don’t repeat yourself), while efficiently suppling access to cross-cutting concerns to enable a fluid processing of request.
Concerns for interacting with a data-storage system should be segregated to a ‘data layer’. Cross-cutting concerns, such as security and data accessibility needs, often impervious to strict separation, should reside with in the business-layer. Controllers in the service-layer are best kept as thin processors directing traffic based on simple conditions derived from the business-layer and the data-layer.
In many front-end JavaScript browser based applications, business logic is implemented in controllers that can easily become sources of monolithic code structures. Controllers with unique responsibilities become intrinsically connected to other controllers with an entirely different set of responsibilities for the sole purpose of sharing global State or business processes. The first step in developing a solution in front-end applications, is to place the processes for managing shared State in separate services that can be injected or imported into controllers.
However, it is often not enough for controllers just to have a reference to a service that can supply global state on instantiation. Depending on when the controller is instantiated, it may not be able to harness the latest changes to State or if the State changes occur while the controller is instantiated it will not be able to update its components unless some apparatus has been created to provide constant awareness of the changes. If the apparatus providing State awareness is implemented as a service with a direct reference to a data-layer, then the component can not stand alone, but must be deployed with the data-layer as well. When this occurs, it becomes increasingly difficult to maintain and expand the capabilities of an application without constant refactoring or branching of the code. This intractable complexity not only stifles time to market, but in the worst case it can cause catastrophic failure.
Something there is that doesn’t love a wall
With the rise of Single Page Applications written in JavaScript and built for browser consumption, developers are often susceptible to the bad practices of conflating concerns. Tools such as Mustache Templates and Handle Bars support the process of conditional logic in the markup which encourages short-term gains in convenience over long term viability. Press for time constraints or just a lack of foresight, developers inject functional programming process in HTML and manipulate CSS to expedite a course of business logic. This behavior seems to have increased exponentially with the adoption of Angular 1.
As many Angular developers took to the concept of Model-View-View Model (MVVM ), which often turned out to be Model-View-Whatever? (MVW). Not only did business logic make its way into template markup but scalability problems became compounded by the concept of Angular 1’s $scope .
“Scope is an object that refers to the application model. It is an execution context for expressions. Scopes are arranged in hierarchical structure which mimic the DOM structure of the application. Scopes can watch expressions and propagate events..”
In the worst architectural practices, $scope becomes the repository of global model data needed across the application rather than just localized data sets between a view and a controller or directive. Components and Services become inextricably linked in order to get access to specific $scope.
While Angular 1 offered conditional logic for markup and provided synchronization of the data State between a controller and a view, it did not offer an equivalent feature for propagating global state across an application. Encapsulating presentational components and services that are able to maintain cognizance of an application’s changing global State became the preoccupation of most Angular developers.
We have to use a spell to make them balance:
In time, many Angular 1 developers began developing various procedures for creating synchronicity between global data State and the controllers and directives of the service-layer. One remedy, that I adopted with great success in several architectural efforts, is a structure that projects an independent component (controller/directive/template) by using a Publish and Subscribe (PubSub) pattern. Utilizing this type of ‘eventbus’ system to decouple components from direct references to global State data services, the creation of a view-layer made up of encapsulated components can be facilitated. Using this approach, a business-layer contains PubSub apparatus which is the only reference a view-layer has to interacting with global State data. The view-layer has no direct reference to services in the data-layer or services in the business-layer. The data-layer is relegated to handling data transactions to a remote server and managing the global State of data, which is only accessible by the business-layer. The requirements for standing up a view-layer component are only dependent on a reference to the PubSub and only have the expectation of making simple request and receiving vanilla objects from the PubSub.
With the popularity of ReactJS over the last several years, a paradigm shift in the way developers structure single page JavaScript applications has occurred. ReactJS garnered much attention through its implementation of a Shadow DOM (virtual DOM) that works in tandem with JSX ( a javascript extension syntax responsible for rendering elements in the browser’s DOM.) However, I believe a greater contribution has manifested in the way ReactJS has promoted and formalized the process by which web components transact with the global state.
Properties, a set of immutable values, are passed to a component’s renderer as properties in its HTML tag. A component cannot directly modify any properties passed to it, but can be passed callback functions that do modify values. This mechanism’s promise is expressed as “properties flow down; actions flow up”
Some of the problems discussed earlier with regard to the detriments of view-layers becoming strictly coupled with data-layer are remedied with ReactJS adoption of Redux, which has really served as a catalyst for a shifting developers approach to JavaScript application architecture. The Redux pattern implements a Store of immutable objects residing in a data-layer. The objects in the Store can only be changed by actions flowing up from components, while the components attributes associated with the Store object can only reflect new values by properties flowing down. Even though PubSub in Angular 1 architecture is an adequate solution for separating view-layer from a data-layer, the unformalized approach to both publishing and subscribing to changes of global state as well how the state is managed in a data-layer can inject a cumbersome level of complexity for developers as the application grows. Thankfully, with the release of a new modern Angular X ( at the time of this writing the version is 4.4.1), much of this complexity has been reduced.
Some key features of Angular X are the inevitable removal of $scope and the Controller class along with the addition of a Components class (the concept of Components was implemented in Angular 1.6). However, the most important new feature is the adoption of Observables, the focal point for working with ReactiveX (RxJS). The concept of Observables seems to be an out growth of JavaScript’s built-in-object Proxy, formally Object.observe. Other frameworks, such as JavaScriptMVC (CanJS / DoneJS), Ember, and KnockoutJs, were early proponents of the ‘observable’ concept, implementing a method of constantly observing a model data set. Using Observables in Angular X, allows components to stay up to date with global data without demanding an implementation of an apparatus such as Promises and routing resolves. Even more exciting is the adoption of a Redux pattern for managing global data State using the NGRx library. Listed as resource in the Data Libraries section of the Angular website, I believe NGRx’s set of tools will become such a prominent feature of Angular X applications that it may eventually become integrated as a standard part of the framework.
While NGRx and Observables add a monumental boost to the architecture of Angular X applications, I still believe using PubSub in conjunction with NGRx and Observables can offer substantial gains toward a more definitive “separation of concerns”, especially between the view-layer and data-layer.
Lessons learned and Practices adopted
I have experimented with many front-end javascript frameworks over the years. Dabbling with Dojo, bending over backwards with BackBone, following to the mercy of JQuery with JavascriptMVC, finding salvation with CanJS/DoneJS and Zepto, and getting onboard with Angular. No matter how familiar I am with a framework and the expected final deliverable of a development effort, taking the time upfront to assess how the “separation of concerns” will be applied to a given framework before coding has served me well. It has enabled me to become quite adept at building prototypes which progress into productions applications conducive to growth without refactoring.
More importantly, as I have gained experience and assumed responsibility in developing and managing teams as a Principal front-end developer and a Web Architect, I must approach application architecture with the assumption that I will not be the only developer to maintain and expand the application. I must consider how the division of architectural concerns will be associated with the various skill sets of individual developers so as to maximize their output. It is often the desire, or precisely the demand, for new developers to begin contributing immediately. Of course “on boarding” becomes much smoother and successful when good documentation exist and an integrated working environment can easily be replicated. Even still, a factor probably the most consequential to overall team development, is the time it takes new developer to accurately assess how data flows and connects each “separation of concern” within the application architecture. The larger and more monolithic an application the longer it takes to perceive the inherent dependencies. With decoupled modules, code development can occur without a developer having to know all the intricacies of an entire application architecture. This is especially useful when on boarding new developers, regardless of their skill level. When a new developer has to spend the first 6 weeks figuring out the entire breath of an application’s code base just to modify model data and html markup in component, the arrival of a new developer can become stultifying for both the team and the new developer.
Over the next couple of articles, I will present examples of how I have implemented the concept of a “separation of concerns” in a browser based application and NodeJS application. I will also demonstrate a method of crafting an integrated development environment with Docker that is easily adaptable.
Docker is my {I.D.E} : demonstrate how the front-end, back-end, database, and web server (NGINX) are bundled together with Docker Compose to facilitate a FullStack integrate development environment.
Optimal Angular : PubSub With NGRx : A simple front-end application enabling a user to register, login, and view protected content. This is accomplished using the Angular (4.1.x) , a PubSubBroker apparatus and the NGRx (Redux pattern).
Swagger, NodeJS, & TypeScript : TSOA: A back-end application built with the latest NodeJS (7.9.+) utilizing Express, developed with TypeScript, employing TSOA for generating Swagger UI, and retrieving and storing data in a Mongo data-store through Mongoose.
