Data in the corporate world is relationally complex, sensitive, and mission critical. The adoption of front end data, or front end browser architectures that consume various APIs, has unique challenges in the corporate world. Front end engineering for a consumer based application is different than for a corporate based application in that consumer applications rarely have hundreds or even thousands of tables, varying levels of authorization, and the political factors associated with siloed data. When considering the upgrade of an existing system or the new development of a corporate single page application, a corporate strategy for front end data should address these challenges.
Corporate data is relationally complex. The database for SAP, for example, can have more than twenty thousand tables in it. Handling relational data for even one hundred tables is a difficult task for a front end engineer. The tool that we normally use for reasoning about data, relational modeling, is only recently becoming available to the front end. The problems related to modeling relational data of this size on the front end can be a daunting task.
Managing Relational Data
The data from foreign key relationships on the back end translate into a method known as side loading on the front end. This is when an API endpoint groups related data, e.g. students and the teachers assigned to those students, into one payload that is returned to the front end. Reconstituting that data on the front end can be problematic if the task requires advanced data manipulation such as looping through multiple entities, e.g. listing all students, per teacher, per school. This problem becomes even more complicated when you take into account that the JavaScript code for these api calls, with possibly hard coded column names, can be duplicated across multiple screens. Modern back end systems handle this problem with data models. A model is the single source of truth about what relational data looks like in the database. All changes for the structure of data happen in one place in a system with data model. This avoids complex custom API management code in JavaScript.
Error Handling
Error handling is raised to another level with corporate data since corporate data is relationally complex. If a large transaction spanning multiple tables is updated, a large set of errors (larger than normally handled on the front end) can be generated. This requires advanced error collections and more front end code to handle those error collections.
Surprisingly, corporate data does not normally have high throughput data requirements (requirements in the order of thousands of hits per second). What is more prevalent in the corporate world is a low tolerance for mission critical failure. Any system that supplies other systems with critical data has the potential to cause a cascading failure to all other dependent systems. In the corporate world there are oftentimes hundreds of applications that rely on the data from a shared mission critical service. This pushes performance requirements for the shared service past the thousand per second mark. Any API that serves as a shared service must therefore be highly available and highly scalable. Standards about quality of service which include rate limiting (how many times a service can be called by a specific consumer before considering that consumer abusive) are needed in order to keep a high level of availability and to reduce the cascading failure effect. Quality of Service (QoS) agreements are also needed so engineers from different teams can reason about the speed, availability, and risk of combined service calls. One new burden presented by corporate front end data is the API endpoint must now sideload data in a standard way so that the front end can consume it properly without causing unneeded, i.e. chatty, calls.
Corporate silos are usually departments in an organization that have autonomous control over data. These departments control who can access their databases and how they are accessed. The drive to provide safe access to departmental siloed data is typically driven from the top level management down through the silos, but it is becoming more common for departments to provide safe access to data through service APIs on their own volition.
Security
Security is usually the first concern when opening up siloed data. Security can be separated into authentication and authorization. Authentication using some open standard like OAuth allows for widespread internal standardization and adoption, but often times is not possible because of legacy reasons. In that case a custom or proprietary solution is used. Authorization within the corporate world presents unique demands because of the numerous hierarchical security levels that corporations have become accustomed to. Within a budget report, for example, a department manager should not only be able to see her own department but also all departments under hers. This corporate hierarchy is authorization data that is usually controlled by a centralized group who answers to top level management.
A corporate front end data solution needs to able to consume not only the authorization data but also its structure as well, which presents three distinct challenges:
- How must the API be called?
- How is the data model shared?
- How is the API managed?
API invocation
The first challenge to opening siloed data involves how the data is retrieved. For a front end data strategy, an endpoint can be retrieved directly from the front end browser or routed through a back end API. The front end browser call has the benefit of no extra back end code having to be written and maintained but it also has the drawback of being a cross-domain call. Cross-domain calls require that the front end and back end data solutions be capable of handling these calls. These solutions are not always straightforward or possible because of legacy concerns.
Data Model Sharing
The second challenge, how the data model is shared, has solutions with varying levels of effectiveness because of different levels of interdepartmental cooperation and competition. One solution is to document the API and let the departmental developers manage their own client source code. This option has the least amount of maintenance required by the API provider but also does not promote adoption as well as the other option: sharing API client code. If the team that provides the API writes a client consumption library or shares a code base, it can check that code into source control and then share the maintenance burden across teams. Any team will be able to send updated code (pull requests) to the API providing team with tweaks or updates. Each consuming team can stay on the version of the client that suits them until they are ready to upgrade or until an upgrade is forced (which we will discuss in API Management). Upgrading is as easy as checking out the latest source and running any tests.
API Management
The third challenge of siloed data, API management, is concerned with the management of breaking changes. Breaking changes can be simplified to the following four rules:
- Any addition of a data node (e.g. a new column or table provided in JSON) should not break any consumer or client (if it does, that client is considered to be written incorrectly and should be updated).
- Any change to the order of data nodes (e.g. a new ordering of columns or a new ordering of tables provided in JSON) should not break any consumer or client (if it does, that client is considered to be written incorrectly and should be updated).
- Any deletion of a data node (e.g. a column or table) is considered a breaking change and needs to be included in a new version of the API and client, with the old version remaining available.
- Any change to the name of an existing data node (e.g. a column or table) is considered a breaking change and needs to be included in a new version of the API and client, with the old version remaining available.
- Any removal of support of a version of the API needs to be negotiated with all of the consumers.
A corporate front end solution needs to be able to handle this kind of process, which usually means the front end data model needs to be able to handle versioning.
Conclusion
There are front end solutions for data models that are currently available, but none are as mature or robust as the back end data model libraries. One front end solution that handles complex data relationships is Ember Data. Ember Data is the most mature front end data model library at this time, and handles most if not all of the above issues. Some other solutions that are available are js-data-angular for the Angular JavaScript framework and backbone-relational for the Backbone JavaScript framework. The corporate front end data scene is still in its infancy, but there is no good reason to forego adopting these best practices when developing new corporate single page applications.
This post was originally given as a talk to several teams at large financial services company. If you would like me to discuss the practices given here with you or your team please email me at w.watson@vulk.coop
