Serverless Enterprise Application Architecture with Zeit, Now Vercel, and GitHub for free.
This article is about the Enterprise Application Architecture which we use at GEOLYTIX in order to facilitate best value from Vercel and GitHub. We are currently building XYZ¹, a free and open source project to provide interfaces for the analysis and presentation of spatial data. Our aim is to implement techniques in modern web development learned from the emergence of the JAMstack.
What’s in a name?
The JAMstack at its core, is simply an attempt to give a name to a set of widely used architectural practices. It gives us one word to communicate a large range of architectural decisions.²
JAMstacks consist of [client-side] JavaScript, APIs, and Markup. The latter usually being HTML + CSS to create browser applications running JavaScript. The term was coined by Netlify CEO and co-founder Mathias Biilmann to describe a modern web development architecture. I highly recommend the ebook provided by Netlify if you feel that delivering great websites should be more about the craft of markup and JavaScript than server setup and administration.
Mathias does well to emphasize that the JAMstack is not one specific technology itself but a mindset and an approach.
For XYZ we combine the idea of a JAMstack with a pattern of enterprise application architecture which has been proposed by Martin Fowler.
In his book Fowler suggest three principle layers.
Data Source Layer
Working in the spatial sector you will have come across the term ‘Spatial is special’. In practice, this means we require a PostGIS extended relational database (RDS). Amazon Aurora provides us with the capacity to configure a serverless RDS to store and process spatial data. I plan to write in-depth on this subject soon. In the meantime, our friends @addresscloud have written a great article in regards to serverless PostGIS with AWS Aurora.
Domain / Service Layer
This architecture layer is a collection of APIs and routing rules which form the application’s domain logic. Once deployed the Application Layer is referred to as the XYZ Host. I use Domain or Service Layer interchangeably referring to the same layer but distinct paths. The business logic of our application is written as transaction script to be processed by the Service Layer. Developed as a Node.js application the domain layer is of a modular structure. Modules to generate and sign token, and ultimately the authentication of client requests are available to all API modules. This allows for secure requests to be passed to the Data Source Layer. Requests may also be proxied to 3rd party services by providing keys as secret environment variables, accessible by the XYZ Host processes alone.
Presentation / Controller Layer
This layer is composed of client facing interfaces as Application Views. Here we apply techniques from the JAMstack. I use Presentation or Controller Layer interchangeably referring to the same layer but distinct methods. The XYZ Library being the controller to send RESTful requests to APIs in the Domain / Service Layer, the XYZ Host. Other libraries such as Openlayers, ChartJS, and Tabulator are used for the presentation of data as maps, charts, or tables.
Application View JAMstack
Client side applications use the JAMstack to retrieve and present content from a Content Delivery Network (CDN). The content is often in the form of articles fed into framework controls to compose web applications such as blogs. In our use case the content is in the form of spatial data which are locations defined by their geometries and associated property values.
Data along with script and markup may be either retrieved from the XYZ Host, from 3rd party service providers or from a CDN. Content may be stored in GitHub repositories to be accessed via the GitHub API. Static assets which do not require to be rendered by the XYZ Host may be cached in a CDN such as JSDelivr and directly loaded into an Application View.
Are dashboards dead?
Taylor Brownlow wrote about the end of dashboards in this excellent article. The argument for notebooks as the dashboard’s spiritual successor are valid and we are big fans of observable, as well as count.
We do want to have the power and flexibility to answer any question. We want to collaborate on, present, and share information. And we want to garner trust in the process by making the code itself as obvious and clearly visible as possible.
These points are not only relevant to the data science within business and location intelligence processes but to the wider audience of open source developers.
These general requirements should not be defined by the form of the application interface. Whether we define a dashboard, notebook, or report, within the XYZ context we refer to these browser interfaces as Application Views.
GitHub for Teams
GitHub is the obvious choice for all of our programming resources, whether they are HTML, JavaScript, or SQL.
Having been GitHub pro user for many years our team at GEOLYTIX got used to the ease of collaborating on and versioning of complex configurations stored in public and private repositories.
In April, GitHub introduced generous new pricing for all developers. This allows us to provide the same capabilities for configuring serverless processes for free to everyone.
XYZ APIs act as a headless Content Management System (CMS) to retrieve, cache, and serve content from GitHub for the Application View JAMstack.
A personal access token may be provided as an environment secret to the XYZ Host processes. The token allows access to resources through the GitHub Rest API v3. All resources as defined in a shared workspace, including the workspace itself as a JSON document will be cached by each of the serverless API functions. Thus requiring access to the API only after cold starts or when requested after a previously cached resource has been modified.
The transition from local to live is seamless. Having a GitHub repository cloned and open in VSCode we commit changes and use our remote resources for local development. With the data source never being local it does not matter whether the configuration is accessed remotely while running the XYZ Host a local Node.js process.
Locally we use Express to develop and debug XYZ APIs. Each API however has been designed to be deployed as an AWS Lambda via Vercel.
[Zeit] Now / Vercel
Zeit (recently rebranded as Vercel) abstract away the complexity of deploying to the AWS Lambda service directly. Official runtimes allow serverless functions access to the Node.js request and response objects.
We refer to a XYZ Host deployed to Vercel as an XYZ Instance.
A deployment configuration allows the domain layer to route requests to individual functions. Each function being a serverless process with access to all good things Node 12 (e.g. ES6 module imports). Rewrites defined in the deployment configuration may route requests to external XYZ Instances using the same domain, e.g. geolytix.xyz.
The node process’ environment variables may be safely deployed with the deployment configuration or stored as secrets with Vercel.
Deployments are possible via a GitHub integration or through the CLI. Once deployed, logs for each serverless function within an XYZ Instance may be accessed through Vercel’s website.
Did somebody say free?
It may sound crazy, but Vercel is available from free. We are not directly affiliated with Vercel in any form but absolutely love their services. When we refactored the XYZ APIs as serverless functions we worked according to the platform limits for hobby user.
XYZ Templating
XYZ APIs use templating to prevent the client from making direct requests to the data source layer. Transactional script may be stored as SQL templates in a public GitHub repository. Each API process will cache the template as part of the workspace and substitute parameters received with requests before returning the database response to the presentation layer.
Parameter substitution in templates constitutes for the server[less]side rendering of dynamic Application Views.
Constituting for the presentation layer, Application Views are the primary user interface for XYZ APIs. Being able to develop, modify, and assign templates to production deployments of the API fits with the project’s policy on developer experience. We try to keep the language as vanilla as possible with a minimum of framework requirements. Being a small team of developers (three) we would be unable to invest in the development of utilities such as a drag & drop to build dashboards.
There is currently an abundance of location intelligence platforms and many have intricate build interfaces such as Carto’s Airship. These utilities are at level which would be unattainable by ourselves in this rat race. We try to keep our development workflow as simple and focused as possible. Microsoft/GitHub are nowadays beacons of open source development and have built integrated development environments (IDE) which many developers are already familiar with. We do not want to introduce new syntax or language unless absolutely necessary. From our personal experience the best experience for a web developer is the capability to use vanilla HTML, CSS, and JavaScript in a live environment to build engaging application interfaces. Similarly we want to give data scientists the capability to write a SQL whose result may be immediately presented as a map, chart, table, or raw JSON.
Prototyping & Development Workflow
We are also big fans of coding sandboxes such as jsfiddle, codepen, or codesandbox. These tools are invaluable for communicating issues and rapid prototyping.
The fiddle above shows a sample of the Retail Points data provided by GEOLYTIX. Once we are happy with the functionality we can push the markup and script into our public GitHub repository to be used as templates.
In this example markup template the {dir} parameter is to be substituted. When requested from XYZ View API with URL parameter for the host, locale, and layers we receive the markup as HTML with the {dir} parameter substituted as ‘/geodata’ which is the path for the geodata instance on our development domain geolytix.dev.
The geodata Application View may also be stored as a static asset by fixing {dir} and other template parameter in this HTML file.
We use GitHub Pages to publish the public repository from its root. This allows us to open the static application view in a browser.
The view from a template requested from the XYZ View API and the static view requested from GitHub Pages load the same script and request data from the same XYZ host.
The default XYZ Application View (MAPP) for the geodata instance can be requested from the root: geolytix.dev/geodata
This workflow allows us to design and play with application views in a sandbox before committing code to a GitHub repository for publication without the need to redeploy our host. Views will be immediately available in production after a request to the XYZ Workspace API for all assets to be cached in all serverless API functions of this instance.
I hope this article may help you organizing your development workflow and perhaps will give you some ideas how to facilitate the best of Vercel & GitHub for free. The XYZ project is free and open source and we are actively looking for developers who would like to contribute. Please feel free to contact me via twitter if there is anything in this article which you would like to discuss in more detail.
¹ While functionally similar, XYZ as referred to in this article is not affiliated with HERE XYZ.
² Mathias Biilmann & Phil Hawksworth; Modern Web Development on the Jamstack
