An Application-Centric View of the Cloud

An application-centric view of the cloud would soon leapfrog the infrastructure view presented by the console. The end goal of a creator such as an architect, developer, product manager is to build an application. The provisioning of the cloud a side effect that can be derived from the configuration of the application.

In this article, we present what we believe to be the next step in cloud evolution.

What an Application Creators Want

As it stands today, creators of the application want to do the following tasks:

1. Create Data Models (table schemas, relationships constraints, validations. etc.).
2. Script business logic.
3. Create workflows.
4. Specify application access control rules.
5. Finally, publish everything as API so a user interface or another system can use its functionality.

What Application Creators are Faced with

In the current state of the cloud, the first few things application creators see is the infrastructure view of the cloud. A partial list of items is as follows.

1. VPC, Firewall Rules
2. Compute Instances
3. Kubernetes, PODs, Workloads
4. Data servers
5. Dataflow tools
6. DevOps
7. CI/CD Etc.

Two views have certain overlaps, but they have major differences. Often, the team gets caught up in the infrastructure view for a long time, and it takes several weeks or months before they start work on the actual application.

Application vs. Infrastructure View

The following diagram shows two views of the cloud through the side menubars of their UIs. This shows the Google Cloud console for the infrastructure view, but it applies to all clouds.

Infrastructure View vis-a-vis Application View of the Cloud

Creator’s Development Process

Let us examine an optimal process for the creator. If the cloud infrastructure is abstracted, the application creator will follow the process shown below. It applies to a wide class of applications — Web Application, Data Warehouse, AI/ML Prediciton, etc. Other use cases would have a process that is a variation of it.

A Typical Application Development Process

The rest of the article digs deeper into the details of each, concludes with the architectural view.

Data Models

In an application-centric cloud, a creator would create new data models using UI. They would specify detailed meta properties at the database, table, and field level. A few examples of these meta properties are access control rules, validation rules, data loss protection settings, etc. This may also include how the data is stored. For example, a creator can specify if the database is multi-tenant using per-tenant DB instances or tenant-id as a key. A user can also introspect existing databases and add meta properties to them. The platform would transparently take care of a) creating/migrating tables, b) hiding columns based on the user’s role, and several such configurable requirements.

An example UI is shown below:

Creating Data Model and Specifying Meta Properties of Columns

Scripting Business Logic

The serverless functions are modular and agile. They work very well for incorporating application logic. They also work well in workflows, i.e., workflows invoking functions to perform tasks. We tried serverless functions and workflows for orchestration for several complex use cases. They work amazingly well. They are very critical for the low-code revolution. They handle cases that require algorithmic processing using data structures such as list, map, tree, etc.

One major problem with the serverless function is how to test and debug them using an IDE. The second problem is how to use Cloud service APIs in a program running locally (without sharing cloud credentials). Third, the APIs can be complex and may require domain-specific abstractions. Everything here may sound like a low-level coding issue. But this where the rubber meets the road.

An application-centric platform can solve these problems using:

1. By providing abstracted and simpler APIs of cloud service as SDK in popular programming languages. The SDK will proxy the call to a process running on the cloud for the actual execution so a developer can use it from the local machine.
2. Enable editing, execution, source line debugging in an IDE running on the user’s machine using the SDK described above.

Developing serverless functions in a local IDE using SDK that proxies invocations to the cloud

Workflows

Backend workflows are directed graphs or directed acyclic graphs (DAGs). They define an ordering of tasks to be executed, including concurrent execution. The actual tasks are implemented as separate scripts or functions. The workflow platform permits dynamic deployment of tasks (therefore, serverless function fits the bill). In addition, the workflow runtime (executor) supports the requirements of scalability, fault tolerance, persistent state management, operation, and monitoring.

Ideally, the workflow creator should define the directed graph using a simple UI tool. An example UI is shown below:

Easy to use Workflow Designer

External Service Integration

The external systems are integrates using restful APIs, webhooks, or pub/sub. In an application-centric platform, the user should be required.

1. Configuring API credentials using a UI.
2. Import API specification as a Swagger document (Open API) or specify using a UI.
3. Be able to invoke an API by giving its name/topic name, endpoint, and parameters.
4. The platform takes care of managing the life of access-token, transport, retrial, timeouts, etc.

File Management

In modern applications, files play a big role (for AI/ML, data ingestion, etc.). Therefore, an application-centric platform should provide Dropbox-like functionality and simple APIs to read, write, move, copy, etc. (without having to seal with service accounts and tokens).

1. UI for file management, bulk upload.
2. SFTP, HTTPS for automated file transfer.
3. Encryption using an application-provided key.

Dropbox-like File Management Functionality

Group and Role-based Access Control

Some of group-based and role-based access control requirements are:

1. Support enterprise identity.
2. Using federated users, create groups (by dept, customer, etc.).
3. Create roles and roles hierarchy (a role including other roles).
4. Assign roles to the user.
5. Specify role and group settings on data, API, function, workflows, etc. (wherever is needed).
6. Be able to query the roles and groups of the user in a function for writing application logic.

Publishing APIs

The platform should automatically publish data, function, workflow, files, etc., as restful APIs so an external client such as UI can use it.

Non-functional Requirements

The non-functional requirements such as security, data encryption in transit and rest, audit logs, monitoring, alerts, scalability, failover, disaster recovery, databack, data loss prevention requirements are mandatory.

User’s View of Application Centric Cloud

A user should see the boxes shown in dark color.