A Vision of a Programmable Cloud Platform

Program Cloud from your Computer (Design Orchestration and Deploy from Git)

This blog discusses a vision of a programmable cloud at the application layer. Imagine if a programmer can write 100–200 lines of code that does a few things on the cloud to complete a task. The programmer does not realize that he is operating a process on the cloud.

Let us take a look at a few example use cases:

“Read CSV files that are loaded today and stored on a cloud storage bucket, clean a few columns, apply some business logic, and store in a temporary database table.

An example of the business logic — when the ‘productId’ matches with a prior row, do not create a new row in the database but add ‘quantity’ columns.”

The next task of the above does the following:

“Aggregate weekly data, use an AI/ML model to predict weekly product movement”.

The above-described use cases are common. In this blog, we will show how they would be implemented using different approaches.

Requirements

The following requirements are for the illustration but also typical.

1. Suppliers (partners) load CSV files each day using SFTP to the cloud storage bucket.
2. The program identifies the files loaded today by the date (yyyy-mm-dd) as a prefix in the name.
3. A file previously processed needs to be skipped.
4. Create the temporary table schema in the database if it does not exist (first time).
5. If the processing of a row fails, it should be recorded in a log table with the name of the file so an administrator can review the error cases.
6. The summary of each file should be logged.
7. A consolidated summary of each file should be logged and sent to configured emails.

Three Implementation Options

There are 3 options to build the application.

1. Conventional Development
2. Low-code & Serverless
3. Programmable Cloud Platform
4. Graphical Business Process Tool — This is not really an option but a demoware until you enumerate several components for a domain. A graphical flowchart becomes unwieldy for real-life use-cases.

Conventional Development

Assuming that you have a team of cloud and application architects and developers, it will begin with the architecture diagrams (functional, deployment, security), CI/CD, DevOps. There will be few weeks/month development, testing period. The development has to account for future maintenance such as additional columns are added.

Low-code & Serverless

This is a much more efficient and low-cost option compared to conventional development. The database schema is created using a UI (Model Driven Architecture). A program executing the logic is deployed in a serverless manner. It is executing using an API call or some other trigger.

Programmable Cloud Platform

This is an extension of a low-code and serverless platform with a well-defined abstraction. It also combines intelligent configurations to support the program execution. As an example, if the program opens a DB connection, it configures a data server instance if not present. If the table is not present, it auto-creates it. Even if the latter is not supported, a well-defined abstraction with standardization will go a long way. The following pseudocode is an example of the program written by a developer to run on a programmable cloud platform described here.

// Once the code is deployed in production, the following 
// statement does not do any thing.

AbstractCloud(url="https://...", credential=credentialFile);

List l = Bucket.list(folder="/partners/supply", namePrefix="2021-04-30*");
List summaries = new ArrayList();
for (fileName : l) {
   Iterator rows = CSV.iterator(fileName);

   Map summary = processRows(rows); // function/method not shown
   Log.info(summary);
   summaries.add(summary);
}

Log.info(summaries);

Email.send(emailConfigKey="config.emails.suppliers.inventory.import", 
   subject="Import summary",
   template="Import.html", summaries)