The Microservice Language for Me — D-ASYNC

What is the Cloud-Native Programming Language? Is it Ballerina or DarkLang? I was quite frustrated with both and other similar attempts of creating something genuinely cloud-native. But that’s not the reason why I created the project “D-ASYNC” for myself which you never heard of. The first adopters independently claimed: “I don’t want to work on other team or a project which does not use D-ASYNC”. I’ll try to explain it in a moment, but first things first.

Expectations

First, the term “Cloud-Native” has numerous meanings depending on what problems you solve. For some folks, it refers to Docker and Kubernetes, for others — an overall managing of Cloud infrastructure, or — application-specific aspects like communication, tracing, and monitoring. Due to this ambiguity, I’m going to use the term “Service-Oriented” (short “SO”; not SOA, not StackOverflow) when it comes to programming languages.

Now think of all of the steps you have to do in order to create a (micro-)service. Services need to communicate with the external world, other internal services, or even themselves — this involves creating an API specification, generating clients and stubs. Then you write an application layer of request-response/message/event handlers that act as a glue between the non-functional aspects and the business logic itself. Then some APIs should be asynchronous due to the nature of a lengthy background process. Now you need to use a message queue and create a DB-backed result store for polling, alternatively deliver the result back using a web-socket or another message queue. Then events invert service dependencies, but you also have to implement the pub-sub in your multi-service app. For the mission-critical application, implementing exactly once execution is not a trivial task when you change data in a database. The list continues.

A lot of things described above are ubiquitous, time-consuming, requires a lot of learning and expertise to do them right, and are considered as an industry-standard practice. While every single thing may not be nagging by itself, the combination of all of them is. Thus a comprehensive solution to the vast of such little things would be my expectation for a service-oriented programming language.

Service-Oriented Paradigms

Before a Service-Oriented Programming (SOP) can be expressed in a language, we need to introduce new paradigms — a higher-level abstraction of what we do today over and over again. Fortunately, we don’t need to speculate or fantasize what developers of SO applications do and which parts of the written code do not convey any business requirements. Here are the basics.

Services

A service encompasses a functional piece of a multi-service application. Services may communicate with each other and can be communicated with externally using a variety of communication primitives (queries, commands, events; see below).

Queries

A query is a request to return data without modifying any internal state of a service. Queries usually execute synchronously (via HTTP or gRPC), and in rare cases run asynchronously when it takes a long time to prepare a result (via a message queue and a result cache). Queries are allowed to execute sub-queries, but must not send commands or publish events (see below).

Commands

A command conveys an intent to perform an operation that modifies the internal state of a service, which can succeed or fail according to the programmed business rules. A command can be canceled with a given timeout (time-sensitive operations) or by the caller (long-running operations). Commands may return a result on success and must return an error on failure. Command handlers may execute queries, publish events (see below), and issue sub-commands that comprise a workflow (see below). The preferred way to execute a command is asynchronous with guaranteed message delivery since it represents a caller’s desire to perform an operation. Otherwise, end-users get very frustrated when something gets stuck and they have to call a support service to resolve an issue for example.

Events

An event tells about a fact that already happened. The past cannot be changed, but services may choose to react to various facts to satisfy imposed business rules. A service can publish events, where other services and the same service may subscribe to the events. A service reaction to an event is a command that conveys an intent to perform an internal operation. The only exception is that the result of such a command does not have any observer. A reaction to an event can trigger a workflow (see below) since a command can call sub-commands. Events allow inverting a dependency between services and have a variety of the pub-sub implementation: event streams like Kafka, message fan-out like in RabbitMQ, HTTP-based webhooks, or other techniques based on web-sockets.

Workflows

A workflow is a series of routines (generally speaking “steps”) that can span across multiple services. A routine is a function that represents a command handler, which persists its state when invokes other (sub-)commands (other (sub-)routines). A workflow, in its essence, is a collection of hierarchical finite state machines with persisted intermediate state and execution guarantees. A routine in a workflow can be compared to an actor in the Actor Model. A typical workflow/actor engine uses the message queues and a database to store the execution state.

Unit of Work

A unit of work (a transaction) dwells on the complex topics of distributed computing, which boils down to an ability to run a command in a conflict-free atomic manner exactly once. For example, when a message-based command handler writes modification to a database and emits an event, make sure that the operation happens exactly once (does not overwrite changes in the DB on re-try and does not emit an event twice). While programming languages cannot provide the Strong Serializability or any other consistency guarantees, they can provide hints that tell the underlying infrastructure about the desired behavior without the need to program them by hand.

A generic description of these perhaps obvious concepts is not that exciting to read about, albeit the ramification of implementing them over and over again is massive for the industry as the whole. Yet, we don’t see an implementation of these paradigms in any programming language.

The Microservice Language

Over a couple of decades, the best minds in the industry tried various approaches to solve the problem of masking network calls as local ones — you may remember earlier Java RMI, .NET Remoting, or even CORBA. There were few attempts to create a new language specifically for distributed computing. However the issue has never been solved, and today we keep naming variables in the code with client or service suffix to visually denote a remote call with all of its implications. The problem was not that sharp 15 years ago. However, with the explosion of microservices, it becomes more palpable, and we try to adapt existing technologies and patterns that were not designed for these types of applications.

It was late 2015 when I started thinking about new language-integrated concepts, and in 2016 I built a prototype that implements the mentioned concepts in C#. Not as pure as it needs to be, but a good start for experiments. The project named “D-ASYNC” — Distributed Asynchronous Functions with its initial emphasis on stateful workflows. Since workflows can span across multiple services, it quickly became evident that this is a generalized approach for inter-service communication as well, without a need for explicitly creating any API or workflow definitions.

Having high-level language abstractions enables you to focus on business logic and chose the infrastructure later as you evolve the app. The same code can be a monolith or a set of microservices. The idea is that the core business logic always remains the same, but infrastructure and developer efficiency are subjects for optimization. You can tune them as you go instead of making a decision upfront that tends to be hard to change.

Lessons and Experience

The very first adopters were a team of five. They were on their way to go with conventional industry-standard tools and solutions to build nine microservices for a brand new product. Instead, by showing the best practices using Domain-Driven Design, Event Storming, Onion Architecture (Ports and Adapters), and capabilities of D-ASYNC, we created those services in no time. We didn’t spend any time on designing APIs in a traditional sense or thinking about using HTTP vs. messages. It was almost straight-forward as building a monolith.

I don’t want to work on other team or a project which does not use D-ASYNC.

Even though the project is far from completion, I may attempt to justify this claim. For example, another small team of 3 people spent one week on programming a small service with basic capabilities and HTTP endpoints, created a simple API and tests for it. Then I spent 40 minutes of removing 90% of the code they wrote to show how the same thing can be done much easier and faster. Besides, the team wanted to have four services, but could not afford to spend a lot of time, so they had to create one only.

Although I don’t think that this is the exact reason why first adopters like it. Have you ever valued electricity? We take it for granted and only perceive the real value during power outages. I believe the same effect applies to the first developers who started using D-ASYNC. Once you take it out, you need to solve many various problems, even if they are trivial, even if there are plenty of high-quality tools available. You still have to spend time to address them, where D-ASYNC abolishes them. And that’s the moment where you start thinking: “why should I do it by hand when it’s already automated?”

I’ll keep posting in this series on what exact problems D-ASYNC solves, what the typical solutions are, and other best practices, principles, and ideology when building multi-service applications. In the meantime, reach me on Twitter if you want to know more!