A playbook for API-first transformation at scale-API Product Development Lifecycle & API Portfolio Alignment

In the previous post, we’ve discussed the business capability model for our API platform and how to go about building one. In this post, we’ll discuss the API product development lifecycle, in particular, the API portfolio alignment phase. The terms API product development lifecycle, API PDLC, Product development lifecycle are used interchangeably throughout this post.

So, you have defined the baseline version of your business capability model, you think, you have coverage for all the business capabilities that your business serves to its customers, and from here on you would like to ensure that all your APIs are designed or built with a clear purpose, and in the context of these business capabilities. But how do you really ensure on what you always build is aligned to these business capabilities and make every investment count?

In every company, regardless of the size, there’s some form of process (or checklist/rules, if you prefer to call it that way) that drives building of products from customer requirements. For any new requirement, whether you are building a new product or adding an incremental product feature, there are a series of steps involved, requiring different stakeholder(s) involvement, and requiring the development of many services or adding a feature to an existing service. API-first thinking should be deeply ingrained and integrated into this process. Remember in the previous post , we talked about APIs as the logical artifacts and services as the physical artifacts exposing these APIs? Your API specification lifecycle is no different than you service development lifecycle and hence you should think of them together, not in isolation. Some companies lay out a separate process for API design and specification lifecycle than the service development, which, not only adds to the friction (with another unnecessary process), but also promotes isolated service development with a project/deadline centric mindset vs a product centric mindset in an API-first methodology, where, API interface design is always the first step of every product development. This is really a key principle to reinforce, failing which, you would develop many overlapping services/APIs, poorly design interfaces (because API design becomes an afterthought), implementations not matching API interface definitions, and tooling focused on optimizing service development and completely ignoring API interface quality, thereby contributing to a sub-optimal end-user experience.

API product development lifecycle

With assumptions, stated in the previous section, a typical API product development lifecycle would look like the one below;

The API PDLC has four lifecycle phases; Define (we’ll call it as Align throughout this post), Design, Develop and Release. Transition from one phase to other is controlled by a set of quality criteria (aka API maturity criteria). When the API moves from one phase to another, the criteria of the preceding phase are evaluated and the evaluation contributes to the overall API quality score. At the end of the API PDLC, in the release phase, the API quality score determines whether the API product meets the expected quality level (set by your organization), and whether it should be released.

The API PDLC kicks off with the API product owner submitting an API proposal. At the least, the API product owner provides the following things in the form of a use-case artifact.

• Problem statement
• High level use cases
• Target audience
• end-2-end developer experience (particularly how the product would be consumed by the end-user)

Align

The Align phase starts with the Portfolio Manager role, looking at the API proposal, making sense of it, to detect any capability overlap, or to evaluate the position of this API in the overall API Portfolio and then finally accepting or rejecting the proposal. The Align phase is termed as API portfolio alignment phase (more on this in the subsequent sections). The Align phase ends with the evaluation of all the Align maturity criteria.

Design

After the API proposal is accepted, the Design phase starts with the API engineering owner role, submitting the API review request in the accepted documentation format (e.g OpenAPI for REST), in the company. The API Designer role reviews the API and works with the API engineering owner to ensure the API Specification compliance with the API design standards and business domain vocabulary (more on the design phase later in the next post). The Design phase ends with the evaluation of all the Design maturity criteria.

Develop

In the develop phase, the engineering owner builds the service implementation for your API, based on the design established in the preceding Design phase. The Develop phase ends with the evaluation of all the Develop maturity criteria.

Release

In the release phase, you would release it to your developers (internal or external), if the API quality criteria are met. The Release phase ends with the evaluation of all the Release maturity criteria.

In this post, we’ll solely focus on the Align phase, which is also called as the API portfolio alignment phase.

API Portfolio Alignment

So, what does API portfolio alignment really mean?

Remember, in the previous post, we built the business capability model by decomposing various business processes. To recap, the business capability model describes all the business domains, the associated business capabilities for each business domain and the expanded domain vocabulary (aggregates or consistency boundary, entities, value objects, domain services etc.). The API portfolio alignment step is to ensure that an API is positioned correctly in a business domain and linked to the business capability it implements. In other words, using the DDD (domain-driven design) language, you repeat the strategic DDD and tactical DDD steps together for each API proposal.

To do this, the API Portfolio Manager carefully studies all the use cases, problem statement, browses through the business domain & capabilities registry to position an API in the appropriate business domain and capability. At this point, unless you have acquired a new company, whose business domains are yet to be incorporated to your business capability model, the Portfolio Manager would be able to position the API correctly. The alignment step also helps detecting any overlaps and the Portfolio Manager may reject the proposal and instead asks the API engineering owner to work on the already existing API, in the portfolio. The output of this step is to align/identify the business domain that the API product belongs to, the business capability that an API product would implement, and finally, the API product name and the service boundary for the API product.

Note, your API discovery tool should support navigation of API products from business capabilities and vice versa to make the API Portfolio alignment a great experience for everyone and smooth adoption of the business capability model.

Let’s understand API portfolio alignment with few examples, and how it is API protocol agnostic so you can support the same API product capability over multiple API protocols, without changing the domain taxonomy.

For elucidation, let’s assume that you have the following business domains, business capabilities and consistency boundaries.

Merchant (Business domain)->Checkout (Business capability)->Orders (consistency boundary)

Identity (Business domain)->Authorization (Business capability)->Consents (consistency boundary)

Now a product team submits a proposal to build an online checkout product for marketplaces (We’ll focus on the functionality where a customer shops for some items in a marketplace and completes the checkout). The Portfolio Manager goes through the proposal use cases, then browses the business capability model registry via the API discovery tool and determines that the functionality, orders lifecycle from creation of an order through making the payment, clearly aligns with that of Orders, in the Checkout business capability, under the Merchant business domain. So, the API product has now the following position in the capability model Merchant (Business domain)->Checkout (Business capability)->Orders (consistency boundary)->Orders API. This also establishes clear service boundary which means the service is positioned as following, Merchant (Business domain)->Checkout (Business capability)->Orders (consistency boundary)->Orders API->Order Service. In the DDD language, Checkout is the bounded context, Orders is an aggregate with order entity as the entity root and having many other sub entities such as Purchase Item and the micro-service implements the Orders aggregate (Usually a micro-service can implement an aggregate or a domain service or a bounded context). Let’s digs a bit deeper now on the API product name, resources, and events.

The API product name is named of the aggregate so it is called, Orders and say you are managing API specifications in GitHub, the portfolio manager creates a repo using the API product name and with the following convention <namespace>.<API_product_name>Specification (more on namespaces in subsequent sections), so you have a repo called checkout.OrdersSpecification. If you are co-hosting both the specification and implementation together, you can create a release GRO in the format <namespace>-R, so it is Checkout-R (R for release) and you create an Orders repo in Checkout-R that hosts both the API specification and the service implementation. The latter is better approach, as it ensures that the implementation and the interface are always in sync.

Expanding furrther the business domain vocabulary and shared domain taxonomy

As you add new business capabilities to the model, it is important to define all entities and value objects that make up the domain model of the business capability (In DDD language it is called the tactical DDD phase). In the process, you would identify that many of the entities that are common across many capabilities in a particular business domain, it is thus useful to maintain a shared taxonomy at the business domain level so that every domain capability can leverage them, without the need to define them at each business capability level. This way you ensure that every concept in your domain model is described in one consistent way. Many companies refer to these as domain common components. Maintaining the sanctity and quality of these domain commons is very important and help create a consistent API product portfolio.

Now that we’ve ironed out everything related to the domain model, let’s talk briefly about the API design on how the portfolio alignment helps with the API definition.

In the above section we introduced a new concept called namespace. A namespace is basically a logical grouping of business capabilities based on how the domain model is shared. If a set of business capabilities share the same domain model, they are all grouped under one namespace. Namespace also plays an important role in the capability model to resolve conflicts. For example, you can later define an orders API prroduct in the gift-card namespace (gift card orders are very much different from checkout orders), and it wouldn’t conflict with the checkout namespace Orders API product. A namespace can be as big as the business domain, or at a level of a business capability (bounded context) itself or a finer bounded context. In the business capability registry tool, each business capability MUST be mapped to a namespace.

Now, applying the notion of a namespace, API product name semantics, you define the checkout Orders API as below.

REST

The base path would start with the major version and the namespace so its v1/checkout

The core API resource is orders (exactly matches the consistency boundary or an API product name) so the resource URI is v1/checkout/orders

The API operations are GET/POST /orders and GET/PUT/PATCH/DELETE orders/{id}

GraphQL

Since we’ve a very matured business capability model with well-defined service boundaries, we’ll straightaway use GraphQL Federation to host all of our GraphQL capabilities. Discussion around why GraphQL federation is for another time, but in the context of this discussion, we use GraphQL federation as the API gateway pattern for our graph, which routes/federates all Orders subgraph capability operations to the Orders micro-service.

In the federated graph, the orders API operations could be placed under the checkout namespace, something like Checkout_Orders. If you don’t like name spacing in a graph, you can have Orders as the subgraph and design the sub-graph as below. Omitting the namespace should be fine until the time, there’s. a conflict, i.e., when you need host say the gift-cards Orders. We should however always ensure that the API discovery tool capture the Orders sub-graph capability alignment with the checkout namespace. The Orders sub-graph operations would like the ones below.

type Query {

order (id: ID!): Order

orders: [Order!]

}

type Mutation {

createOrder(input: OrderInput!) : Order

…other mutations

}

Event-driven APIs

Events are core to every micro-services architecture. In the context of Orders API, the resource, order, may transition through several state changes, and these are emitted as domain events to all interested parties (for example, a merchant may be interested in an event when an order is paid and completed). The event name can use the following convention (Event name is the channel name in the AsyncAPI specification, if you are using AsyncAPI to describe events and the type attribute in a CloudEvent, if you are using the CloudEvents specification as your envelope format).

com.<event-producer>.<capability-namespace>.<capability-resource>.<event-action>.<event-major-version>

com.acme.checkout.order.created.v1 is the event that is emitted, when an order is created.

com.acme.checkout.order.completed.v1 is the event that is emitted, when an order is completed.

As you can see from above, your business capability model and vocabulary are very much API protocol agnostic. Regardless of the API protocol, you have now, or introduce a new protocol in future, the domain taxonomy and vocabulary don’t change.

Now, let’s consider an incremental addition proposal, wherein a product manager proposes a new capability to help merchants create invoices, for their customers. Another product manager also submits a new proposal to help manage checkout sessions. The portfolio manager browses the business capability registry and doesn’t find an invoices capability, so it adds the new API product, invoices, as a new capability and it is positioned in the model like the following, Merchant (Business domain)->Invoices (Business capability)->Invoices-> Invoices API->Invoices micro-service. In the second use-case, the portfolio manager locates the Checkout business capability in the model and positions the new Checkout Sessions aggregate as follows- Merchant (Business domain)->Checkout (Business capability)->Checkout Sessions-> Checkout Sessions API->Checkout Sessions Micro-service.

Final thoughts

As discussed in the previous post, creating a business capability is a great first step, but you won’t realize its value, until it is fully integrated with your API product development lifecycle. Integration of the business capability model with the product development lifecycle, reinforces product thinking with clear customer-focus throughout the lifecycle, engages every stakeholder in the organization, reduces process churn, increases productivity, helps overcome/reduce tech debt, provides visibility into all service assets, helps direct the technology investments, where it is needed or critical, and creates a competitive advantage. Integrating the business capability model with the product development lifecycle is critical to the success of an API-first transformation and to build successful digital platforms.

This article is the fourth of seven that explores the API-first transformation playbook in detail. In the next post, we’ll discuss the Design phase of the API PDLC and how you build and employ API standards to help guide the API design.