Welcoming APICTL 4.0.0 to the Family of Heroes

Let’s embrace the newest member of the APICTL family released with WSO2 API Manager 4.0.0

WSO2 API Controller (apictl) 4.0.0 has been released on the 5th May 2021 with WSO2 API Manager (WSO2 API-M) 4.0.0, as the API-M family member to perform CI/CD tasks for APIs/API Products and Applications. Early releases of apictl had the capabilities to manage API-M environments, list APIs, API Products and Applications, create API projects, import/export APIs, API Products and Applications, generate tokens for APIs and API Products for testing purposes, manage APIs on a Kubernetes (k8s) deployment, etc.

apictl 4.0.0, the newest member of the apictl family has been enhanced with a vast variety of new features and improvements. It can be used as a developer CLI tool for Choreo Connect (API Microgateway) now and it can perform WSO2 Micro Integrator (WSO2 MI) server-specific operations such as monitoring Synapse artifacts and performing MI management/administrative tasks from the command line thus expanding its responsibilities now more than ever.

In this article, I am going to show you the power of the newborn child of the apictl family, apictl 4.0.0, by giving you some snapshots of its features.

You can download the apictl 4.0.0 from this link and the official documentation can be found here. The newly introduced catalog of apictl is included here which contains the abstract view of the commands and their usages.

Introducing new namespaces aka “aliases” for commands

Let me ask you a question. If you are familiar with the apictl 3.2.x, you may recall the apictl delete api command. Do you remember the behaviour of it?

If you use it without setting the k8s mode (which means in the default mode), it had the ability to delete an API from a WSO2 API-M 3.2.0 environment. But, if you have used it after setting the mode to k8s (you can use apictl set --mode k8s to set the mode to k8s) you had the ability to remove an API from a k8s cluster. But there can be situations you may have tried to delete an API in an API-M environment after setting the modes to k8s by mistake and vice versa, which may have given you unexpected outcomes.

Think now, apictl has been expanded to other products as well such as Choreo Connect and WSO2 MI. Setting the modes for each of these products will be definitely problematic. Hence, apictl 4.0.0 has introduced namespaces (aliases) to avoid these confusions. Three (3) aliases have been introduced as below.

1. k8s for Kubernetes related commands
2. mg for Choreo Connect related commands
3. mi for WSO2 MI related commands

You just need to add the above aliases after the keyword apictl when executing a command. Suppose you want to execute WSO2 MI related command. All you need to do is, execute apictl mi <command>. The same goes for the other two (2) as well.

You do not need to worry about the modes anymore. You can directly use the aliases. With this, the --mode flag of the apictl set command has been deprecated.

Wondering how to execute apictl commands related to “WSO2 API-M”?

Wondering?

If you want to execute an apictl command related to WSO2 API-M, you can directly use apictl <command> as usual. No need for an alias. Handy isn’t it?

New features of apictl 4.0.0

1. Support import/export revisioning of APIs and API Products

Revisioning of APIs/API Products is a new major feature that has been introduced with the WSO2 API-M 4.0.0 release. The main advantage of this feature is that a user can do the changes to an API and expose it to the outside world with the same version without confusing the consumers. For example, let’s say you have an API named PizzaShackAPI with version 1.0.0, and you have decided to change the endpoints of this API. You can do the change as you wish and create a new revision and deploy it within the same version. So the consumers of this API will be able to use this changed API with the same version without a problem. They do not even feel any difference since the underlying changes have happened to the same version. If there is a problem or a drawback of this change, the API creator can roll back to the previous revision and deploy it back. Again, the API consumers won’t feel any difference since they will be using the same API version. This is the main advantage of this feature. To learn more about this, refer here. (Make sure to learn about the concept named working copy of an API as well, which will be easy for you when understanding the revisioning more)

Oh, wait… You might be thinking about what this feature has to do with the apictl tool. apictl 4.0.0 provides the ability to export or import API/API Product revisions as per your need. Two (3) flags have been introduced to the apictl export api, apictl export api-product and export apis commands related to revisions.

1. --latest — This flag can be used when exporting the latest revision of an API/API Product using the apictl export api or apictl export api-product commands.
2. --rev — You can use this flag to specify the particular revision number of the API/API Product to be exported as per your need when using the apictl export api or apictl export api-product commands. If --rev is not provided, the working copy of the API will be exported.
3. --all — This flag can be used with the apictl export apis command to export the working copy and all the revisions of APIs irrespective of the deployment status.

Note that, if you specify both of the above flags in your export command, the --latest flag will get the priority and the value of the --rev flag will be ignored.

After exporting, you can use the import command as usual to import an exported API/API Product revision. In the importing environment, the revisions will be created for the same API/API Product in the same order that you are importing. For example, let’s say you have three (3) exported revisions of the same API, and you can use the apictl import api command with the --update flag to import those since those belongs to the same API.

But note that, the maximum number of revisions that you can import is five (5), and after that the import will fail. You can specify the --rotate-revision flag during the import so that it will delete the oldest revision and create a new revision, if the maximum revision limit (5) is reached.

Apart from the export/import tasks, the listing of the revisions of a particular API/API Product has been introduced as well with apictl 4.0.0 onwards (apictl get api-revisions and apictl list api-product-revisions commands).

2. Concept of the deployment environments

With the introduction of the revisions feature, another improvement has been done to the concept of deploying. In the previous API-M versions, when you publish an API, it will be deployed to the gateway(s) as well. But with WSO2 API-M 4.0.0 onwards these two (2) concepts (publish and deploy) have been decoupled as shown below.

With the introduction of this decoupled behaviour, there can be APIs/API Products that have been published but not yet deployed (similar to Advertised Only APIs) and vice versa. To support this improvement, apictl 4.0.0 has done some improvements on its side as well.

When you export an API/API Product (revision)a file named deployment_environments.yaml will be included in the exported archive if that particular API/API Product revision has been already deployed. It will contain the details about the list of the deployed gateways. Refer here to learn more.

Furthermore, another new command has been introduced named apictl undeploy, which can be used to un-deploy a particular API/API Product revision from a gateway as mentioned here.

3. Introducing apictl bundle command (archives a project)

Zipping

You can use the newly introduced apictl bundle command to archive a project directory without the need for external dependencies. This command will generate a .zip archive file of a given directory. For more information refer here.

4. Ability to extract APIs from the AWS API Gateway and to initialize an apictl API project

A new command named apictl aws init has been introduced that has the ability to extract an API from the AWS API gateway and to create an apictl project. This command uses Amazon Web Services (AWS) CLI underneath it to do the retrieval of the API from the AWS API gateway.

After the API project is created, you can use the apictl import api command to import it to an API-M environment. When this type of an AWS API has been imported to the API-M, it will be visible as an Advertised Only API without the deployments. That is because the actual deployment of this API is available in the AWS portal but not in the WSO2 API-M. API-M only just does the advertising of this API, nothing else. For more information refer to the link here.

5. Merged Micro Integrator (MI) CLI commands with apictl

With apictl 4.0.0, the monitoring tasks of the Synapse artifacts (deployed in a specified MI server) can be done along with various management and administration tasks using the command line. As mentioned under the topic Introducing new namespaces aka “aliases” for commands, a new alias named mi has been allocated to the commands related to MI. Refer here to learn more.

6. apictl as a developer CLI tool for Choreo Connect (API Microgateway)

This feature of apictl 4.0.0, enables interacting with one or more Choreo Connect environments without having to perform the tasks such as deploy an API, view the list of deployed APIs and un-deploy an API via the WSO2 API-M. As mentioned under the topic Introducing new namespaces aka “aliases” for commands, a new alias named mg has been allocated to the commands related to Choreo Connect. Refer here to learn more.

Maintenance features and improvements of apictl 4.0.0

Let’s move on to the next phase. In this section, I will briefly explain the major improvements done via apictl 4.0.0 to accomplish the goals for maintenance.

1. Adding proxy environment variables support for apictl

With this improvement, you can set proxy related HTTP_PROXY, HTTPS_PROXY, http_proxy, and https_proxy standard environment variables, with or without basic authentication as explained here to send the requests initiated from apictl via a proxy server. After one of these environment variables is set in your environment where apictl is used, all the requests will go through the proxy server specified.

2. Support TLS renegotiation configuration

By default, TLS renegotiation is disabled for the apictl. However, from apictl 4.0.0 onwards, the TLS renegotiation mode can be set using the command apictl set --tls-renegotiation-mode as shown here, which will be applied globally to all subsequent TLS connections established by apictl.

3. Introducing new file structures and schemas for API/API Product/Application projects of apictl

A wide change has been done to the file structures and schemas of the apictl projects in apictl 4.0.0. The file structures of the exported APIs, API Products and Applications have been changed as mentioned in the official documentation. Some tips to remember are listed below regarding the new file structures.

• Each of the exported project archives will contain a definition file consisting of the API/API Product/Application definition. In the previous releases this definition file of the API/API Product was named as api.yaml (or api.json if you have used the format JSON file exporting) and it was inside the Meta-information directory of the exported archive. But from apictl 4.0.0, the definition file of an API/API Product will be located directly inside the project archive (there will not be a directory named Meta-information anymore).
• The APIs and API Products' definition files have been renamed as api.yaml (or api.json) and api-product.yaml (or api-product.json) respectively.
• The exported definition name of an Application had the format <Application-Name>.json in the previous apictl releases. But from apictl 4.0.0 onwards, the Application definition will also have a consistent name (as in APIs and API Products). It is application.yaml (or application.json)
• The exported archive will have a file named api_meta_data.yaml, api_product_meta_data.yaml or application_meta_data.yaml based on the project type (API/API Product and Application). This file will contain some details such as the project name, version, owner and the information related to the apictl vcs command. It is recommended not to change the details inside this file unless you have an identified use cases or for a reason to do that.

The .yaml (or json) files that contain the definitions of the exported artifacts such as APIs (api.yaml), API Products (api-product.yaml), Applications (application.yaml), Deployments (deployment_environments.yaml), Documents (document.yaml), Certificates (client_certificates.yaml and endpoint_certificates.yaml ) and GraphQL complexity data (graphql-complexity.yaml) will have a consistent schema as shown below.

type: api
version: v4.0.0
data:
  name: PizzaShackAPI
  description: 
  context: /pizzashack
  version: 1.0.0
  provider: admin
  lifeCycleStatus: PUBLISHED

The above is a sample of an api.yaml file. Every definition file has the field type that contains the type of the artifact (can be api-product, application, document, etc), the field version that contains the major API-M version and the field data that contains the definition of the artifact.

4. Introducing a structure for deployment and source repositories

This is another major improvement done in the apictl 4.0.0 to reduce the burden of the DevOps during CI/CD. Do you remember where we had the api_params.yaml and its usages in the previous apictl versions? What I am going to tell you might shock you for a moment… You had two (2) options to consider when playing around with the parameters file.

• Option 1 — You had the ability to include the api_params.yaml inside the API project so that when importing the API, the details will be overridden based on the parameters.
• Option 2 — You had the ability to use the --params flag to specify the path to the parameter file when you are importing the API so that it will be considered.

But from apictl 4.0.0 onwards there is no point in including the api_params.yaml inside the project. You will be left with Option 2 only.

Furthermore, this is much improved now. You can not only provide a params file now, but also you can give a directory as the params during the import of an API/API Product.

This params directory will be called the Deployment directory from here onwards. The source project archive/directory will be called the Source directory. Basically, what I wanted to highlight was, in apictl 4.0.0, a user does not need to extract the Source repository and put the params file inside it blah blah blah like in previous versions. It is much easier now.

Many surprises, right? Also, remember that you can generate this deployment repository using the command apictl gen deployment-dir as shown here. You can find more surprises and interesting things that you can do with the deployment directory in the official documentation. Go on!! Explore it…

5. Resolve parameters at the server-side while importing an API/API Product project

In the earlier versions of the apical, the api_params.yaml file content was resolved from the client’s side before reaching the API-M server even though this cannot be observed from the user’s perspective. But from apictl 4.0.0 onwards, the parameters will be resolved from the API-M server side. This has been done to reduce the burden of the apictl client.

6. Params file support for API Products

Earlier apictl versions did not have the ability to override the parameters of an API Product, but with apictl 4.0.0 you have the power to do that. Refer here to find out more about the parameters file of an API Product.

7. Other improvements to the parameter file of an API/API Product

The page here consists of many surprises as explained in the previous few topics, and I am going to tell highlight some other eye-openers now.

• The structure of the parameters file has been changed slightly. The parameters file will have a field named configs (which did not exist in the previous versions) from apictl 4.0.0 onwards. It is the field that contains the parameters of the API/API Product.
• You can use the newly introduced policies field to specify the subscription policies of an API/API Product.
• In the early releases, you had the ability to override the security of the production and sandbox endpoints together only. But from here onwards, you can do it separately by specifying the security of the production and the sandbox endpoints individually.

8. Support for import/export of Streaming APIs

Streaming APIs refer to APIs that have AsyncAPI definitions and use the WebSocket, WebSub (WebHook), and Server-Sent Events (SSE) protocols. WSO2 API-M 4.0.0 introduced the creation of the Streaming APIs as mentioned here.

The responsibility of the apictl has been broadened with this new feature. Using the apictl 4.0.0, you can export/import these types of Streaming APIs in a glimpse.

9. VCS (Version Control System) support for both the deployment and source repositories

Let’s go back to the apictl 3.2.x release. Do you remember the feature that was introduced for GIT integration? It is the apictl vcs command. In apictl 3.2.x, we didn’t have the concept of the “Source and Deployment repository structure”. We were only using the vcs commands to perform the GIT-based integration tasks using the API/API Product/Application project repositories. In other words,

apictl 3.2.0 had the ability to do GIT integration only with the source repositories.

But, apictl 4.0.0 has been improved much more.

apictl 4.0.0 has the ability to do GIT integration with both the Source and Deployment repositories.

• You have to keep the source repositories of each of your APIs/API Products and Applications in a separate repository which will be the parent source repository as shown below.

Source
├── admin_TestApp
│   └── admin-TestApp
│       ├── application_meta.yaml
│       └── application.yaml
├── MyProduct-1.0.0
│   ├── api_product_meta.yaml
│   ├── api_product.yaml
│   ├── APIs
│   │   ├── PizzaShackAPI-1.0.0
│   │   │   ├── api.yaml
│   │   │   └── Definitions
│   │   │       └── swagger.yaml
│   │   └── SwaggerPetstore-1.0.5
│   │       ├── api.yaml
│   │       └── Definitions
│   │           └── swagger.yaml
│   └── Definitions
│       └── swagger.yaml
├── PizzaShackAPI-1.0.0
│   ├── api_meta.yaml
│   ├── api.yaml
│   ├── Definitions
│   │   └── swagger.yaml
│   └── deployment_environments.yaml
└── SwaggerPetstore-1.0.5
    ├── api_meta.yaml
    ├── api.yaml
    ├── Definitions
    │   └── swagger.yaml
    ├── deployment_environments.yaml
    └── README.md

Note that the naming convention of each API/API Product and Application should be as shown above. It is necessary to main that naming convention.

• You can keep the deployment repositories of each of your APIs and API Products in a separate repository which will be the parent deployment repository as shown below.

Deployment
├── DeploymentArtifacts_PizzaShackAPI-1.0.0
│   ├── api_meta.yaml
│   ├── certificates
│   └── params.yaml
└── DeploymentArtifacts_SwaggerPetstore-1.0.5
    ├── api_meta.yaml
    ├── certificates
    │   ├── alice.crt
    │   ├── bob.crt
    │   └── carol.crt
    └── params.yaml

Note that the naming convention of each API and API deployment repository should be as shown above. It is necessary to main that naming convention. Additionally, those deployment repositories can be generated using the apictl gen deployment-dir command.

• Furthermore, it is mandatory to set the parent source directory path before executing the apictl vcs deploy command (using the command apictl set --vcs-source-repo-path). Also, you can set the parent deployment directory path as well (using the command apictl set --vcs-deployment-repo-path).
• When the execution of the apictl vcs deploy command, it will automatically track the changes that have been committed to both the Source and Deployment repositories of each API/API Product/Application and perform the deploying task, which will ultimately reduce the DevOps burden much more.
• If you like to maintain only the source repository (similar to apictl 3.2.0), you can still do it without setting the deployment repository.

10. Standardized the apictl flags with a common convention

apictl 4.0.0 has thought of the User Experience (UX) in several places. One place is when using the flags. In the previous apictl versions, when there are two (2) words for a flag it has used camelCase and “-” both in the apictl commands. Some examples from the apictl 3.2.x are shown below.

Examples:
      --preserveStatus -> camel case
      --skipSubscriptions -> camel case
      --skipCleanup -> camel case
      --skipKeys -> camel case
      --preserve-status -> with "-"
      --initial-state -> with "-"
      --rotate-revision -> with "-"
      --update-api-product -> with "-"

Since most of the command keywords use “-” (such as “apictl import api-product”), it is more consistent to use the same thing for flags as well.

From apictl 4.0.0 onwards, all the flags with two or more words will be separated with a “-” instead of using the camelCase.

11. Introduced new improved commands while deprecating some old apictl commands

Several commands have been deprecated to maintain consistency and to improve the UX of the command line.

Visit here to find out the list of the deprecated commands and the alternative newly introduced commands for each of them.

Last but not least…

With the above-mentioned features and improvements, we can see that apictl has become a giant and supports almost most of the features that are needed for CI/CD.

Before wrapping things up, I would like to suggest you something. You can be a newbie to the apictl or not, try to explore this more with our official documentation that can be found in the below links.

Getting Started with WSO2 API Controller (apictl) - WSO2 API Manager Documentation 4.0.0

API Controller (APICTL) - WSO2 API Manager Documentation 4.0.0

You can also join the official WSO2 API-M slack channel (wso2-apim.slack.com) via this for better communication. You can talk to the developers directly regarding any issues, concerns about the product. We encourage you to start discussions or join any ongoing discussions with the team, via our slack channel.

Goodbye until the next time…. Stay safe everyone. Happy exploring!!