Deploy WSO2 API Microgateway in Kubernetes

WSO2 API Manager Microgateway is a lightweight message processor for APIs. The microgateway is used for message security, transport security, orchestration, routing, and other common API Management related quality of services. It has the throttling capabilities as well as the usage metering capabilities.

The microgateway is designed to be a container native and this aligns with microservices architecture. This is being designed to work in decentralized environments and the connectivity to the API Management system is optional.

Prerequisites:

1. WSO2 API Manager 2.5.0
2. WSO2 API Manager Microgateway Toolkit 2.5.0
3. Kubernetes Cluster

WSO2 API Manager Microgateway toolkit is responsible for microgateway related management tasks. It comes with a command line tool which accepts commands issued by the user. With this command line tool, users are able to communicate with the API management core runtime (API publisher) and build the microgateway runtime. The API microgateway toolkit can communicate with the API publisher to retrieve API metadata that is required to create the microgateway runtime.

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. You can run K8s cluster which runs on minikube, Kubernetes Engine in Gcloud, Amazon EKS, On-Premise, etc. This project was originally designed by Google and is now maintained by the Cloud Native Computing Foundation (CNCF). Kubernetes (K8s) is written in Go language.

In this tutorial we are going to perform the following tasks.

• API Create, subscribe and JWT token generation
• Prepare Configurations
• Setting up API Microgateway project
• Building the API Microgateway project
• Load the docker image in Kubernetes
• Deploy API Microgateway in Kubernetes
• Access the API in the API Microgateway

1. API Create, subscribe and JWT token generation

Create an API with the following details in API Manager 2.5.0. You can create an application which supports JWT tokens. Then you can subscribe to the API and get a JWT token to invoke the API.

Please refer [1] for more information on getting a JWT token.

API details

2. Prepare Configurations

Create a deployment.toml file in with the following content. You can place this file inside the wso2am-micro-gw-toolkit-2.5.0 root directory. Let’s call this location as the TOOLKIT_HOME.

In the following configuration, replace the <TOOLKIT_HOME> with the correct directory path.

[kubernetes]
  [kubernetes.kubernetesDeployment]
    enable = true
    #name = ''
    #labels = ''
    #replicas = ''
    #enableLiveness = ''
    #initialDelaySeconds = ''
    #periodSeconds = ''
    #livenessPort = ''
    #imagePullPolicy = ''
    #image = ''
    #env = ''
    #buildImage = ''
    #copyFiles = ''
    #dockerHost = ''
    #dockerCertPath = ''
    #push = ''
    #username = ''
    #password = ''
    #baseImage = ''
    #singleYAML = ''
  [kubernetes.kubernetesService]
    enable = true
    #name = ''
    #labels = ''
    serviceType = 'NodePort'
    #port = ''
  [kubernetes.kubernetesConfigMap]
    enable = true
    ballerinaConf = '<TOOLKIT_HOME>/resources/conf/micro-gw.conf'
    #[[kubernetes.kubernetesConfigMap.configMaps]]
    #    name = ''
    #    mountPath = ''
    #    readOnly = false
    #    data = ['']

This is deployment.toml file which contains the relevant deployment configurations related to Kubernetes. For this use case we have enabled K8s deployment, K8s service and K8s config maps. In this example we are using NodePort service type in Kubernetes.

In addition to that if you want to connect to an external docker host, you can provide those details as below. Then the docker image will be created in that docker host’s registry. For an example, you can provide the minikube docker host and docker cert path under Kubernetes Deployment configuration.

Note: Use minikube docker-env command to list down the docker host and the docker cert path.

dockerHost = 'tcp://192.168.99.100:2376'
dockerCertPath = '/home/user/.minikube/certs'

Note: The API Microgateway has all the configurations in a single file and it is called the micro-gw.conf. This file has the configurations such as key manager configurations, jwt configurations, etc. In order to function the API Microgateway, micro-gw.conf has to be present. We have used a Kubernetes Config map to provide the configurations to the API Microgateway.

[kubernetes.kubernetesConfigMap]
    enable = true
    ballerinaConf = '<TOOLKIT_HOME>/resources/conf/micro-gw.conf'

3. Setting up API Microgateway project

When setting up the API Microgateway project, the API Microgateway tool kit connects to the API Management Core (API publisher) to get the API details, policies, etc. Based on the details received, using the mustache templates, it creates the API definitions.

To get started setting up the microgateway project, go to the wso2am-micro-gw-toolkit-2.5.0/bin location.

Let’s create a project call k8s_project with the following command. We need to provide the deployment.toml file as an input.

./micro-gw setup <Project_Name> -a <API_Name> -v <API_Version> — deployment-config <Location_of_Deployment.toml_File>

./micro-gw setup k8s_project -a hello_world -v v1 — deployment-config <TOOLKIT_HOME>/deployment.toml

This commands creates the following folders under the k8s_project folder.

The folder structure of the project

If you check the hello_world_v1.bal and listeners.bal files, you can notice @kubernetes annotations are being added to the files. These annotation enables the Kubernetes support for the API Microgateway.

@kubernetes:ConfigMap {
    ballerinaConf: "<TOOLKIT_HOME>/resources/conf/micro-gw.conf"
}

@kubernetes:Deployment {
    name:"k8s_project-deployment-79f5c8d9-e02a-4e3e-9e5a-0a77ffd74107"
}
@kubernetes:Service {
name:"k8s_project-rest-6ad216e3-9041-4091-8d12-cf613eea6bd1",
serviceType:"NodePort"
}
@kubernetes:Service {
name:"k8s_project-rest-edba9cdb-675c-467a-a649-0369163ea7d1",
serviceType:"NodePort"
}

4. Building the API Microgateway project

Let’s build the k8s_project as follows.

./micro-gw build <Project_Name>

./micro-gw build k8s_project
@kubernetes:Service     - complete 2/2
@kubernetes:Deployment     - complete 1/1
@kubernetes:Docker     - complete 3/3 
Run following command to deploy kubernetes artifacts: 
kubectl apply -f <TOOLKIT_HOME>/bin/k8s_project/target/kubernetes/
Build successful for the project - k8s_project

When you build the project, it generates yaml files such as deployment, service, configmaps, etc to be deployed in Kubernetes. You can find the sample Kubernetes resources generated in <TOOLKIT_HOME>/bin/k8s_project/target/kubernetes location as follows.

Kubernetes Resources

In addition to that, it builds the docker image to be deployed in Kubernetes in your local docker registry. You can find the image as k8s_project:latest when you execute docker images command. If you provide a dockerhost such as minikube as explained in previous steps, you can find the docker image in the minikube.

5. Load the docker image in Kubernetes

When you build the API Microgateway project, it creates the docker image in your local registry. In order to run your API Microgateway in Kubernetes, your docker image should be available within Kubernetes cluster.

Note: If you specify the docker host and docker cert paths of minikube when building the project, then your docker image will be getting created in your minikube docker registry. In that case, you can skip this image loading part.

There are mainly 3 ways how you can make it available in a Kubernetes cluster.

1.) Push the image to docker hub

eg: — https://hub.docker.com/r/pubudu/k8s_project/

You can push your docker image to docker hub. Once you do that, you need to change the image name to pubudu/k8s_project:latest in your k8s_project_deployment.yaml file which resides in <TOOLKIT_HOME>/bin/k8s_project/target/kubernetes location.

When you deploy the API Microgateway in Kubernetes, your image will be pulled from the docker hub.

2.) Push to a Private docker registry

You can push your docker image to a private docker registry and accommodate the changes listed in [2].

3.) Load the docker image manually

Basically when you create a Kubernetes cluster in GCE, Amazon, on-premise or any IaaS there will be around 2 or more Kubernetes nodes. Then you need to load your image in every Kubernetes node. This is because your pod can be scheduled in any of your Kubernetes nodes.

To load the image, you can do the following.

• Save the docker image to a tar file.

Now the docker image resides in your local machine docker registry. You can save that docker image to a tar file as follows.

docker save k8s_project:latest > image.tar

• SCP the image to the Kubernetes nodes

scp -i <identity file> image.tar username@K8s_NODE_IP:
Note: In minikube, username is docker and IP address can be found using the command minikube ip.

Note: Identity file is the public key of the Kubernetes node. For an example you can get a google_compute_engine.pub for GCE. For each and every Kubernetes node, you need to scp the docker image.

• Load the docker image in Kubernetes nodes.

In order to load the docker image in Kubernetes nodes, you need to ssh to each and every Kubernetes node and load the docker image.

ssh -i <identity file> username@K8s_NODE_IP
docker load < image.tar

6. Deploy API Microgateway in Kubernetes

By pointing the K8s resources, you can deploy the K8s resources using the following commands. Kubectl is the Kubernetes command line tool which is used to deploy and manage applications.

kubectl create -f <TOOLKIT_HOME>/bin/k8s_project/target/kubernetes

Also you can check for the deployed pods, services and configmaps in Kubernetes as above.

7. Access the API in the API Microgateway

In this project we have used NodePort service type in Kubernetes. When you use NodePort type in your service, you can expose your service from each and every Kubernetes node that is running in your Kubernetes cluster. By default, your service port will be mapped to a port between 30000–32767.

How NodePort traffic works in K8s

Source : https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0

Simply this means that using any of the Kubernetes nodes IP and service NodePort, you can access your API as follows.

https://<Any_Kubernetes_Node_IP>:<NodePort>/hello/v1/check

You can use kubectl get services command to list down the services and you can find the relevant port mapping as below. Eg: 9095:30123

List down Kubernetes Services

In order to access the API in HTTPS, respective Node port is 30123.

URL — https://<Any_Kubernetes_Node_IP>:30123/hello/v1/check

Headers: Authorization: Bearer <JWT_TOKEN>

Method — GET

Note: As JWT is a self contained access token, the microgateway does not need to connect to the key manager. But if you are using an Oauth2 access token, you need to point the microgateway to the key manager. The configuration file of microgateway which is micro-gw.conf has the key manager details as below.

[keyManager]
serverUrl="https://localhost:9443"
username="admin"
password="admin"
tokenContext="oauth2"
timestampSkew=5000

In the K8s use case, if you are running key manager in K8s, you can provide the service name of the key manager to the serverUrl. If you are running key manager outside the K8s, key manager serverUrl has to be accessible for the API microgateway pods. As SSL hostname verification is enabled by default, you need to add the public key to the API Microgateway truststore which resides in the <API_Microgateway>/runtime/bre/security location.

[1] — https://docs.wso2.com/display/AM250/Working+with+the+API+Microgateway

[2] — https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/