How to dynamically manage Kubernetes Objects

Kubernetes is now the defacto “Application Server” when you build Cloud Native applications using technologies such as Docker, Microservices etc. You are also familiar with various Kubernetes Objects such as Pods, ReplicationContainers and Services. Further, you would have deployed these objects using either YAML files (by running kubectl commands) or using Helmcharts (by running Helm commands). However, there are situations when you may be forced to dynamically deploy objects inside Kubernetes. For example, let’s say, you want to execute some task on-demand based on end-user interaction. If this task runs infrequently, you may not want to create a Pod that sits idle most of the time and only runs when a user performs certain action. In such cases, you can use Serverless technology such as AWS Lambda or GCP Cloud Function to run your container on the fly. The other option is to dynamically create a Pod using Kubernetes API. In this article, we will explore how you can use Kubernetes API to manage objects inside Kubernetes. Also, we are going to use the official Java Client library of Kubernetes.

So, let’s get started. Create a Maven project in Eclipse and add the following entry in your POM file.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>dynamickubernetesprovision</groupId>
<artifactId>dynamickubernetesprovision</artifactId>
<version>0.0.1-SNAPSHOT</version>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.4.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java</artifactId>
<version>5.0.0</version>
<scope>compile</scope>
</dependency>
</dependencies>
<properties>
<java.version>1.8</java.version>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

In the above POM file, the key thing to note is the dependency you have to add for Java Client library of Kubernetes.

<dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java</artifactId>
<version>5.0.0</version>
<scope>compile</scope>
</dependency>

Now, the basic concept of using Kubernetes API is to connect to the Kubernetes API Server and then invoke appropriate commands. There are two ways you can connect to API Server — one is In-Cluster and another is from Outside-Cluster. In-Cluster means your Java program is running inside the same Kubernetes Cluster to which you are planning to connect. Outside-Cluster means your Java program is running outside of the Kubernetes Cluster. In our examples, we will use In-Cluster; however, if you want to run the examples outside of your Kubernetes cluster, then, you will have to connect to API Server using a KubeConfig file — which actually stores your Kubernetes API server details. Do note that once you connect to API Server, from then onward, both approaches work the same way.

The other thing you need to do is to create a ServiceAccount inside your Kubernetes with appropriate roles assigned to it. And then assign the ServiceAccount to your Java program. Which means your Java program is now authorized to call only those APIs that fall under the purview of your ServiceAccount role. In our example, we have created a ServiceAccount with Cluster-Admin role, so that we can create any kind of objects within Kubernetes Cluster. However, in a real-life scenario, you should assign only those roles that are appropriate for your ServiceAccount.

Here is the serviceaccount.yaml we will use. Change it according to your requirements and then run kubectl apply -f serviceaccount.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-service-account
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: test-rbac
subjects:
  - kind: ServiceAccount
    # Reference to upper's `metadata.name`
    name: test-service-account
    # Reference to upper's `metadata.namespace`
    namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Now, we will create a simple RESTful service that acts as a Java client and will perform operations on Kubernetes cluster. So, let’s go ahead and write a SpringBoots application.

package com.test.dynamickubernetes.controller;
import java.io.IOException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.CrossOrigin;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;
import io.kubernetes.client.ApiClient;
import io.kubernetes.client.ApiException;
import io.kubernetes.client.Configuration;
import io.kubernetes.client.apis.CoreV1Api;
import io.kubernetes.client.custom.IntOrString;
import io.kubernetes.client.models.V1ConfigMap;
import io.kubernetes.client.models.V1Pod;
import io.kubernetes.client.models.V1PodBuilder;
import io.kubernetes.client.models.V1PodList;
import io.kubernetes.client.models.V1PodTemplateSpec;
import io.kubernetes.client.models.V1PodTemplateSpecBuilder;
import io.kubernetes.client.models.V1ReplicationController;
import io.kubernetes.client.models.V1ReplicationControllerBuilder;
import io.kubernetes.client.models.V1Scale;
import io.kubernetes.client.models.V1ScaleBuilder;
import io.kubernetes.client.models.V1Service;
import io.kubernetes.client.models.V1ServiceBuilder;
import io.kubernetes.client.models.V1ServicePort;
import io.kubernetes.client.models.V1ServicePortBuilder;
import io.kubernetes.client.models.V1ConfigMapBuilder;
import io.kubernetes.client.models.V1Status;
import io.kubernetes.client.util.Config;
@CrossOrigin
@Controller
public class DynamicKubernetesProvisionController {
private ApiClient client = null;
public DynamicKubernetesProvisionController() {
System.out.println("DynamicKubernetesProvisionController created");
try {
client = Config.defaultClient();
Configuration.setDefaultApiClient(client);
} catch (IOException e) {
// TODO Auto-generated catch block
System.err.println("Error while creating Kubernetes APIClient");
e.printStackTrace();
}
}
}

With the basic boilerplate code in place, let’s create a Pod dynamically. Here is the code — see createpod API below. It takes three arguments, namely, the namespace where the Pod has to be created, the docker image to be run as a Container inside the Pod and the Pod name.

The code below also shows how to get list of Pods and how to delete a Pod.

// Provision Pod
@GetMapping({ "/provisionpod" })
@ResponseBody
public String createpod(@RequestParam(name = "namespace", required = true) final String namespace,
@RequestParam(name = "image", required = true) final String image,
@RequestParam(name = "podname", required = true) final String podname) {
CoreV1Api api = new CoreV1Api();
Map<String,String> label = new HashMap<String, String>();
label.put("name", podname);
V1Pod pod = new V1PodBuilder().withNewMetadata().withName(podname).withLabels(label).endMetadata().withNewSpec().addNewContainer().withName(podname).withImage(image).endContainer().endSpec().build();
try {
V1Pod newPod = api.createNamespacedPod(namespace, pod, null, "true", null);
return "Pod Created: " + newPod.toString();
} catch (ApiException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return e.getResponseBody();
}
}
// Delete Pod
@GetMapping({ "/deletepod" })
@ResponseBody
public String deletepod(@RequestParam(name = "namespace", required = true) final String namespace,
@RequestParam(name = "podname", required = true) final String podname) {
CoreV1Api api = new CoreV1Api();
try {
V1Status status = api.deleteNamespacedPod(podname, namespace, "true", null, null, null, null, null);
return "Delete Pod Operation Status" + status.getStatus() + " with message " + status.getMessage();
} catch (ApiException e) {
// TODO Auto-generated catch block
return "Error: " + e.getResponseBody();
}
}
// Get list of pods
@GetMapping({ "/pods" })
@ResponseBody
public String getPods() {
StringBuffer buffer = new StringBuffer();
CoreV1Api api = new CoreV1Api();
try {
V1PodList podlist = api.listNamespacedPod("default", false, null, null, null, null, null, null, null, null);
for (V1Pod pod : podlist.getItems()) {
buffer.append(pod.getMetadata().getName());
buffer.append("\n");
System.out.println("Pod Name: " + pod.getMetadata().getName());
}
return buffer.toString();
} catch (ApiException e) {
// TODO Auto-generated catch block
System.err.println("Error while getting Pods in Default namespace");
return "Error: " + e.getResponseBody();
}
}

However, creating a Pod dynamically is equivalent to a “Hello World” — in a real world, you will have more complex requirements. So, next, let’s create a ReplicationController with X number of replicas. This will ensure that certain number of Pods of a given image are always running in your Kubernetes. Also notice the two other APIs — how to scale up/scale down ReplicationControllers and how to delete a ReplicationController.

// Provision ReplicationController
@GetMapping({ "/provisionreplicationcontroller" })
@ResponseBody
public String createreplicationcontroller(@RequestParam(name = "namespace", required = true) final String namespace,
@RequestParam(name = "replicationcontrollername", required = true) final String replicationcontrollername,
@RequestParam(name = "image", required = true) final String image,
@RequestParam(name = "replicas", required = true) final int replicas) {
System.out.println("Provision Replication Controller Data: " + namespace + " " + replicationcontrollername + " " + image);
CoreV1Api api = new CoreV1Api();
try {
Map<String,String> selector = new HashMap<String, String>();
selector.put("name", replicationcontrollername);
V1PodTemplateSpec podTemplate = new V1PodTemplateSpecBuilder().withNewMetadata().withName(replicationcontrollername).withLabels(selector).endMetadata().withNewSpec().addNewContainer().withName(replicationcontrollername).withImage(image).endContainer().endSpec().build();
V1ReplicationController replication = new V1ReplicationControllerBuilder().withNewMetadata().withName(replicationcontrollername).withNamespace(namespace).endMetadata().withNewSpec().withReplicas(Integer.valueOf(replicas)).withTemplate(podTemplate).withSelector(selector).endSpec().build();
replication = api.createNamespacedReplicationController(namespace, replication, null, "true", null);
return "Create ReplicationController Operation Output: " + replication.toString();
} catch (ApiException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return "Error: " + e.getResponseBody();
}
}
// Scale Up/Down ReplicationController
@GetMapping({ "/scalereplicationcontroller" })
@ResponseBody
public String scalereplicationcontroller(@RequestParam(name = "namespace", required = true) final String namespace,
@RequestParam(name = "replicationcount", required = true) final String replicationcount,
@RequestParam(name = "rcname", required = true) final String rcname) {
System.out.println("Scale Replication Controller Data: " + namespace + " " + replicationcount);
CoreV1Api api = new CoreV1Api();
try {
V1Scale scale = new V1ScaleBuilder().withNewMetadata().withNamespace(namespace).withName(rcname).endMetadata().withNewSpec().withReplicas(Integer.getInteger(replicationcount)).endSpec().build();
scale = api.replaceNamespacedReplicationControllerScale(rcname, namespace, scale, null, null);
return "Scale ReplicationController Operation Output: " + scale.toString();
} catch (ApiException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return "Error: " + e.getResponseBody();
}
}
// Delete ReplicationController
@GetMapping({ "/deletereplicationcontroller" })
@ResponseBody
public String deletereplicationcontroller(@RequestParam(name = "namespace", required = true) final String namespace,
@RequestParam(name = "replicationcontrollername", required = true) final String replicationcontrollername) {
System.out.println("Delete Replication Controller Data: " + namespace + " " + replicationcontrollername);
// First, scale the number of Pods associated with this ReplicationController to zero
this.scalereplicationcontroller(namespace, Integer.toString(0), replicationcontrollername);
// Now, delete the ReplicationController
CoreV1Api api = new CoreV1Api();
try {
V1Status status = api.deleteNamespacedReplicationController(replicationcontrollername, namespace, null, null, null, null, null, null);
return "Delete ReplicationController Operation Status" + status.getStatus() + " with message " + status.getMessage();
} catch (ApiException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return "Error: " + e.getResponseBody();
}
}

So far so good. Now, let’s suppose you want to expose these Pods to the outside world as a service. Which means you have to create a Service dynamically. Here is the code for a NodePort type of Service- you can similarly create a Cluster or LoadBalancer type of Service.

// Provision Service
@GetMapping({ "/provisionservice" })
@ResponseBody
public String provisionservice(@RequestParam(name = "namespace", required = true) final String namespace,
@RequestParam(name = "name", required = true) final String name,
@RequestParam(name = "type", required = true) final String type,
@RequestParam(name = "protocol", required = true) final String protocol,
@RequestParam(name = "port", required = true) final String port,
@RequestParam(name = "targetport", required = true) final String targetport,
@RequestParam(name = "nodeport", required = false) final String nodeport,
@RequestParam(name = "podselectorkey", required = true) final String podselectorkey,
@RequestParam(name = "podselectorvalue", required = true) final String podselectorvalue) {
System.out.println("Provision Service Data: " + namespace + " " + podselectorkey + " " + podselectorvalue);
CoreV1Api api = new CoreV1Api();
try {
Map<String,String> selector = new HashMap<String, String>();
selector.put(podselectorkey, podselectorvalue);
V1ServicePort serviceport = new V1ServicePortBuilder().withPort(Integer.valueOf(port)).withTargetPort(new IntOrString(Integer.valueOf(targetport))).withProtocol(protocol).build();
V1Service svc = new V1ServiceBuilder().withNewMetadata().withName(name).withLabels(selector).endMetadata().withNewSpec().withType(type).withSelector(selector).withPorts(serviceport).endSpec().build();
svc = api.createNamespacedService(namespace, svc, null, "true", null);
return "Create Service Operation Output: " + svc.toString();
} catch (ApiException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return "Error: " + e.getResponseBody();
}
}
// Delete Service
@GetMapping({ "/deleteservice" })
@ResponseBody
public String deleteservice(@RequestParam(name = "namespace", required = true) final String namespace,
@RequestParam(name = "name", required = true) final String name) {
System.out.println("Delete Service Data: " + namespace + " " + name);
CoreV1Api api = new CoreV1Api();
try {
V1Status status = api.deleteNamespacedService(name, namespace, "true", null, null, null, null, null);
return "Delete Service Operation Status" + status.getStatus() + " with message " + status.getMessage();
} catch (ApiException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return "Error: " + e.getResponseBody();
}
}

Many a time, you have to pass data to your Pod through Environment Variables. One possible way you can accomplish that is to use ConfigMaps. Here is how to create ConfigMaps dynamically.

// Provision ConfigMap
@GetMapping({ "/provisionconfigmap" })
@ResponseBody
public String provisionconfigmap(@RequestParam Map<String, String> params) {
Iterator<String> iter = params.keySet().iterator();
String namespace = null;
String name = null;
Map<String,String> mapdata = new HashMap<String, String>();
while (iter.hasNext()) {
String key = (String) iter.next();
if (key.equals("namespace")) {
namespace = params.get(key);
}
else {
if (key.equals("name")) {
name = params.get(key);
}
else {
mapdata.put(key, params.get(key));
}
}
System.out.println("Provision ConfigMap Data: " + key + " " + params.get(key));
}
CoreV1Api api = new CoreV1Api();
try {
Map<String,String> selector = new HashMap<String, String>();
selector.put("name", name);
V1ConfigMap configMap = new V1ConfigMapBuilder().withNewMetadata().withName(name).withNamespace(namespace).withLabels(selector).endMetadata().withData(mapdata).build();
configMap = api.createNamespacedConfigMap(namespace, configMap, null, "true", null);
return "Provision ConfigMap Operation Output" + configMap.toString();
} catch (ApiException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return "Error: " + e.getResponseBody();
}
}
// Delete ConfigMap
@GetMapping({ "/deleteconfigmap" })
@ResponseBody
public String deleteconfigmap(@RequestParam(name = "namespace", required = true) final String namespace,
@RequestParam(name = "name", required = true) final String name) {
System.out.println("Delete ConfigMap Data: " + namespace + " " + name);
CoreV1Api api = new CoreV1Api();
try {
V1Status status = api.deleteNamespacedConfigMap(name, namespace, "true", null, null, null, null, null);
return "Delete ConfigMap Operation Status" + status.getStatus() + " with message " + status.getMessage();
} catch (ApiException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return "Error: " + e.getResponseBody();
}
}

At this time, you would have seen how to create most of the popular Kubernetes objects dynamically. I will wrap up this article by showing you how to create Jobs in Kubernetes dynamically. Why would you need dynamic jobs? An example could be your end-user wants to run an adhoc report or adhoc task on-the-fly. You can even extend this concept and run recurrent jobs (that do not follow Cron Job pattern) on Kubernetes. How? After a job finishes, you provision another job in Kubernetes using the API mechanism.

Here is the API to provision a Job in Kubernetes. Since a Job can have different specs, you can modify below code to suit your needs.

// Provision Job
@GetMapping({ "/provisionjob" })
@ResponseBody
public String createjob(@RequestParam(name = "namespace", required = true) final String namespace,
@RequestParam(name = "image", required = true) final String image,
@RequestParam(name = "jobname", required = true) final String jobname,
@RequestParam(name = "configmapname", required = false) final String configmapname,
@RequestParam(name = "commands", required = false) final List<String> commands,
@RequestParam(name = "arguments", required = false) final List<String> arguments,
@RequestParam Map<String, String> params) {
System.out.println("Provision Job called");
BatchV1Api api = new BatchV1Api();
Map<String,String> label = new HashMap<String, String>();
label.put("name", jobname);
//V1Job job = new V1JobBuilder().withKind("Job").withNewMetadata().withGenerateName(jobname).withLabels(label).endMetadata().withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec().build();
V1JobBuilder jobBuilder = new V1JobBuilder().withKind("Job").withNewMetadata().withGenerateName(jobname).withLabels(label).endMetadata();
if (configmapname != null && !configmapname.equals("")) {
System.out.println("ConfigMap name provided -> " + configmapname);
List<V1EnvFromSource> envFrom = new ArrayList<V1EnvFromSource>();
V1ConfigMapEnvSource configMapRef = new V1ConfigMapEnvSourceBuilder().withName(configmapname).build();
V1EnvFromSource envSource = new V1EnvFromSourceBuilder().withConfigMapRef(configMapRef).build();
envFrom.add(envSource);
if (commands != null) {
System.out.println("Commands Provided");
Iterator<String> iter = commands.iterator();
while (iter.hasNext()) {
System.out.println("Command -> " + iter.next());
}
if (arguments != null && !arguments.equals("")) {
System.out.println("Arguments Provided");
jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withCommand(commands).withArgs(arguments).withEnvFrom(envFrom).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();
}
else {
System.out.println("Arguments NOT Provided");
jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withCommand(commands).withEnvFrom(envFrom).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();
}
}
else {
System.out.println("Commands NOT Provided");
jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withEnvFrom(envFrom).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();
}
}
else {
System.out.println("No configmapname provided, so, add all parameters in the Environment Variable !!");
List<V1EnvVar> env = new ArrayList<V1EnvVar>();
Iterator<String> iter = params.keySet().iterator();
while (iter.hasNext()) {
String key = (String) iter.next();
String value = params.get(key);
System.out.println("Env Data: " + key + " with value: " + value);
env.add(new V1EnvVarBuilder().withNewName(key).withNewValue(value).build());
}
if (commands != null) {
System.out.println("Commands Provided");
Iterator<String> iter1 = commands.iterator();
while (iter1.hasNext()) {
System.out.println("Command -> " + iter1.next());
}
if (arguments != null && !arguments.equals("")) {
System.out.println("Arguments Provided");
jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withCommand(commands).withArgs(arguments).withEnv(env).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();
}
else {
System.out.println("Arguments NOT Provided");
jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withCommand(commands).withEnv(env).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();
}
}
else {
System.out.println("Commands NOT Provided");
jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withEnv(env).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();
}
}
try {
V1Job job = api.createNamespacedJob(namespace, jobBuilder.build(), null, "true", null);
return "Job Created: " + job.toString();
} catch (ApiException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return e.getResponseBody();
}
}

‘hope you have found this article useful.