How to dynamically manage Kubernetes Objects

Sumit Datta
May 19, 2020 · 10 min read

Kubernetes is now the defacto “Application Server” when you build Cloud Native applications using technologies such as Docker, Microservices etc. You are also familiar with various Kubernetes Objects such as Pods, ReplicationContainers and Services. Further, you would have deployed these objects using either YAML files (by running kubectl commands) or using Helmcharts (by running Helm commands). However, there are situations when you may be forced to dynamically deploy objects inside Kubernetes. For example, let’s say, you want to execute some task on-demand based on end-user interaction. If this task runs infrequently, you may not want to create a Pod that sits idle most of the time and only runs when a user performs certain action. In such cases, you can use Serverless technology such as AWS Lambda or GCP Cloud Function to run your container on the fly. The other option is to dynamically create a Pod using Kubernetes API. In this article, we will explore how you can use Kubernetes API to manage objects inside Kubernetes. Also, we are going to use the official Java Client library of Kubernetes.

So, let’s get started. Create a Maven project in Eclipse and add the following entry in your POM file.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>dynamickubernetesprovision</groupId><artifactId>dynamickubernetesprovision</artifactId><version>0.0.1-SNAPSHOT</version><parent><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-parent</artifactId><version>2.1.4.RELEASE</version></parent><dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency><dependency><groupId>io.kubernetes</groupId><artifactId>client-java</artifactId><version>5.0.0</version><scope>compile</scope></dependency></dependencies><properties><java.version>1.8</java.version></properties><build><plugins><plugin><groupId>org.springframework.boot</groupId><artifactId>spring-boot-maven-plugin</artifactId></plugin></plugins></build></project>

In the above POM file, the key thing to note is the dependency you have to add for Java Client library of Kubernetes.

<dependency><groupId>io.kubernetes</groupId><artifactId>client-java</artifactId><version>5.0.0</version><scope>compile</scope></dependency>

Now, the basic concept of using Kubernetes API is to connect to the Kubernetes API Server and then invoke appropriate commands. There are two ways you can connect to API Server — one is In-Cluster and another is from Outside-Cluster. In-Cluster means your Java program is running inside the same Kubernetes Cluster to which you are planning to connect. Outside-Cluster means your Java program is running outside of the Kubernetes Cluster. In our examples, we will use In-Cluster; however, if you want to run the examples outside of your Kubernetes cluster, then, you will have to connect to API Server using a KubeConfig file — which actually stores your Kubernetes API server details. Do note that once you connect to API Server, from then onward, both approaches work the same way.

The other thing you need to do is to create a ServiceAccount inside your Kubernetes with appropriate roles assigned to it. And then assign the ServiceAccount to your Java program. Which means your Java program is now authorized to call only those APIs that fall under the purview of your ServiceAccount role. In our example, we have created a ServiceAccount with Cluster-Admin role, so that we can create any kind of objects within Kubernetes Cluster. However, in a real-life scenario, you should assign only those roles that are appropriate for your ServiceAccount.

Here is the serviceaccount.yaml we will use. Change it according to your requirements and then run kubectl apply -f serviceaccount.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
name: test-service-account
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: test-rbac
subjects:
- kind: ServiceAccount
# Reference to upper's `metadata.name`
name: test-service-account
# Reference to upper's `metadata.namespace`
namespace: default
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io

Now, we will create a simple RESTful service that acts as a Java client and will perform operations on Kubernetes cluster. So, let’s go ahead and write a SpringBoots application.

package com.test.dynamickubernetes.controller;import java.io.IOException;import java.util.HashMap;import java.util.Iterator;import java.util.Map;import org.springframework.stereotype.Controller;import org.springframework.web.bind.annotation.CrossOrigin;import org.springframework.web.bind.annotation.GetMapping;import org.springframework.web.bind.annotation.RequestParam;import org.springframework.web.bind.annotation.ResponseBody;import io.kubernetes.client.ApiClient;import io.kubernetes.client.ApiException;import io.kubernetes.client.Configuration;import io.kubernetes.client.apis.CoreV1Api;import io.kubernetes.client.custom.IntOrString;import io.kubernetes.client.models.V1ConfigMap;import io.kubernetes.client.models.V1Pod;import io.kubernetes.client.models.V1PodBuilder;import io.kubernetes.client.models.V1PodList;import io.kubernetes.client.models.V1PodTemplateSpec;import io.kubernetes.client.models.V1PodTemplateSpecBuilder;import io.kubernetes.client.models.V1ReplicationController;import io.kubernetes.client.models.V1ReplicationControllerBuilder;import io.kubernetes.client.models.V1Scale;import io.kubernetes.client.models.V1ScaleBuilder;import io.kubernetes.client.models.V1Service;import io.kubernetes.client.models.V1ServiceBuilder;import io.kubernetes.client.models.V1ServicePort;import io.kubernetes.client.models.V1ServicePortBuilder;import io.kubernetes.client.models.V1ConfigMapBuilder;import io.kubernetes.client.models.V1Status;import io.kubernetes.client.util.Config;@CrossOrigin@Controllerpublic class DynamicKubernetesProvisionController {private ApiClient client = null;public DynamicKubernetesProvisionController() {System.out.println("DynamicKubernetesProvisionController created");try {client = Config.defaultClient();Configuration.setDefaultApiClient(client);} catch (IOException e) {// TODO Auto-generated catch blockSystem.err.println("Error while creating Kubernetes APIClient");e.printStackTrace();}}}

With the basic boilerplate code in place, let’s create a Pod dynamically. Here is the code — see createpod API below. It takes three arguments, namely, the namespace where the Pod has to be created, the docker image to be run as a Container inside the Pod and the Pod name.

The code below also shows how to get list of Pods and how to delete a Pod.

// Provision Pod@GetMapping({ "/provisionpod" })@ResponseBodypublic String createpod(@RequestParam(name = "namespace", required = true) final String namespace,@RequestParam(name = "image", required = true) final String image,@RequestParam(name = "podname", required = true) final String podname) {CoreV1Api api = new CoreV1Api();Map<String,String> label = new HashMap<String, String>();label.put("name", podname);V1Pod pod = new V1PodBuilder().withNewMetadata().withName(podname).withLabels(label).endMetadata().withNewSpec().addNewContainer().withName(podname).withImage(image).endContainer().endSpec().build();try {V1Pod newPod = api.createNamespacedPod(namespace, pod, null, "true", null);return "Pod Created: " + newPod.toString();} catch (ApiException e) {// TODO Auto-generated catch blocke.printStackTrace();return e.getResponseBody();}}// Delete Pod@GetMapping({ "/deletepod" })@ResponseBodypublic String deletepod(@RequestParam(name = "namespace", required = true) final String namespace,@RequestParam(name = "podname", required = true) final String podname) {CoreV1Api api = new CoreV1Api();try {V1Status status = api.deleteNamespacedPod(podname, namespace, "true", null, null, null, null, null);return "Delete Pod Operation Status" + status.getStatus() + " with message " + status.getMessage();} catch (ApiException e) {// TODO Auto-generated catch blockreturn "Error: " + e.getResponseBody();}}// Get list of pods@GetMapping({ "/pods" })@ResponseBodypublic String getPods() {StringBuffer buffer = new StringBuffer();CoreV1Api api = new CoreV1Api();try {V1PodList podlist = api.listNamespacedPod("default", false, null, null, null, null, null, null, null, null);for (V1Pod pod : podlist.getItems()) {buffer.append(pod.getMetadata().getName());buffer.append("\n");System.out.println("Pod Name: " + pod.getMetadata().getName());}return buffer.toString();} catch (ApiException e) {// TODO Auto-generated catch blockSystem.err.println("Error while getting Pods in Default namespace");return "Error: " + e.getResponseBody();}}

However, creating a Pod dynamically is equivalent to a “Hello World” — in a real world, you will have more complex requirements. So, next, let’s create a ReplicationController with X number of replicas. This will ensure that certain number of Pods of a given image are always running in your Kubernetes. Also notice the two other APIs — how to scale up/scale down ReplicationControllers and how to delete a ReplicationController.

// Provision ReplicationController@GetMapping({ "/provisionreplicationcontroller" })@ResponseBodypublic String createreplicationcontroller(@RequestParam(name = "namespace", required = true) final String namespace,@RequestParam(name = "replicationcontrollername", required = true) final String replicationcontrollername,@RequestParam(name = "image", required = true) final String image,
@RequestParam(name = "replicas", required = true) final int replicas) {
System.out.println("Provision Replication Controller Data: " + namespace + " " + replicationcontrollername + " " + image);CoreV1Api api = new CoreV1Api();try {Map<String,String> selector = new HashMap<String, String>();selector.put("name", replicationcontrollername);V1PodTemplateSpec podTemplate = new V1PodTemplateSpecBuilder().withNewMetadata().withName(replicationcontrollername).withLabels(selector).endMetadata().withNewSpec().addNewContainer().withName(replicationcontrollername).withImage(image).endContainer().endSpec().build();V1ReplicationController replication = new V1ReplicationControllerBuilder().withNewMetadata().withName(replicationcontrollername).withNamespace(namespace).endMetadata().withNewSpec().withReplicas(Integer.valueOf(replicas)).withTemplate(podTemplate).withSelector(selector).endSpec().build();replication = api.createNamespacedReplicationController(namespace, replication, null, "true", null);return "Create ReplicationController Operation Output: " + replication.toString();} catch (ApiException e) {// TODO Auto-generated catch blocke.printStackTrace();return "Error: " + e.getResponseBody();}}// Scale Up/Down ReplicationController@GetMapping({ "/scalereplicationcontroller" })@ResponseBodypublic String scalereplicationcontroller(@RequestParam(name = "namespace", required = true) final String namespace,@RequestParam(name = "replicationcount", required = true) final String replicationcount,@RequestParam(name = "rcname", required = true) final String rcname) {System.out.println("Scale Replication Controller Data: " + namespace + " " + replicationcount);CoreV1Api api = new CoreV1Api();try {V1Scale scale = new V1ScaleBuilder().withNewMetadata().withNamespace(namespace).withName(rcname).endMetadata().withNewSpec().withReplicas(Integer.getInteger(replicationcount)).endSpec().build();scale = api.replaceNamespacedReplicationControllerScale(rcname, namespace, scale, null, null);return "Scale ReplicationController Operation Output: " + scale.toString();} catch (ApiException e) {// TODO Auto-generated catch blocke.printStackTrace();return "Error: " + e.getResponseBody();}}// Delete ReplicationController@GetMapping({ "/deletereplicationcontroller" })@ResponseBodypublic String deletereplicationcontroller(@RequestParam(name = "namespace", required = true) final String namespace,@RequestParam(name = "replicationcontrollername", required = true) final String replicationcontrollername) {System.out.println("Delete Replication Controller Data: " + namespace + " " + replicationcontrollername);// First, scale the number of Pods associated with this ReplicationController to zerothis.scalereplicationcontroller(namespace, Integer.toString(0), replicationcontrollername);// Now, delete the ReplicationControllerCoreV1Api api = new CoreV1Api();try {V1Status status = api.deleteNamespacedReplicationController(replicationcontrollername, namespace, null, null, null, null, null, null);return "Delete ReplicationController Operation Status" + status.getStatus() + " with message " + status.getMessage();} catch (ApiException e) {// TODO Auto-generated catch blocke.printStackTrace();return "Error: " + e.getResponseBody();}}

So far so good. Now, let’s suppose you want to expose these Pods to the outside world as a service. Which means you have to create a Service dynamically. Here is the code for a NodePort type of Service- you can similarly create a Cluster or LoadBalancer type of Service.

// Provision Service@GetMapping({ "/provisionservice" })@ResponseBodypublic String provisionservice(@RequestParam(name = "namespace", required = true) final String namespace,@RequestParam(name = "name", required = true) final String name,@RequestParam(name = "type", required = true) final String type,@RequestParam(name = "protocol", required = true) final String protocol,@RequestParam(name = "port", required = true) final String port,@RequestParam(name = "targetport", required = true) final String targetport,@RequestParam(name = "nodeport", required = false) final String nodeport,@RequestParam(name = "podselectorkey", required = true) final String podselectorkey,@RequestParam(name = "podselectorvalue", required = true) final String podselectorvalue) {System.out.println("Provision Service Data: " + namespace + " " + podselectorkey + " " + podselectorvalue);CoreV1Api api = new CoreV1Api();try {Map<String,String> selector = new HashMap<String, String>();selector.put(podselectorkey, podselectorvalue);V1ServicePort serviceport = new V1ServicePortBuilder().withPort(Integer.valueOf(port)).withTargetPort(new IntOrString(Integer.valueOf(targetport))).withProtocol(protocol).build();V1Service svc = new V1ServiceBuilder().withNewMetadata().withName(name).withLabels(selector).endMetadata().withNewSpec().withType(type).withSelector(selector).withPorts(serviceport).endSpec().build();svc = api.createNamespacedService(namespace, svc, null, "true", null);return "Create Service Operation Output: " + svc.toString();} catch (ApiException e) {// TODO Auto-generated catch blocke.printStackTrace();return "Error: " + e.getResponseBody();}}// Delete Service@GetMapping({ "/deleteservice" })@ResponseBodypublic String deleteservice(@RequestParam(name = "namespace", required = true) final String namespace,@RequestParam(name = "name", required = true) final String name) {System.out.println("Delete Service Data: " + namespace + " " + name);CoreV1Api api = new CoreV1Api();try {V1Status status = api.deleteNamespacedService(name, namespace, "true", null, null, null, null, null);return "Delete Service Operation Status" + status.getStatus() + " with message " + status.getMessage();} catch (ApiException e) {// TODO Auto-generated catch blocke.printStackTrace();return "Error: " + e.getResponseBody();}}

Many a time, you have to pass data to your Pod through Environment Variables. One possible way you can accomplish that is to use ConfigMaps. Here is how to create ConfigMaps dynamically.

// Provision ConfigMap@GetMapping({ "/provisionconfigmap" })@ResponseBodypublic String provisionconfigmap(@RequestParam Map<String, String> params) {Iterator<String> iter = params.keySet().iterator();String namespace = null;String name = null;Map<String,String> mapdata = new HashMap<String, String>();while (iter.hasNext()) {String key = (String) iter.next();if (key.equals("namespace")) {namespace = params.get(key);}else {if (key.equals("name")) {name = params.get(key);}else {mapdata.put(key, params.get(key));}}System.out.println("Provision ConfigMap Data: " + key + " " + params.get(key));}CoreV1Api api = new CoreV1Api();try {Map<String,String> selector = new HashMap<String, String>();selector.put("name", name);V1ConfigMap configMap = new V1ConfigMapBuilder().withNewMetadata().withName(name).withNamespace(namespace).withLabels(selector).endMetadata().withData(mapdata).build();configMap = api.createNamespacedConfigMap(namespace, configMap, null, "true", null);return "Provision ConfigMap Operation Output" + configMap.toString();} catch (ApiException e) {// TODO Auto-generated catch blocke.printStackTrace();return "Error: " + e.getResponseBody();}}// Delete ConfigMap@GetMapping({ "/deleteconfigmap" })@ResponseBodypublic String deleteconfigmap(@RequestParam(name = "namespace", required = true) final String namespace,@RequestParam(name = "name", required = true) final String name) {System.out.println("Delete ConfigMap Data: " + namespace + " " + name);CoreV1Api api = new CoreV1Api();try {V1Status status = api.deleteNamespacedConfigMap(name, namespace, "true", null, null, null, null, null);return "Delete ConfigMap Operation Status" + status.getStatus() + " with message " + status.getMessage();} catch (ApiException e) {// TODO Auto-generated catch blocke.printStackTrace();return "Error: " + e.getResponseBody();}}

At this time, you would have seen how to create most of the popular Kubernetes objects dynamically. I will wrap up this article by showing you how to create Jobs in Kubernetes dynamically. Why would you need dynamic jobs? An example could be your end-user wants to run an adhoc report or adhoc task on-the-fly. You can even extend this concept and run recurrent jobs (that do not follow Cron Job pattern) on Kubernetes. How? After a job finishes, you provision another job in Kubernetes using the API mechanism.

Here is the API to provision a Job in Kubernetes. Since a Job can have different specs, you can modify below code to suit your needs.

// Provision Job@GetMapping({ "/provisionjob" })@ResponseBodypublic String createjob(@RequestParam(name = "namespace", required = true) final String namespace,@RequestParam(name = "image", required = true) final String image,@RequestParam(name = "jobname", required = true) final String jobname,@RequestParam(name = "configmapname", required = false) final String configmapname,@RequestParam(name = "commands", required = false) final List<String> commands,@RequestParam(name = "arguments", required = false) final List<String> arguments,@RequestParam Map<String, String> params) {System.out.println("Provision Job called");BatchV1Api api = new BatchV1Api();Map<String,String> label = new HashMap<String, String>();label.put("name", jobname);//V1Job job = new V1JobBuilder().withKind("Job").withNewMetadata().withGenerateName(jobname).withLabels(label).endMetadata().withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec().build();V1JobBuilder jobBuilder = new V1JobBuilder().withKind("Job").withNewMetadata().withGenerateName(jobname).withLabels(label).endMetadata();if (configmapname != null && !configmapname.equals("")) {System.out.println("ConfigMap name provided -> " + configmapname);List<V1EnvFromSource> envFrom = new ArrayList<V1EnvFromSource>();V1ConfigMapEnvSource configMapRef = new V1ConfigMapEnvSourceBuilder().withName(configmapname).build();V1EnvFromSource envSource = new V1EnvFromSourceBuilder().withConfigMapRef(configMapRef).build();envFrom.add(envSource);if (commands != null) {System.out.println("Commands Provided");Iterator<String> iter = commands.iterator();while (iter.hasNext()) {System.out.println("Command -> " + iter.next());}if (arguments != null && !arguments.equals("")) {System.out.println("Arguments Provided");jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withCommand(commands).withArgs(arguments).withEnvFrom(envFrom).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();}else {System.out.println("Arguments NOT Provided");jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withCommand(commands).withEnvFrom(envFrom).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();}}else {System.out.println("Commands NOT Provided");jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withEnvFrom(envFrom).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();}}else {System.out.println("No configmapname provided, so, add all parameters in the Environment Variable !!");List<V1EnvVar> env = new ArrayList<V1EnvVar>();Iterator<String> iter = params.keySet().iterator();while (iter.hasNext()) {String key = (String) iter.next();String value = params.get(key);System.out.println("Env Data: " + key + " with value: " + value);env.add(new V1EnvVarBuilder().withNewName(key).withNewValue(value).build());}if (commands != null) {System.out.println("Commands Provided");Iterator<String> iter1 = commands.iterator();while (iter1.hasNext()) {System.out.println("Command -> " + iter1.next());}if (arguments != null && !arguments.equals("")) {System.out.println("Arguments Provided");jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withCommand(commands).withArgs(arguments).withEnv(env).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();}else {System.out.println("Arguments NOT Provided");jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withCommand(commands).withEnv(env).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();}}else {System.out.println("Commands NOT Provided");jobBuilder = jobBuilder.withNewSpec().withNewTemplate().withNewSpec().addNewContainer().withName(jobname).withImage(image).withEnv(env).endContainer().withRestartPolicy("Never").endSpec().endTemplate().withTtlSecondsAfterFinished(new Integer(0)).endSpec();}}try {V1Job job = api.createNamespacedJob(namespace, jobBuilder.build(), null, "true", null);return "Job Created: " + job.toString();} catch (ApiException e) {// TODO Auto-generated catch blocke.printStackTrace();return e.getResponseBody();}}

‘hope you have found this article useful.

The Startup

Get smarter at building your thing. Join The Startup’s +793K followers.

Sign up for Top 10 Stories

By The Startup

Get smarter at building your thing. Subscribe to receive The Startup's top 10 most read stories — delivered straight into your inbox, once a week. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Sumit Datta

Written by

Loves dabbling in new technologies. All views expressed are mine.

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +793K followers.

Sumit Datta

Written by

Loves dabbling in new technologies. All views expressed are mine.

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +793K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store