Open in app

Sign in

Write

Sign in

How Does Kubectl Work: Writing Custom Kubectl Commands

Hasan Bingölbali
7 min readJan 16, 2024

--

Kubernetes works like magic but it is not magic. It is fundamentally grounded in the simplicity of REST API calls. This straightforward mechanism is key to its powerful capabilities. Today, we’re going to dive into the inner workings of Kubernetes, specifically focusing on what happens behind the scenes when we execute a kubectl command.

At its core, kubectl is more than just a command-line interface; it's the bridge between the user and the Kubernetes api server. We will start with kubectl authentication and proceed with communication with api-server and finish off by forking Kubernetes repository and writing our own kubectl commands. You can access the fork in GitHub.

AUTHENTICATION

While Kubernetes offers a variety of authentication methods to interact with the API server, such as service account tokens for pods or certificate based authentication for Kubelet, our focus here is on the authentication process of kubectl. Here is an step by step explaination the of how kubectl authenticates:

  1. By default, kubectl looks for a file named config in the $HOME/.kube directory. You can specify other kubeconfig files by setting the KUBECONFIG environment variable or by setting the --kubeconfig flag.
https://github.com/kubernetes/client-go/blob/b13c4f4b008a6c102834944b6191ae3c30422b0d/tools/clientcmd/loader.go#L39

2. The user section in a kubeconfig file details the credentials for authentication. These can vary significantly:

  • Client Certificates: A common method where kubectl utilizes a client certificate and key for authentication.
  • Bearer Tokens: kubectl can also authenticate using bearer tokens, included in the Authorization header of each request.
  • External Authentication Plugins: For more sophisticated authentication needs, kubectl can use external plugins, often essential in cloud-based Kubernetes environments.
- name: eksUser@goodjob.eu-west-2.eksctl.io
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - eks
      - get-token
      - --output
      - json
      - --cluster-name
      - goodjob
      - --region
      - eu-west-2
      command: aws
      env:
      - name: AWS_STS_REGIONAL_ENDPOINTS
        value: regional
      interactiveMode: IfAvailable
      provideClusterInfo: false
- name: minikube
  user:
    client-certificate: /Users/hasanbingolbali/.minikube/profiles/minikube/client.crt
    client-key: /Users/hasanbingolbali/.minikube/profiles/minikube/client.key

Provided kubeconfig illustrates two distinct user entries in a kubeconfig file, each demonstrating a different method of authentication for kubectl.

** EKS User Authentication via AWS CLI

  • User Entry: eksUser@goodjob.eu-west-2.eksctl.io
  • Authentication Method: This entry uses an external command for authentication (aws eks get-token). It's a common pattern for cloud-based Kubernetes services like AWS EKS (Elastic Kubernetes Service).

** Minikube User Authentication via Client Certificates

  • User Entry: minikube
  • Authentication Method: This entry uses client certificates for authentication, a common method for local or self-managed Kubernetes clusters

REST CALL

Now that we know Kubectl is an utility command line tool that make it easier to send rest-api calls to the apiserver, we should be able to send rest-calls by ourselves. However we’ll also need to CA certificate file-path or certificate-data which is also accessible from kube-config file.

- cluster:
    certificate-authority: /Users/hasanbingolbali/.minikube/ca.crt
    server: https://127.0.0.1:60749
  name: minikube

Let’s send the following request in order to fetch information about the pods in default namespace to our api-server:

curl --cacert /Users/hasanbingolbali/.minikube/ca.crt \
     --cert /Users/hasanbingolbali/.minikube/profiles/minikube/client.crt \
     --key /Users/hasanbingolbali/.minikube/profiles/minikube/client.key \
     https://localhost:60749/api/v1/namespaces/default/pods
As expected it worked

However, the rest-api contains so much information and you probably won’t need that much information for listing pods. And that’s where kubectl is making it easy for us to work with kubernetes.

WRITING CUSTOM KUBECTL COMMAND

Since we know that Kubernetes is an open-source project, we can explore and play with it. First step is to clone GitHub repository.

Kubectl package is under the ./staging/src/k8s.io/kubectl. Isolated kubectl package is also available in a separate repository. But for build purposes, you must clone the main repository. Under the cmd repository the keywords for kubectl are grouped into the folders as follows:

Let’s get into the kubectl/cmd/create folder and observe and talk about create_namespace.go file. For you to follow easily the code is provided below:

package create

import (
 "context"
 "fmt"

 "github.com/spf13/cobra"
 corev1 "k8s.io/api/core/v1"
 metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

 "k8s.io/apimachinery/pkg/runtime"
 coreclient "k8s.io/client-go/kubernetes/typed/core/v1"
 "k8s.io/kubectl/pkg/scheme"
 "k8s.io/kubectl/pkg/util"

 "k8s.io/cli-runtime/pkg/genericclioptions"
 "k8s.io/cli-runtime/pkg/genericiooptions"
 cmdutil "k8s.io/kubectl/pkg/cmd/util"
 "k8s.io/kubectl/pkg/util/i18n"
 "k8s.io/kubectl/pkg/util/templates"
)

var (
 namespaceLong = templates.LongDesc(i18n.T(`
  Create a namespace with the specified name.`))

 namespaceExample = templates.Examples(i18n.T(`
   # Create a new namespace named my-namespace
   kubectl create namespace my-namespace`))
)

// NamespaceOptions is the options for 'create namespace' sub command
type NamespaceOptions struct {
 // PrintFlags holds options necessary for obtaining a printer
 PrintFlags *genericclioptions.PrintFlags
 // Name of resource being created
 Name string

 DryRunStrategy      cmdutil.DryRunStrategy
 ValidationDirective string
 CreateAnnotation    bool
 FieldManager        string

 Client *coreclient.CoreV1Client

 PrintObj func(obj runtime.Object) error

 genericiooptions.IOStreams
}

// NewNamespaceOptions creates a new *NamespaceOptions with sane defaults
func NewNamespaceOptions(ioStreams genericiooptions.IOStreams) *NamespaceOptions {
 return &NamespaceOptions{
  PrintFlags: genericclioptions.NewPrintFlags("created").WithTypeSetter(scheme.Scheme),
  IOStreams:  ioStreams,
 }
}

// NewCmdCreateNamespace is a macro command to create a new namespace
func NewCmdCreateNamespace(f cmdutil.Factory, ioStreams genericiooptions.IOStreams) *cobra.Command {

 o := NewNamespaceOptions(ioStreams)

 cmd := &cobra.Command{
  Use:                   "namespace NAME [--dry-run=server|client|none]",
  DisableFlagsInUseLine: true,
  Aliases:               []string{"ns"},
  Short:                 i18n.T("Create a namespace with the specified name"),
  Long:                  namespaceLong,
  Example:               namespaceExample,
  Run: func(cmd *cobra.Command, args []string) {
   cmdutil.CheckErr(o.Complete(f, cmd, args))
   cmdutil.CheckErr(o.Validate())
   cmdutil.CheckErr(o.Run())
  },
 }

 o.PrintFlags.AddFlags(cmd)

 cmdutil.AddApplyAnnotationFlags(cmd)
 cmdutil.AddValidateFlags(cmd)
 cmdutil.AddDryRunFlag(cmd)
 cmdutil.AddFieldManagerFlagVar(cmd, &o.FieldManager, "kubectl-create")

 return cmd
}

// Complete completes all the required options
func (o *NamespaceOptions) Complete(f cmdutil.Factory, cmd *cobra.Command, args []string) error {
 name, err := NameFromCommandArgs(cmd, args)
 if err != nil {
  return err
 }

 restConfig, err := f.ToRESTConfig()
 if err != nil {
  return err
 }
 o.Client, err = coreclient.NewForConfig(restConfig)
 if err != nil {
  return err
 }

 o.Name = name
 o.DryRunStrategy, err = cmdutil.GetDryRunStrategy(cmd)
 if err != nil {
  return err
 }
 o.CreateAnnotation = cmdutil.GetFlagBool(cmd, cmdutil.ApplyAnnotationsFlag)
 cmdutil.PrintFlagsWithDryRunStrategy(o.PrintFlags, o.DryRunStrategy)
 printer, err := o.PrintFlags.ToPrinter()
 if err != nil {
  return err
 }
 o.PrintObj = func(obj runtime.Object) error {
  return printer.PrintObj(obj, o.Out)
 }

 o.ValidationDirective, err = cmdutil.GetValidationDirective(cmd)
 return err
}

// Run calls the CreateSubcommandOptions.Run in NamespaceOpts instance
func (o *NamespaceOptions) Run() error {
 namespace := o.createNamespace()
 if err := util.CreateOrUpdateAnnotation(o.CreateAnnotation, namespace, scheme.DefaultJSONEncoder()); err != nil {
  return err
 }

 if o.DryRunStrategy != cmdutil.DryRunClient {
  createOptions := metav1.CreateOptions{}
  if o.FieldManager != "" {
   createOptions.FieldManager = o.FieldManager
  }
  createOptions.FieldValidation = o.ValidationDirective
  if o.DryRunStrategy == cmdutil.DryRunServer {
   createOptions.DryRun = []string{metav1.DryRunAll}
  }
  var err error
  namespace, err = o.Client.Namespaces().Create(context.TODO(), namespace, createOptions)
  if err != nil {
   return err
  }
 }
 return o.PrintObj(namespace)
}

// createNamespace outputs a namespace object using the configured fields
func (o *NamespaceOptions) createNamespace() *corev1.Namespace {
 namespace := &corev1.Namespace{
  TypeMeta:   metav1.TypeMeta{APIVersion: corev1.SchemeGroupVersion.String(), Kind: "Namespace"},
  ObjectMeta: metav1.ObjectMeta{Name: o.Name},
 }
 return namespace
}

// Validate validates required fields are set to support structured generation
func (o *NamespaceOptions) Validate() error {
 if len(o.Name) == 0 {
  return fmt.Errorf("name must be specified")
 }
 return nil
}

It is not that straightforward, but most of the code is about processing options and validation. I want to take your attention to the following line:

namespace, err = o.Client.Namespaces().Create(context.TODO(), namespace, createOptions)

This is where it sends request to the api-server. Now that we observed how we send request to the api-server. Let’s create file create_lovely_pod.go, that will create a pod with full of love.

type RunOptions struct {
 cmdutil.OverrideOptions
 PrintFlags *genericclioptions.PrintFlags
 PrintObj   func(obj runtime.Object) error
 Client     *coreclient.CoreV1Client
 genericiooptions.IOStreams
}

func NewRunOptions(ioStreams genericiooptions.IOStreams) *RunOptions {
 return &RunOptions{
  PrintFlags: genericclioptions.NewPrintFlags("created").WithTypeSetter(scheme.Scheme),
  IOStreams:  ioStreams,
 }
}

func NewCmdRun(f cmdutil.Factory, streams genericiooptions.IOStreams) *cobra.Command {
 o := NewRunOptions(streams)

 cmd := &cobra.Command{
  Use:                   "lovely-pod",
  DisableFlagsInUseLine: true,
  Short:                 i18n.T("Run a particular image on the cluster"),
  Long:                  "runLong",
  Example:               "runExample",
  Run: func(cmd *cobra.Command, args []string) {
   cmdutil.CheckErr(o.Run(f, cmd, args))
  },
 }

 return cmd
}

func (o *RunOptions) Run(f cmdutil.Factory, cmd *cobra.Command, args []string) error {
 var err error
 restConfig, err := f.ToRESTConfig()
 if err != nil {
  return err
 }
 o.Client, err = coreclient.NewForConfig(restConfig)
 if err != nil {
  return err
 }
 printer, err := o.PrintFlags.ToPrinter()
 if err != nil {
  return err
 }
 o.PrintObj = func(obj runtime.Object) error {
  return printer.PrintObj(obj, o.Out)
 }

 pod, err := o.Client.Pods("default").Create(context.TODO(), &corev1.Pod{
  ObjectMeta: metav1.ObjectMeta{
   Name: "iloveyou",
  },
  Spec: corev1.PodSpec{
   Containers: []corev1.Container{
    {
     Name:  "nginx",
     Image: "nginx",
    },
   },
  },
 }, metav1.CreateOptions{})
 if err != nil {
  return err
 }
 return o.PrintObj(pod)
}

now that we created a custom kubectl command, let’s build kubectl using:

make WHAT=cmd/kubectl KUBE_VERBOSE=5

now that we should have binaries under the _output directory. Let’s test it.

Yes we wrote our custom kubectl.

Yes it worked, It is very basic implementation for writing custom kubectl but it is a big step to grasp kubectl fundamentals.

FINAL WORDS

I think, It was a very informative article. We went through the fundamentals and applied it by writing our custom kubectl command. You can access my fork in GitHub. If you liked the article please give it a like. In the upcoming article I will run performance benchmark tests for ORM and plain sql and I’ll share the creation process and result with you. You can follow me for more content like this. Good bye!

Kubectl
Kubernetes

--

--

Hasan Bingölbali
Hasan Bingölbali

Written by Hasan Bingölbali

71 Followers
6 Following

No responses yet

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams