Every Wednesday, Nick Chase and Eric Gregory from Mirantis go over the week’s cloud native and industry news on the Radio Cloud Native podcast.

This week, Eric and regular guest-host John Jainschigg discussed:

• New numbers on Kubernetes management challenges
• Google releases open source OSV-Scanner
• Major vulnerability in AWS Elastic Container Registry discovered and fixed
• And more on the podcast, including a wide-ranging conversation about cloud native trends in 2022 and the year ahead

You can watch the entire episode below or download the podcast from Apple Podcasts, Spotify, or wherever you get your podcasts. If you’d like to tune into the next show live, follow Mirantis on LinkedIn to receive our announcement of the next broadcast.

New numbers on Kubernetes management challenges

John: Mike Vizard, writing in Container Journal a few weeks back, did some analysis on the November update to DataDog’s several-years-running Container Report, pointing up some of the management challenges Kubernetes users are confronting. The report itself, though, is pretty upbeat: Kubernetes use among container-oriented organizations grew by 5%-ish between January and June of this year. Serverless is now used by 35% of these same orgs. Multi-cloud is also growing, and scaling — as you’d expect — with organization size: smaller orgs tend to have single clouds, but about 35% of orgs with 1000+ employees have more than one.

Ingress is now used by about 35% of orgs — DataDog itself points to Kubernetes Gateway API as the new frontier here: they’re waiting to see if Gateway displaces ingress or if the two functions will be used together. Service mesh is still way early — only 10% of orgs are using Istio and another roughly 2% are using LinkerD, with others seen more rarely.

The management challenges Vizard talks about in his piece are daunting, though. Most hosts are still running Kubernetes 1.21, which reached end of life earlier this year. A lot of hosts, moreover, are using an unsupported version of containerd to run containers, and about 30% of hosts are running version 1.4, which is also past EOL. The problem is that — for some orgs — staying with a Kubernetes version has become a necessity, at least in the short term, since existing apps will break because of changes in newer releases of the platform.

Organizations are doing a little better with access management this year — only 40%, down from 50% in 2019, are configuring Kubernetes with “overly permissive RBAC privileges.” But now the improvement seems to have flattened. We saw a few new projects at Kubecon, in October, that want to improve matters by making RBAC easier to use, and finer-grained. Hopefully they’ll have some success. We’d hope that widespread use of Open Policy Agent in DevSecOps frameworks will also start making this stat fall again.

Overall, the DataDog report is pretty illuminating. Vizard spoke with John Kendall, Senior Product Manager of Containers at Datadog, who said that the way forward was with managed Kubernetes services that provide organizations access to curated instances of the platform. Working with such services, Vizard pointed out, can both help solve the update challenge — for those who decide to update K8S — and make staying with a legacy version much less risky: the managed service provider taking responsibility for securing and continually optimizing. Managed services providers may also be able to help by updating existing cloud native applications to thrive on newer versions of Kubernetes, and by helping to build software supply chains that make it easier to create apps without Kubernetes version dependencies.

Google releases open source OSV-Scanner

Eric: Google announced a new Go-based vulnerability scanner that utilizes the OSV.dev vulnerability database, fittingly titled OSV-Scanner.

Introduced in 2021, the OSV (or Open Source Vulnerabilities) project consists of…

1. a standardized format for vulnerability data that is mapped to open source versioning schemes (and is basically just a standardized organizational pattern in JSON, so it’s human and machine readable)
2. a site and API that aggregates OSV-formatted data from distributed databases

The new OSV-Scanner utility evaluates the whole open source dependency tree against its mega-database. The major advantage here is the breadth of package types covered, including not just Go but (deep breath) Linux kernel, Debian, Alpine, Android, npm, PyPI, crates.io, Maven, RubyGems, NuGet, Packagist, and OSS-Fuzz. Some estimates reckon OSV.dev as the largest existing vulnerability database, so it’s nice to have an automatable tool to leverage that.

According to Google’s announcement post, the next steps for the project should bring standalone CI actions and support for C/C++ vulnerabilities.

Major vulnerability in AWS Elastic Container Registry discovered and fixed

John: Container registries are part of any container development setup, and are now critical to the broader ecosystem allowing for centralized storage and distribution of both private and public/commercial container images. And they’re not super-super complicated — there are lots of ways to improve container registry functionality, like encryption and image scanning and image signing and the ability to store artifact formats beyond container images. And there’s a vivid competitive market around such improvements. But basic registry stuff is typically imagined to be pretty solid.

Nah. AWS just fixed a severe vulnerability in their Elastic Container Registry that could have been bad news. Discovered by Gafnit Amiga, director of security research at Lightspin, the vuln would let bad people (and security researchers, okay) delete, update, and modify ECR public images, as well as fool with images stored in ECR belonging to private AWS accounts. ECR hosts many super-popular public images, including a “top six” that alone account for something like 13 billion cumulative pulls.

Amiga said that the vulnerability might allow an attacker to poison public images and then see these used widely, based on the trust created around public ECR’s commitment to verification. With essentially every detail of containers open to modification, potential attacks that could be launched via this supply chain vulnerability would be, she added: “limited only by the craftiness and goals of the adversary.”

Well, adversaries can sit back down, because the vulnerability was reported to AWS on November 12, and fixed by 24 hours later. AWS did a serious audit thereafter and are confident that the only activity associated with the issue was between accounts owned by the researcher. No other customer accounts were touched. Brava to Lightspin and this researcher. Side-note: women in cybersecurity are saving the world’s butt every day.

Check out the podcast for more of this week’s stories.

The cloud native year-in-review was originally published in Mirantis on Medium, where people are continuing the conversation by highlighting and responding to this story.

Kubernetes provides a portable, resilient, and resource-aware substrate for code that lives on the cloud. As Kubernetes has grown more popular, cloud infrastructure patterns have grown more complex: organizations may run hundreds or thousands of Kubernetes clusters at edge sites, and workloads may be orchestrated across multiple cloud providers or on-prem data centers.

That complexity calls for a unified interface through which clusters may be provisioned and managed, programmatically and at scale. The Kubernetes community created just such a tool in Cluster API.

In this tutorial, we will…

• Explain the fundamentals of Cluster API — what it is, how it works, and the core concepts that govern its design
• Show you how to set up a local management cluster that will enable you to deploy new clusters to a cloud provider such as AWS
• Walk you through the process of provisioning a workload cluster on AWS, and explain how to interact with that cluster via Lens

This primer will serve as a foundation for further tutorials on Cluster API, in which we will show you how to control clusters across multiple environments and create a management interface that responds programmatically to your needs.

Ready? Let’s get started!

What is the Cluster API?

Cluster API is a tool for programmatically configuring and deploying Kubernetes clusters on a variety of different infrastructures. It is open source and maintained as a sub-project of Kubernetes; as of this writing it is in version 1.2.4 and is regarded as production-ready.

For example, Cluster API is an upstream component of Mirantis Container Cloud, which uses the API to deliver a powerful infrastructure control plane managed through a single-pane GUI, while integrating management for components like Ceph storage.

What is the purpose of Cluster API?

Cluster API begins from a simple premise:

What if we could apply the “Kubernetes way of doing things” — that is, declarative configuration and deployment via API — to the provisioning and management of clusters themselves. What if spinning up a cluster to your specifications was functionally no different than deploying a Kubernetes resource like a Pod?

The answer to that “What if?” is that clusters would be much easier to configure and deploy programmatically and at scale — the same as any other Kubernetes resource. And that’s very useful for operators who want to manage lifecycles for hundreds or even thousands of clusters: whether those are edge clusters at retail locations, individual developer clusters, or clusters dedicated to sensitive, isolated workloads.

How does Cluster API work?

Cluster API not only gives us a way to manage Kubernetes clusters; it also runs on Kubernetes. Ultimately, Cluster API is simply a set of components that we install on a cluster and interact with in Kube-like style, through Kube-like tools.

This raises a sort of chicken-and-egg problem: as operators, we will need to provide a first cluster from which all of our other clusters will be initiated. This initial cluster may be temporary — a bootstrap cluster that will be discarded later — or it may serve an important and ongoing role as a management cluster.

In the conventions of Cluster API, the management cluster is one of two essential cluster types:

• Management clusters: These clusters are responsible for the creation and oversight of other clusters through Cluster API. They are your agents for infrastructure management, and in that respect are a bit of an oddity — they’re probably not running application workloads like a normal Kubernetes cluster, but instead focusing entirely on provisioning, monitoring, and managing other clusters.
• Workload clusters: These are the clusters that will actually handle application workloads for your users — the business-as-usual clusters that do exactly what you would expect a Kubernetes cluster to do, running microservices and handling requests.

This framework should sound familiar — it is, of course, reminiscent of the manager and worker nodes within a given Kubernetes cluster.

In any case, to get started with Cluster API, we need only furnish a few prerequisites:

• An initial cluster: This is the starting cluster we will need to create the rest, and it can live in a lot of different places — on any number of clouds or on our local machine.
• kubectl: You’ll need the kubectl CLI installed on your workstation and all set to control your initial cluster.
• A provider: “Provider” is Cluster API’s abstraction for the infrastructure on which your newly created clusters will run. That might mean the big public cloud providers such as AWS, Azure, and Google Cloud, but it could also refer to more specialized providers such as Equinix Metal or DigitalOcean, or a host infrastructure such as OpenStack.

With these pieces in place, we will be able to configure and create clusters at scale entirely programmatically.

Indeed, if you were so inclined, you could use the clusters you provision with Cluster API to run Cluster API components and provision yet another cluster. You can see, then, why the Cluster API logo is three turtles with Kubernetes logos on their shells: it’s Kubernetes turtles all the way down.

Deploying a cluster with Cluster API

In this walkthrough, we will use a local development cluster as our management cluster, and we will deploy a workload cluster to AWS.

You can launch a local development cluster with Lens Desktop Kubernetes (a feature of Lens Pro), the Lens for Docker Desktop extension, Minikube, or another dev cluster implementation of your choice. Go ahead and start your local cluster now.

Next we’ll install the clusterctl command line tool. Download the binary from GitHub (making sure to grab the correct binary for your system):

% curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.4/clusterctl-darwin-amd64 -o clusterctl

From the same directory, use chmod to modify permissions so the binary is executable.

% chmod +x ./clusterctl

Put the clusterctl binary in your PATH.

% sudo mv ./clusterctl /usr/local/bin/clusterctl

The clusterctl CLI tool should be installed now. In a moment, we’re going to use it to initialize our local Kubernetes cluster as a management cluster. But before we do, we need to do some configuration for the provider to which we intend to deploy workload clusters — in this case, AWS.

Initializing the management cluster for AWS

We could configure for multiple providers, at this point, and clusterctl would enable us to manage all of them — enabling us to create and manage hybrid/multi-cloud architecture from a single interface. But for this primer, we’ll keep things simple and stick to a single provider.

Now, in order to do our AWS configuration, we’re going to download another CLI tool called clusterawsadm that will help us generate a CloudFormation stack with appropriate IAM resources. (CloudFormation is an AWS tool for infrastructure-as-code automation.) We’ll install this tool exactly the same way we did with clusterctl — again, verifying that you have the right binary for your system:

% curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v1.5.0/clusterawsadm-darwin-amd64 -o clusterawsadm
% chmod +x clusterawsadm
% sudo mv clusterawsadm /usr/local/bin

Note: The rest of this walkthrough uses AWS credentials and, once a workload cluster is deployed, will incur some costs if you follow along — proceed advisedly! Make sure to use the credentials for an IAM user with policies that make sense for you.

The clusterawsadm tool draws on a set of environment variables in order to run its configuration. Now we’ll use the export command to define those environment variables:

% export AWS_REGION=us-east-1
% export AWS_ACCESS_KEY_ID=<Your access key>
% export AWS_SECRET_ACCESS_KEY=<Your secret access key>
% export AWS_SESSION_TOKEN=<Session token>

Note: the session token is only necessary if you’re using multi-factor authentication.

With those environment variables defined, we can use clusterawsadm to generate our CloudFormation stack:

% clusterawsadm bootstrap iam create-cloudformation-stack

The system will return…

Attempting to create AWS CloudFormation stack cluster-api-provider-aws-sigs-k8s-io

…and it might take a moment. When it’s done, you’ll get a completion notification for various resources that looks like this:

AWS::IAM::Role            |nodes.cluster-api-provider-aws.sigs.k8s.io                                            |CREATE_COMPLETE

Now we’ll create another environment variable — this one with our newly-created credentials, base64-encoded.

% export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

And finally, three last environment variables which will be required to specify details about our new workload clusters.
For the SSH key, you’ll need to actually have a key pair named default — alternatively, change the value here to an existing key pair. If you need to create an SSH key pair, you can do that on the Key pairs page found under Network & Security in the EC2 menu.

​​% export AWS_SSH_KEY_NAME=default
% export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
% export AWS_NODE_MACHINE_TYPE=t3.large

That’s it for our AWS-specific configuration. Now we’re ready to initialize our local cluster as a management cluster — with resources for deploying clusters to the provider AWS.

% clusterctl init --infrastructure aws

Your output should look something like this:

Fetching providers
Installing cert-manager Version="v1.9.1"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.2.4" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.2.4" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.2.4" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-aws" Version="v1.5.0" TargetNamespace="capa-system"
 
Your management cluster has been initialized successfully!

Before we move on, let’s take a moment to look at what we’ve done. If we take a look at our namespaces, we’ll see that we have several new namespaces in our management cluster dedicated to Cluster API tooling:

% kubectl get ns
NAME                                STATUS   AGE
capd-system                         Active   43h
capi-kubeadm-bootstrap-system       Active   43h
capi-kubeadm-control-plane-system   Active   43h
capi-system                         Active   43h
cert-manager                        Active   43h

We’ve also got quite a few new Custom Resource Definitions (CRDs). Here are a select handful:

% kubectl get crds
NAME                                                             CREATED AT
clusterclasses.cluster.x-k8s.io                                  2022-10-17T19:19:36Z
clusterissuers.cert-manager.io                                   2022-10-17T19:18:31Z
clusterresourcesetbindings.addons.cluster.x-k8s.io               2022-10-17T19:19:36Z
clusterresourcesets.addons.cluster.x-k8s.io                      2022-10-17T19:19:36Z
clusters.cluster.x-k8s.io                                        2022-10-17T19:19:36Z
machinedeployments.cluster.x-k8s.io                              2022-10-17T19:19:37Z
machinehealthchecks.cluster.x-k8s.io                             2022-10-17T19:19:37Z
machinepools.cluster.x-k8s.io                                    2022-10-17T19:19:37Z
machines.cluster.x-k8s.io                                        2022-10-17T19:19:38Z
machinesets.cluster.x-k8s.io                                     2022-10-17T19:19:38Z
providers.clusterctl.cluster.x-k8s.io                            2022-10-17T19:18:17Z

These CRDs are really the core mechanism of Cluster API, enabling the management cluster to handle, say, clusters and machines as Kubernetes resources.

Creating and managing a workload cluster

Now we’re ready to generate a workload cluster! We’ll use clusterctl’s generate cluster command to create a YAML manifest in our working directory — which we can use, in turn, to deploy our cluster.

% clusterctl generate cluster test-cluster --infrastructure aws --kubernetes-version v1.25.0 --control-plane-machine-count=3 --worker-machine-count=3 > test-cluster.yaml

Note that we’re provisioning three worker machines for a reason: this is the requirement for minimum availability.

Let’s take a look at the YAML manifest we generated. Open the file with the code or text editor of your choice. You’ll see multiple manifests in this one file, defining several different kinds of resources:

• Cluster
• AWSCluster
• KubeadmControlPlane
• AWSMachineTemplate
• MachineDeployment
• KubeadmConfigTemplate

Here are several of those new Kubernetes resource types that we added through CRDs. Look through the manifests and observe how many of the details we’ve specified so far are rendered in the YAML. I’ll zoom in on just one of those manifests here:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSCluster
metadata:
 name: test-cluster
 namespace: default
spec:
 region: us-east-1
 sshKeyName: default

Here we have an abstraction for an AWS-hosted cluster. It includes details like region and sshKeyName that we specified earlier through environment variables. And it’s linked to the resource instance of our more general Cluster object by name: test-cluster.

Let’s provision our workload cluster. From the directory where test-cluster.yaml is stored:

% kubectl apply -f test-cluster.yaml

Your output should look something like this:

cluster.cluster.x-k8s.io/test-cluster created
awscluster.infrastructure.cluster.x-k8s.io/test-cluster created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/test-cluster-control-plane created
awsmachinetemplate.infrastructure.cluster.x-k8s.io/test-cluster-control-plane created
machinedeployment.cluster.x-k8s.io/test-cluster-md-0 created
awsmachinetemplate.infrastructure.cluster.x-k8s.io/test-cluster-md-0 created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/test-cluster-md-0 created

Now we can use kubectl to manage those custom resource types. We can run…

% kubectl get clusters
NAME           PHASE         AGE  
test-cluster   Provisioned   30s

…and this gives us a view of all our clusters managed by Cluster API. We can get more granular and view all of our machines:

% kubectl get machines
NAME                               CLUSTER          
test-cluster-control-plane-9cz2s   test-cluster

Or perhaps we only wish to see our workload clusters on AWS:

% kubectl get awsclusters
NAME           CLUSTER        READY
test-cluster   test-cluster   true

The clusterctl CLI tool can help us gain even more insight. Run:

% clusterctl describe cluster test-cluster

The output should look like this:

NAME                                                 READY  SEVERITY  REASON  SINCE                SINCE  MESSAGE                                                                               
Cluster/test-cluster                                 True                     28s                                                                                                               
├─ClusterInfrastructure                              True                     5m                                                                                                                
├─ControlPlane                                       True                     28s                                                                                           
│ └─3 Machines...                                    True                     2m                             
└─Workers                                                                                                                                                                                                       
  └─MachineDeployment/test-cluster-md-0              False  Warning           10m                             
    └─3 Machines...                                  True                     2m11s

(I’ve omitted some detail for legibility here — the real output will give you names for all your nodes and more informative warnings for unready machines.)

Your results may not show the control plane as ready, but give it a couple of minutes and it should get there. The worker MachineDeployment will not become ready, however, because our new cluster is missing a final ingredient: a Container Network Interface (CNI) plugin to handle cluster networking.

Once a control plane node is ready, we can communicate with it — and our first order of business is to grab the kubeconfig for the worker cluster. The clusterctl CLI makes this easy:

% clusterctl get kubeconfig test-cluster > test-cluster.kubeconfig

The command above will download a file called test-cluster.kubeconfig to our current working directory. Now we can use that kubeconfig to manage our new worker cluster, and the first thing we will do is add the open source Calico CNI plugin, which provides network connectivity between workloads:

% kubectl --kubeconfig=./test-cluster.kubeconfig \
  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml

Here, we’re telling kubectl to use the kubeconfig in our working directory and to apply to that cluster — our new workload cluster — the manifest for Calico, which we’re grabbing from GitHub.

If you check out the Calico YAML, or simply skim the console output, you’ll see that we’re adding a pile of Custom Resource Definitions, a Controller, and several other resources:

poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created

Now that we’ve added Calico, the worker nodes in our workload cluster should quickly become ready:

% clusterctl describe cluster test-cluster

The output for a fully ready cluster:

NAME                                                   READY  SEVERITY  REASON  SINCE                                                                                
Cluster/test-cluster                                   True                     4m48s                                                                                         
├─ClusterInfrastructure                                True                     9m20s                                                                                         
├─ControlPlane                                         True                     4m48s                                                                                         
│ └─3 Machines...                                      True                     6m20s       
└─Workers                                                                                                                                                                                  
  └─MachineDeployment/test-cluster-md-0                True                     78s                                                                                           
    └─3 Machines...                                    True                     6m31s

At this point, we’re fully operational, and we can use the workload cluster’s kubeconfig to do whatever we need to do on the cluster.

For ongoing interaction with the cluster, we could continue to specify the kubeconfig with kubectl, or merge it with our local kubeconfig and switch contexts, but one of the easier ways to hop between cluster contexts is using Lens.

After downloading and starting Lens, simply click on the plus sign in the lower-right corner and select Sync kubeconfig.

Now you can easily switch between clusters and manage your workload clusters, install charts with a few clicks via Lens’ Helm interface, or just as easily set up a Prometheus monitoring stack.

Here we can see all of our nodes organized for easy monitoring and management:

When we’re done with the workload cluster, the clean-up is mercifully easy:

% kubectl delete cluster test-cluster
cluster.cluster.x-k8s.io "test-cluster" deleted

That brings us to the end of this introductory primer, but it’s by no means the end of what you can do with Cluster API. In future walkthroughs, we’ll show you how to programmatically control clusters in other environments, and how to create a hybrid environment that is programmatically responsive to your needs.

How to Use Cluster API to Programmatically Configure and Deploy Kubernetes Clusters was originally published in Mirantis on Medium, where people are continuing the conversation by highlighting and responding to this story.

Tech folks love to learn. This is a good thing. Climbing up steep learning curves is an absolute requirement for success in IT, DevOps, software development, and allied fields.

Of course, that’s true in other parts of the organization, as well. Business, Sales, and Marketing leaders don’t reach the C-suite unless they’re constantly learning. Once there, their impact strongly reflects their willingness to keep on digging deep.

But there are differences between Tech and Managerial approaches to learning.

In my experience, managerial learning tends to focus on business relevant issues. Leaders study how to communicate differentiating benefits better; how to build category dominance. They read about how to quantify business progress and improve business processes. The best surround themselves with smart and trusted counselors inside and outside the orgs they serve, and depend on these coaches to keep them focused, challenging them with new ideas and directions for learning.

Even when executives dip into theory, they tend to concentrate on what’s business-relevant: that is, unpacking business culture, or understanding macro-scale phenomena that influence business growth and survival. lt was a CEO, for example, who first pointed me in the direction of Nicholas Taleb’s explorations of disruption and resilience (“Black Swan,” “Antifragile,” and so on), and Daniel Kahneman’s work on modes of thinking (notably “Thinking Fast and Slow”) — all material I’ve used, over the past decade, to build better clouds and applications.

Technical learning should be results-focused, too. And sometimes it is. In fact, what’s arguably the most powerful mechanism for results-focus, category creation, business growth, and wealth creation of the late 20th/early 21st centuries emerged from tech: “commoditize your complements.” That’s shorthand for two things:

• “Focus on what’s truly business-relevant and open up to let others do everything else.”
• “Help make the things that you don’t do, and upon which you depend, as cheap as possible.”

The actual technique is much older; as old as the idea of standardized parts and supply chains (which dates back to the dawn of mechanized manufacturing). More recent poster children for “commoditize your complements” were IBM and Microsoft, whose open hardware architecture and non-exclusive OS licenses ushered in modern computing, with its vast ecosystem of competing software and hardware makers.

Software engineer and StackOverflow co-founder Joel Spolsky’s famous 2002 blog popularized the phrase, and Spolsky also used it to explain phenomena like “Why is IBM paying people to contribute to open source projects?” Answer: Because IBM was, at the time, repositioning as an IT consultancy, so their complements were software components. Ergo, they invested in helping other folks build and provide this software at low- or no cost.

These days, you see “commoditize your complements” at work everywhere in tech — not least in the flood of “rent, don’t own” SaaS solutions and cloud technologies we all use every day.

But failure modes remain

And yet … tech orgs often still fall into failure modes like “not manufactured here disease,” where engineers feel they can’t trust “outsiders” to deliver parts of a product or service.

This can lead to misallocation of time, effort, headspace, and headcount — as engineers study and train, and orgs hire and tool up to design, build, and manage parts of a solution stack (like OpenStack and/or Kubernetes) that are technically required for many use-cases, but (in all their details) not business-essential.

Especially with respect to complex cloud platforms, the line between “technically required” and “business essential” can seem blurry, even (maybe especially) to sophisticated experts.

Read the rest of this article on the Mirantis blog

Technical Learning Curves: Not All Have Equal Value was originally published in Mirantis on Medium, where people are continuing the conversation by highlighting and responding to this story.

A recent SlashData report with the Cloud Native Computing Foundation found a 67% year-over-year increase in Kubernetes adoption–but it’s clear that security practices aren’t keeping pace. For example, organizations such as Tesla have experienced cryptojacking attacks on their Kubernetes clusters because of misconfigured components. In June 2022, security analysts estimated that more than 900,000 Kubernetes clusters are improperly exposed to the open Internet, creating an unnecessary attack vector and opportunity for data leakage.Kubernetes enables organizations to achieve High Availability and resilience, but at the cost of complexity that can lead to serious vulnerabilities if a team doesn’t have experience standing up a cluster.

Beyond the inherent complexity, cloud native computing requires a dramatic change in thinking about security. A cloud native environment is fundamentally different from traditional data centers, in that it is usually distributed and designed to run many containerized workloads. The castle-and-moat thinking that predominated in classical data center security — treating the network perimeter as a moat and the network as a secure stronghold — simply doesn’t cut it anymore (if it ever did).

In fact, operating with those metaphors is apt to put your systems at risk, even in a traditional data center. Fortunately, cloud native security means we must act as if there is no perimeter, proceeding with a presumption of “zero trust.” That means monitoring and validating everything within your system–not only users and devices, but software components, including the dependencies of dependencies within every containerized microservice. A minimum viable cloud native security strategy means creating a secure software supply chain.

That may sound daunting, but there is good news. Kubernetes is a highly resource-aware and extensible system, and the Kubernetes ecosystem includes a range of components to support a secure software supply chain and other security measures. This can help you incorporate four key elements of Kubernetes security:

1. Private container registries (and scanning)
2. RBAC and secure CI/CD
3. Software Bill of Materials (SBOM)
4. Disaster recovery measures

Private container registries (and scanning)

In Kubernetes, workloads are based on container images. Additionally, software is often built on the foundations of existing container images for common components like programming languages or web frameworks.

In other words, container images are essential building blocks. But like any other element in your system, they can’t be trusted. Original container images should be signed and verified. Existing container images should never be drawn from public registries like Docker Hub, which are vulnerable to tampering and malware–one discovery firm found over 4 million Docker Hub images with vulnerabilities open to exploitation.

Instead, organizations should rely on private container registries with robust image signing and registry scanning capabilities. A private registry enables you to use Role-Based Access Control (RBAC) to determine who can use and update container images. It helps you to validate the software that flows out of the registry–and if you have a good registry scanner, you can run regular scans against a CVE database to ensure that there are no known vulnerabilities lurking in your images.

There are a number of options for private registries available. Harbor is an open source private registry option in the CNCF ecosystem, while Mirantis Secure Registry provides a supported option for enterprises, with RBAC and registry scanning.

RBAC and secure CI/CD

Role-based Access Control (RBAC) applies to more than your private container registry. It should be enforced across a Kubernetes cluster. Fortunately, Kubernetes makes that relatively easy, enabling you to define specific roles using the principle of least privilege — meaning any user should have only as much system privilege as they require for their function. Kubernetes namespaces give you a way to enforce separation based on team or other variables, while still permitting controlled access to shared resources.

Of course, workflows are a bigger issue than individual users and the resources to which they have access. It is a good idea to implement policy-as-code to enforce security policies as part of a security-conscious CI/CD process. There are a number of powerful tools to help you manage workflows with Kubernetes, but it’s important that you evaluate them for security concerns. For example, Argo is a popular suite of open source tools in the CNCF ecosystem that help to automate, observe, and manage deployment, but it poses serious security considerations that users need to evaluate carefully, implementing mitigations or workarounds as appropriate.

A thoughtful approach to CI/CD can both avoid vulnerabilities and help to harden security. Tools such as Mirantis Secure Registry can support workflow automation with features such as automatic image promotion based on security validation, reducing the opportunity for human error.

Read the rest of this article at the Mirantis blog.

No more moats: Hardening cloud native security was originally published in Mirantis on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the biggest challenges for implementing cloud native technologies is learning the fundamentals — especially when you need to fit your learning into a busy schedule. In this series, we’ll break down core cloud native concepts, challenges, and best practices into short, manageable exercises and explainers, so you can learn five minutes at a time.

In the last lesson, we investigated the architecture of a Kubernetes cluster, including the major components of the control plane and worker nodes, as well as common architectural patterns for a cluster. Today, we’ll start to break down the core abstractions that organize activity on the cluster — starting with Pods.

Table of Contents

1. What is Kubernetes?
2. Setting Up a Kubernetes Learning Environment
3. The Architecture of a Kubernetes Cluster
4. Introducing Pods (and other Kubernetes objects) ← You are here

These lessons assume a basic understanding of containers. If you need to get up to speed on Docker and containerization, download our free ebook, Learn Containers 5 Minutes at a Time. This concise, hands-on primer explains:

- The key concepts underlying containers — and how to use core tools like image registries

- Fundamentals of container networking that are essential for understanding container orchestrators like Kubernetes

- How to deploy containerized apps in single-container and multi-container configurations

What are Kubernetes objects?

Kubernetes defines entities on the cluster as objects. The term functions the same way here as in most computer science contexts: Kubernetes objects are instances of a class with certain attributes, and they give us a model for abstraction. In Kubernetes, we’re using objects to define entities in our cluster — and more specifically, the desired state and status of those entities. Kubernetes is all about intent, and objects are the model we use to express our intent.

So far we’ve been a little vague — what kind of “entities” are we talking about here, and what might we intend for them? Well, let’s have a look. We can use kubectl to retrieve a complete list of abstractions that are manageable through the Kubernetes API. Start Minikube and then enter:

% kubectl api-resources

You should see a list something like this:

NAME                             SHORTNAMES   APIVERSION
bindings                                       v1
componentstatuses                 cs           v1
configmaps                        cm           v1
endpoints                         ep           v1
events                            ev           v1
limitranges                       limits       v1
namespaces                        ns           v1
nodes                             no           v1
persistentvolumeclaims            pvc          v1
persistentvolumes                 pv           v1
pods                              po           v1
podtemplates                                   v1
replicationcontrollers            rc           v1
resourcequotas                    quota        v1
secrets                                        v1
serviceaccounts                   sa           v1
services                          svc          v1
mutatingwebhookconfigurations                  admissionregistration.k8s.io/v1
validatingwebhookconfigurations                admissionregistration.k8s.io/v1
customresourcedefinitions         crd,crds     apiextensions.k8s.io/v1
apiservices                                    apiregistration.k8s.io/v1
controllerrevisions                            apps/v1
daemonsets                        ds           apps/v1
deployments                       deploy       apps/v1
replicasets                       rs           apps/v1
statefulsets                      sts          apps/v1
tokenreviews                                   authentication.k8s.io/v1
localsubjectaccessreviews                      authorization.k8s.io/v1
selfsubjectaccessreviews                       authorization.k8s.io/v1
selfsubjectrulesreviews                        authorization.k8s.io/v1
subjectaccessreviews                           authorization.k8s.io/v1
horizontalpodautoscalers          hpa          autoscaling/v2
cronjobs                          cj           batch/v1
jobs                                           batch/v1
certificatesigningrequests        csr          certificates.k8s.io/v1
leases                                         coordination.k8s.io/v1
endpointslices                                 discovery.k8s.io/v1
events                            ev           events.k8s.io/v1
flowschemas                                    flowcontrol.apiserver.k8s.io/v1beta2
prioritylevelconfigurations                    flowcontrol.apiserver.k8s.io/v1beta2
nodes                                          metrics.k8s.io/v1beta1
pods                                           metrics.k8s.io/v1beta1
ingressclasses                                 networking.k8s.io/v1
ingresses                         ing          networking.k8s.io/v1
networkpolicies                   netpol       networking.k8s.io/v1
runtimeclasses                                 node.k8s.io/v1
poddisruptionbudgets              pdb          policy/v1
podsecuritypolicies               psp          policy/v1beta1
clusterrolebindings                            rbac.authorization.k8s.io/v1
clusterroles                                   rbac.authorization.k8s.io/v1
rolebindings                                   rbac.authorization.k8s.io/v1
roles                                          rbac.authorization.k8s.io/v1
priorityclasses                   pc           scheduling.k8s.io/v1
csidrivers                                     storage.k8s.io/v1
csinodes                                       storage.k8s.io/v1
csistoragecapacities                           storage.k8s.io/v1beta1
storageclasses                    sc           storage.k8s.io/v1
volumeattachments                              storage.k8s.io/v1

Read through the complete list; some concepts like “nodes’’ should be familiar already. (We’ve removed the NAMESPACED and KIND columns for readability, but those are informative — review them, too!)

Kubernetes uses these abstractions to record our intent for system resources in a way that the API server can use in communicating with other components. A resource might have to do with workloads, services, policies, or another area of the system. These are essentially the classes that Kubernetes objects instantiate, and objects give us a common model for all of these resources.

In this series, we’ll focus on the most essential Kubernetes objects for developers. Let’s start with the elementary particle of Kubernetes: the Pod.

What is a Pod?

We can ask Kubernetes itself. In the terminal, enter:

% kubectl explain pod

You should get the following answer:

KIND:     Pod
VERSION:  v1
DESCRIPTION:
Pod is a collection of containers that can run on a host. This resource is created by clients and scheduled onto hosts.
…

So what does this mean? What is a Pod? A Pod is a unit of one or more containers that share a network namespace and storage volumes while running on a Kubernetes node. It is also Kubernetes’ most granular unit of scheduling for a containerized workload.

Up until now, we’ve used the general term “workload” to refer to the containerized processes that Kubernetes is orchestrating. As we’ve seen previously, the system assigns these workloads to worker nodes, the physical or virtual machines that are part of the cluster. These are the “hosts” in the definition above. But Kubernetes adds a conceptual layer separating the node from the container, and that’s the Pod.

Each Pod is assigned a unique IP address, so containers within the Pod can communicate with one another over localhost, and Pods can communicate with one another at fixed addresses.

These two ideas — the abstraction of the Pod, and each Pod having its own IP address — are fundamental, defining concepts for Kubernetes. They extend all the way back to the Borg project at Google, and they distinguish Kubernetes from other container orchestrators like Swarm. It is difficult to overstate the importance of the Pod; it contributes to the high scalability of Kubernetes, and it informs many of the networking patterns we will encounter going forward.

We’ve said that a Pod consists of “one or more containers.” The conventional wisdom on containers-per-Pod is that a given Pod should include only one container if possible, and multiple containers only when they are tightly coupled. Sometimes, your Kubernetes configuration may necessitate a multi-container Pod — when using an Istio service mesh, for example, each Pod will include a “sidecar” container in addition to the primary container. But we wouldn’t expect to see more than two or three containers in most Pods. (We’ll talk more about service meshes later, but if you’re ready to dive into the deep end now, you can check out Service Mesh for Mere Mortals.)

Testing the limits

“Sure,” I hear you say, “I understand that we should run no more than a handful of containers per Pod. But I’m a maverick! I want to test the limits. How many containers can we run in a Pod?”

Short answer: There’s no defined limit. As of version 1.23, the Kubernetes docs note that Kubernetes is designed for a maximum of 300,000 total containers. Technically, you could cram all of them into one Pod (assuming you could satisfy the underlying compute and storage requirements). But the docs also tell us that the system is designed for no more than 150,000 Pods, and that ratio is instructive. Operating at the extremities of scale, we would have two containers per Pod.

By separating out the functionality of your containerized services into minimal viable units, you can scale those units — those services or microservices — independently. Say, for example, that you have a Pod representing the core functionality of the auth service on your web app, and you suddenly experience a barrage of sign-ups. The cluster can scale up your auth service as needed without needlessly duplicating other, unrelated elements of the app.

Read the rest of this tutorial on the Mirantis blog.

Introducing Pods (and other Kubernetes objects) was originally published in Mirantis on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the biggest challenges for implementing cloud native technologies is learning the fundamentals — especially when you need to fit your learning into a busy schedule.

In this series, we’ll break down core cloud native concepts, challenges, and best practices into short, manageable exercises and explainers, so you can learn five minutes at a time. These lessons assume a basic familiarity with the Linux command line and a Unix-like operating system — beyond that, you don’t need any special preparation to get started.

Last time, we set up a local Kubernetes environment and learned three different ways to interact with a cluster: the kubectl command line tool, the Kubernetes Web Dashboard, and the Lens IDE. In this lesson, we’ll explore the architecture of a Kubernetes cluster: its core components, how they interact with one another, and how we interact with them.

Table of Contents

1. What is a Container?
2. Creating, Observing, and Deleting Containers
3. Build Image from Dockerfile
4. Using an Image Registry
5. Volumes and Persistent Storage
6. Container Networking and Opening Container Ports
7. Running a Containerized App
8. Multi-Container Apps on User-Defined Networks
9. Docker Compose and Next Steps
10. What is Kubernetes?
11. Setting Up a Kubernetes Learning Environment
12. The Architecture of a Kubernetes Cluster ←You are here

Universal node components

A Kubernetes cluster is made up of nodes. A node is simply a compute resource: it can be either a virtual or physical computer. Last time, we noted that our Minikube cluster is a single-node cluster, meaning that it consists of only one machine–typically a containerized or virtual machine running on our laptop or desktop.

Any node might need to step up and run workloads at any time, so the components required to make that happen are found on every node. Those components include:

• A container runtime: This is the component that actually runs containers on a node. Today, Kubernetes supports a variety of runtimes, including containerd, CRI-O, Mirantis Container Runtime, and others.
• kubelet: A kubelet runs on every node and mediates between the control plane and the node in question, passing instructions to the container runtime — ”Start this workload!” — and then monitoring the status of the job. In order to facilitate the use of different runtimes, the kubelet communicates with the container runtime through a standardized Container Runtime Interface (or CRI).
• kube-proxy: Containerized workloads will need to be able to communicate with one another and the world outside the cluster. In other words, we’re going to have a lot of complicated networking going on! The kube-proxy component is a given node’s on-site network expert, managing network rules and connections on the node.

We can visualize the fundamental components of a node like this:

The Kubernetes control plane

Nodes in Kubernetes clusters are assigned roles. Start your Minikube cluster and then run the following command:

% kubectl get nodes

You should see one node, named minikube, because this is a single-node cluster.

NAME      STATUS   ROLES                  AGE     VERSION
minikube  Ready    control-plane,master   99s     v1.23.1

Note the roles listed here. The minikube node is fulfilling the roles of control-plane and master. What does this mean?

The control plane coordinates the assignment of workloads — meaning containerized applications and services — to other nodes. If there are no other nodes in the cluster, as in our case, then the control plane node has to roll up its sleeves and get its hands dirty.

In a real-world, production-grade Kubernetes deployment, you will almost certainly have many nodes dedicated to running workloads — these are sometimes called worker nodes. Additionally, you will likely have multiple nodes in the control plane. (We’ll discuss this and other architectural patterns shortly.)

In such a scenario, the control plane is dedicated to ensuring that the current state of the cluster matches a pre-defined specification (or spec) — the desired state of the cluster. If our specification says that an instance of NGINX should be running, and the control plane learns that this isn’t the case, it sets out to assign the job and resolve the discrepancy.

A note on terminology.

So what about the master role? At the time of writing (as of release 1.24), it is interchangeable with control-plane. “Master” was the original term and is currently retained to maintain integrations between components that use the label, but the Kubernetes project plans to phase out the use of the word in this context. Eventually, the get nodes command above will only return control-plane for the node role.

This change reflects an ongoing conversation about the suitability of the term “master” in the tech industry and broader culture; many technologies are moving away from the use of “master” terminology just as they moved away from the use of “slave” metaphors several years ago. GitHub, for example, now encourages defining a “main” branch rather than a “master” branch.

The important thing to know when learning Kubernetes is that many third-party resources will discuss master nodes, though the term is no longer used in the Kubernetes documentation. These older resources are referring to nodes with the control-plane role, which we can think of as members of a cluster’s leadership committee. From here on out, we’ll use “control plane” to refer to this role.

Ultimately, the control plane is a group of components that work together to make the actual state of the cluster match its spec. It responds to unexpected events out there in the real world (like a service going down), and it takes onboard any changes to the spec and assigns resources accordingly.

So what are these control plane components?

• kube-apiserver: Last lesson, we said that every interaction with the cluster from tools like kubectl or Lens is mediated by the Kubernetes API. The API server exposes that API. The API server is the hub at the center of activity on the cluster, and the only control plane component that communicates directly with nodes running workloads. It also mediates communications between the various elements of the control plane.
• kube-scheduler: Put simply, the scheduler assigns work. When we tell the API server that we want to run a workload, the API server passes that request to the scheduler. The scheduler, in turn, consults with the API server to check the cluster status and then matches the new job to a node that meets the relevant resource requirements — whether that means hardware specifications or policy constraints.
• etcd: By default, the Kubernetes cluster runs on an assumption of ephemerality: any node and any workload may be spun up or stopped at any moment and replaced with an identical instance. The processes in the cluster are meant to match a specification, but they’re not expected to persist. This is a philosophy of statelessness, and it resembles the operational assumptions of containers themselves. But even if you’re running entirely stateless apps on your cluster, you will need to save and persist configuration and state data for the cluster, and etcd is the default means of doing so. It’s a key-value store that may be located inside the cluster or externally. Only the API server communicates directly with etcd. Note that etcd may be used outside the context of Kubernetes as well; it’s ultimately designed to serve as a distributed data store for relatively small amounts of data.
• kube-controller-manager and cloud-controller-manager are the watchdogs: they constantly monitor the state of the cluster, compare it to the spec, and initiate changes if the state and spec don’t match. The kube-controller-manager focuses on the upkeep of workloads and resources within the cluster, while the cloud-controller-manager is essentially a translator who speaks the language of your cloud provider–dedicated to coordinating resources from your cloud infrastructure. The cloud-controller-manager is an optional component designed for clusters deployed on the cloud.

It’s tempting to think of the control plane as one or more “boss” nodes — that’s essentially true, but it’s a bit of a simplification that obscures important details. For example, etcd may be integrated into a control plane node or hosted externally. And as we’ve seen, a control plane node includes all of the components of a worker node. Typically, components of the control plane are initially installed on a single machine that doesn’t run container workloads…but that’s not a necessary or optimal configuration for every situation.

We can visualize a control plane node like this:

Kubernetes is a distributed system designed in large part for resiliency. But as you can imagine, the control plane could become a single point of failure, especially if it is consolidated on one machine. For this reason, the system offers a “High Availability” configuration, which is the recommended approach for production environments. A High Availability (or HA) cluster will include multiple replicas of the control plane node.

While one of the control plane nodes is the primary cluster manager, the replicas work constantly to remain in sync — and a load balancer can apportion traffic between the various replicas of the API server as needed. Similarly, if duplicates of etcd are sitting on multiple control plane nodes (or an external host), they form a collective that runs its own consensus mechanism to ensure uniformity between the members. These measures ensure that if a control plane node goes down, there are replicas available to pick up the slack.

Kubernetes architectural patterns

Before we move on, let’s pause and take stock:

• A cluster is made up of one or more nodes.
• Any node can run container workloads.
• At least one node also has the components and role to act as the control plane and manage the cluster.
• A cluster configured for High Availability (that is, production-grade resilience) includes replicas of all control plane components.

Now, with all of these takeaways in place, let’s look at the big picture. What are the major architectural patterns for a Kubernetes cluster?

• Single node: This pattern is what we have running right now via Minikube — a single cluster that fulfills the role of the control plane and runs any workloads we wish to run. This node walks and chews bubblegum at the same time: great for learning and development, but not suited for most production uses.
• Multiple nodes with a single node control plane: If we were to connect another node to our cluster as a dedicated worker node, then we would have a cluster with clear separation of responsibilities…but also a single point of failure in the control plane node.
• Multiple nodes with High Availability: When control plane components are replicated across multiple nodes, a cluster can achieve high levels of resiliency through strategic redundancy.

For each of the models above, there are two major sub-variants: you might include the etcd data store on the control plane node(s) — a model called “stacked etcd” — or you might host etcd externally. High Availability configurations are possible with both of these approaches to the data store.

As we’ve established, we’re currently working with a single-node cluster. Let’s get a little more information about that node and see how it correlates with what we’ve learned about Kubernetes architecture:

% kubectl describe node minikube

This command will give us a good deal of information on our node. Read through the output and look for references to the components we’ve discussed. A few sections to note:

…
System Info:
  Machine ID:                 8de776e053e140d6a14c2d2def3d6bb8
  System UUID:                45bc9478-1425-44bd-a441-bfe282c0ce3e
  Boot ID:                    fbdac8bf-80eb-4939-86f1-fcf792c9468f
  Kernel Version:             5.10.76-linuxkit
  OS Image:                   Ubuntu 20.04.2 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.12
  Kubelet Version:            v1.23.1
  Kube-Proxy Version:         v1.23.1
…

My Minikube cluster is running on a container, so the information on my system refers to those details. Here we also find version numbers for our universal node components like the container runtime, kubelet, and kube-proxy.

Further down, we can find information on node resource usage. Note some of the entries in the “Non-terminated Pods” list:

Namespace      Name                               CPU Requests  CPU Limits  Memory Requests
 ---------      ----                               ------------  ----------  ---------------
  …      
  kube-system    etcd-minikube                      100m (2%)     0 (0%)      100Mi (1%)
  kube-system    kube-apiserver-minikube            250m (6%)     0 (0%)      0 (0%)
  kube-system    kube-controller-manager-minikube   200m (5%)     0 (0%)      0 (0%)
  kube-system    kube-proxy-9rjzk                   0 (0%)        0 (0%)      0 (0%)
  kube-system    kube-scheduler-minikube            100m (2%)     0 (0%)      0 (0%)
  …

Here we find a number of familiar faces: our instances of the API server, etcd, the scheduler, and the controller manager — the whole control plane team — as well as kube-proxy.

But wait — what’s a pod? That’s a question for our next lesson. Now that we understand the architecture of a Kubernetes cluster, next time we’ll take a look at the core abstractions that govern activity on the cluster: Kubernetes Objects.

The Architecture of a Kubernetes Cluster was originally published in Mirantis on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the biggest challenges for implementing cloud native technologies is learning the fundamentals — especially when you need to fit your learning into a busy schedule.

In this series, we’ll break down core cloud native concepts, challenges, and best practices into short, manageable exercises and explainers, so you can learn five minutes at a time. These lessons assume a basic familiarity with the Linux command line and a Unix-like operating system — beyond that, you don’t need any special preparation to get started.

In the last lesson, we learned how Kubernetes orchestrates containers, who runs Kubernetes, and why. Today, we’ll set up a cluster on our local machine so we can get some hands-on experience.

Table of Contents

1. What is a Container?
2. Creating, Observing, and Deleting Containers
3. Build Image from Dockerfile
4. Using an Image Registry
5. Volumes and Persistent Storage
6. Container Networking and Opening Container Ports
7. Container Networking and Opening Container Ports
8. Running a Containerized App
9. Multi-Container Apps on User-Defined Networks
10. Docker Compose and Next Steps
11. What is Kubernetes?
12. Setting Up a Kubernetes Learning Environment ←You are here

Choices when Accessing Clusters

As we’ll see in the next few lessons, a Kubernetes cluster can run in a wide variety of configurations according to its use case. For learning purposes, we’re going to start with a single-node cluster on our local machine.

In the “real world,” clusters will typically spread across multiple nodes (or machines), each of which will have dedicated roles like managing the cluster or running workloads. In this case, everything will happen on a single node. That will give us a relatively straightforward development environment in which we can figure out the fundamentals, before progressing to more complex configurations.

The first decision we need to make is which version of Kubernetes to install. Because Kubernetes is an open source system, it has given rise to a variety of alternative distributions with their own particular use-cases, just as the Linux kernel is the foundation of numerous Linux distributions. The k0s (pronounced “kay-zeros”) project, for example, is a distribution designed for maximum resource efficiency, so it might run (and scale) anywhere from Raspberry Pis to powerful servers in public or private clouds. The creators of Kubernetes distributions usually try to maintain full compatibility with “upstream” Kubernetes — the original, baseline project — so that users can utilize the full suite of open source tooling developed by the community.

To get started, we’re going to use a distribution called Minikube, which is designed specifically for learning Kubernetes.

Installing Minikube

Minikube requires about 2GB of RAM to run, and it wants 20GB of disk space, which is the amount it assigns for cluster usage by default. (You can configure it to use as little as 2GB if you’re short on space.) It works on macOS, Linux, or Windows, and depending on its configuration, it runs on a container, a virtual machine, a combination of the two, or bare metal. Most people will be running it through VMs, containers, or both. If you have Docker Desktop or Docker Engine installed (as in our previous unit), then you’re ready to get started.

On an x86–64 machine running macOS with the Homebrew Package Manager, installation is a simple terminal command:

% brew install minikube

Users with other CPU architectures and operating systems will want to consult the official installation instructions to download the version that is right for them.

Once Minikube is installed, make sure Docker is running, and then run the following command in the terminal:

% minikube start

You should see some friendly, emoji-illustrated status updates confirming that the system is running:

👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.23.1 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ... 
    ▪ Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    ▪ Using image kubernetesui/dashboard:v2.3.1
    ▪ Using image k8s.gcr.io/metrics-server/metrics-server:v0.4.2
    ▪ Using image kubernetesui/metrics-scraper:v1.0.7
🌟  Enabled addons: storage-provisioner, metrics-server, default-storageclass, dashboard
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

We can confirm that Kubernetes is running with another command:

% minikube kubectl get nodes

This will list the nodes associated with our Kubernetes cluster. We should get a result that looks something like this:

NAME      STATUS   ROLES                  AGE     VERSION
minikube   Ready    control-plane,master   5m26s   v1.23.1

All right — we have one node, and that makes sense, because we said that this would be a single-node cluster. But what’s kubectl, and why am I using it?

For the answer, we need to break down an important element of Kubernetes development: cluster access.

Read the rest of this tutorial on the Mirantis blog.

Setting Up a Kubernetes Learning Environment was originally published in Mirantis on Medium, where people are continuing the conversation by highlighting and responding to this story.

Note: This blog post was originally published by HelpNetSecutiry. You can view the original post here.

In this article, I’ll go over three fundamental areas you need to consider to protect a Kubernetes cluster:

• Role-based access control (RBAC)
• Open Policy Agent (OPA)
• Network policies

Let’s dive in!

Role-based access control

Let’s consider an organization with three application teams (Blue, Green, and Red). Because these teams are working on different products, they should be given different access to the Kubernetes cluster. For example, the Green and Red teams should not be able to see, access, or delete what the Blue team deployed.

RBAC is a way to control what Kubernetes resources users can access. While RBAC is enabled by default, you must configure it to use it.

There are 5 key elements to RBAC:

• Subjects — Users and processes
• Resources — Objects to which access should be restricted
• Verbs — Sets of operations that can be executed, often referred to as actions
• Roles — An object that connects API Resources with Verbs
• RoleBinding — An object that connects Roles with Subjects

Let’s go back to our organization example and define a policy where only the Blue team can create, delete and list Pods, Deployments, and Services.

First, we create a Role object named `role-blue`, where we define the actions that can be performed on specific Kubernetes resources. In this specific case, the Role enables the actions `create`, `delete`, and `list` to be performed on the resources; `pods`, `deployments`, and `services`.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: blue-role
  namespace: blue
rules:
- apiGroups: ["", "apps"]
  resources: ["pods", "deployments", "services"]
  verbs: ["create", "delete", "list"]

Next, we create a RoleBinding named `blue-rb`. This RoleBinding belongs to `blue-ns`, which links the above-created role `role-blue` with the blue team, named `blue`.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: blue-rb
  namespace: blue-ns
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: blue
roleRef:
  kind: Role
  name: blue-role
  apiGroup: rbac.authorization.k8s.io

Once these resources are applied to the cluster, the users from the `blue` team will have the ability to perform the operations defined in `role-blue`.

Open Policy Agent

Open Policy Agent (OPA) is a general-purpose policy engine that unifies policy enforcement across the stack. Its high-level declarative language provides flexibility to specify policy as code. You can use OPA to enforce policies in Kubernetes, CI/CD pipelines, API gateways, and more. Let’s dive into how to use and implement it in Kubernetes.

The Kubernetes implementation of OPA is called Gatekeeper. It is designed and deployed as an AdmissionController that intercepts requests, processes them, and responds back with a permit or deny response.

When permitted, the object gets deployed on the cluster; otherwise, the request will be rejected and feedback provided to the user. Administrators can define policies that instruct Kubernetes to limit the resources such as memory or CPU that containers or namespaces can consume, approve only containers based on images from specific registries, restrict NodePort service creation, or enforce standard naming.

For example, here is a sample template and constraint policy that allows creation of Pods only when ResourceQuota is configured in a namespace.

constraint.yaml

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResourceQuota
metadata:
  name: namespace-must-have-resourcequota
spec:
  match:
    kinds:
      - apiGroups: ["", "apps"]
        kinds: ["Pod"]
    excludedNamespaces:
      - gatekeeper-system

template.yaml

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredresourcequota
  annotations:
  description: Requires Pods to launch in Namespaces with ResourceQuotas.
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredResourceQuota
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredresourcequota
        violation[{"msg": msg}] {
          input.review.object.kind = ["Pod"]
          requestns := input.review.object.metadata.namespace
          not data.inventory.namespace[requestns]["v1"]["ResourceQuota"]
          msg := sprintf("container <%v> could not be created because the <%v> namespace does not have ResourceQuotas defined", [input.review.object.metadata.name,input.review.object.metadata.namespace])
        }

Network policy

Network policies are very similar to a regular firewall but differ in the sense that they are application-centric. When you define a network policy for an application, Kubernetes automatically applies those rules to the associated containers due to the highly dynamic environment in which containers are constantly created and terminated. Network policies control the flow of traffic to or from these containers.

By default, network traffic to and from Pods is not restricted. A good starting point is to set a deny-all traffic rule, and then only allow necessary traffic.

By default, Kubernetes uses a flat network structure allowing any Pod to communicate with other Pods or Services in the cluster. In a cluster with multiple applications or multi-level applications, defense-in-depth plays a key role in securing the communication layer. Network policies allow us to achieve that.

Here is an `app1-network-policy` that applies the following rules in the `blue` namespace for the Pods with label “role=db”:

• [Ingress] Allows connections from the ipBlock 172.17.0.0/24 on port 6379
• [Ingress] Allows connections from other pods if they are labeled as “role=frontend” and if they belong to the namespace with labels “project=myproject” on port 6379
• [Egress] The pod can communicate with other pods with IP range 10.0.0.0/24 on port 5978.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: app1-network-policy
  namespace: blue
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/24
    - namespaceSelector:
        matchLabels:
          project: myproject
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

Conclusion

With RBAC, OPA, and network policies in place, you can protect your Kubernetes cluster by assuring that contributors have the proper access, that security policies are enforced, and that the network is tightly secured.

About Mirantis

Mirantis helps organizations ship code faster on public and private clouds. The company provides a public cloud experience on any infrastructure from the data center to the edge. With Lens and the Mirantis Cloud-Native Platform, Mirantis empowers a new breed of Kubernetes developers by removing infrastructure and operations complexity and providing one cohesive cloud experience for complete app and DevOps portability, a single pane of glass, and automated full-stack lifecycle management with continuous updates.

For more details, visit https://mirantis.com

3 key elements to protect a Kubernetes cluster was originally published in Mirantis on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the biggest challenges for implementing cloud native technologies is learning the fundamentals — especially when you need to fit your learning into a busy schedule.

In this series, we’ll break down core cloud native concepts, challenges, and best practices into short, manageable exercises and explainers, so you can learn five minutes at a time. These lessons assume a basic familiarity with the Linux command line and a Unix-like operating system — beyond that, you don’t need any special preparation to get started.

Last time, we ran a web app–the open source wiki platform Mediawiki–from a container in a single-container configuration, with a persistent volume and a container port published to the host machine. That gave us a functional development environment, from which we could modify and build on the application.

But cloud native deployments often consist of applications spread across multiple containers. How can we connect the constituent parts of a single application once they’ve been containerized? In this tutorial, we’ll learn how to let containers talk to one another by deploying Mediawiki in a multi-container configuration on a user-defined network.

Table of Contents

1. What is a Container?
2. Creating, Observing, and Deleting Containers
3. Build Image from Dockerfile
4. Using an Image Registry
5. Volumes and Persistent Storage
6. Container Networking and Opening Container Ports
7. Running a Containerized App
8. Multi-Container Apps on User-Defined Networks←You are here

User-defined networks

When containers reside on the default bridge network together, they should in theory be able to communicate with each other by name via Domain Name System (DNS)–but they can’t. Instead, those containerized apps need to know one another’s specific IP addresses to transmit data back and forth.

This is a deliberate restriction on the default bridge network. But why? Well, simply by virtue of being default, the docker0 bridge is apt to be a busy place, full of Docker containers without any necessary relationship to one another — without any need to communicate. That could be a security risk in a system predicated on isolation. So on the default bridge, Docker makes containers jump through a few extra hoops to communicate effectively.

When we have a group of containers that need to communicate, instead of using the default bridge we can place them in their own user-defined network. While this isn’t the only way to let containers communicate, it is the Docker-preferred way of doing things, since this creates a precisely scoped layer of isolation. We can create a user-defined network with the command:

docker network create <network name>

The -d (or --driver) argument for this command specifies a model for the new network: bridge, overlay, or a custom driver option added by the user.

• Bridge networks allow containers within the network — all of which must be on the same Docker daemon host — to communicate with one another while isolating them from other networks.
• Overlay networks allow containers within the network — which may be spread across multiple Docker daemon hosts–to communicate with one another while isolating them from other networks. This driver is used by the container orchestrator Docker Swarm.
• Custom drivers allow for custom network rules.

The bridge driver is the default, so if you don’t specify a driver, Docker will create a bridge network. The bridge driver is what we’ll be using in this lesson.

What about linking?

Another way to name-based communication between containers on the default bridge is container linking, which involves creating manual links between containers using the --link argument. This was once the standard technique for connecting containers, and you’ll still see it used in the docs for many images. But Docker considers this a legacy option, which means that it is not recommended and may be disabled in the future; linking has been superseded by user-defined networks.

Exercise: A multi-container wiki using container networking

You can follow along in this video tutorial:

To follow the rest of the tutorial, read the rest at the Mirantis blog.

Multi-Container Apps on User-Defined Networks was originally published in Mirantis on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the biggest challenges for implementing cloud native technologies is learning the fundamentals — especially when you need to fit your learning into a busy schedule.

In this series, we’ll break down core cloud native concepts, challenges, and best practices into short, manageable exercises and explainers, so you can learn five minutes at a time. These lessons assume a basic familiarity with the Linux command line and a Unix-like operating system — beyond that, you don’t need any special preparation to get started.

In our last lesson, we defined some core concepts in container networking and learned how to publish container ports, opening a container’s services to traffic from the outside world. This time, we’re going to bring together most of what we’ve learned so far to run a containerized web app with persistent volumes and published ports: in this case, a knowledge-sharing wiki platform.

Table of Contents

1. What is a Container?
2. Creating, Observing, and Deleting Containers
3. Build Image from Dockerfile
4. Using an Image Registry
5. Volumes and Persistent Storage
6. Container Networking and Opening Container Ports
7. Running a Containerized App ←You are here

Bringing the pieces together

Wikipedia is built on an open source platform called Mediawiki, and an official Docker image for the application is maintained on Docker Hub. That means it should be easy to get our own wiki running quickly — in less than five minutes, even!

The default architecture for this deployment is very straightforward: a single container for the application, which can either write data to a volume all on its own — the default configuration — or work in tandem with a second container running a production-grade database — the option you would want to use if you were taking this wiki to production, pushing it live for real-world use.

In this lesson, we’re going to start with a single-container configuration — but we won’t stop there. Next time, we’ll also explore how to link containers for a more production-ready configuration.

But we’re getting ahead of ourselves. For now, let’s focus on getting our wiki up and running!

Exercise: A Wiki as a Containerized App

Let’s start by creating a new volume that will store the persistent data for our wiki:

docker volume create wiki-data

Now we’re going to run a Docker container based on the official Mediawiki image. We’ll call our new container solo-wiki since it’s going to run in a single-container configuration. We’ll also publish a port so we can access the application on our host machine, and add our volume for persistent storage.

docker run --name solo-wiki -v wiki-data:/wiki-data -p 8000:80 -d mediawiki

Now when we visit localhost:8000 in our web browser, we should see a starting page for our wiki. But the app still needs to be set up. On the starting page, click “Complete the installation.”

To follow the rest of the tutorial, read the full post at the Mirantis blog.

How to Run a Containerized App with Docker was originally published in Mirantis on Medium, where people are continuing the conversation by highlighting and responding to this story.