Introduction to GKE On-Prem
Hey everyone, I hope you all are doing well. In this article, I wanted to give you an overview of GKE On-prem a.k.a Anthos on VMware.
Subscribe to my channel, The Cloud Pilot
Anthos clusters on VMware, also known as GKE on-prem, is a software that brings Google Kubernetes Engine (GKE) to on-premises data centers. With Anthos clusters on VMware, you can create, manage, and upgrade Kubernetes clusters in your on-premises environment.
It will also give users:
- Ability to manage and maintain Kubernetes clusters running in Google Cloud or inside their own data centers.
- Ability to monitor their clusters using Cloud Operations.
- Ability to deploy Marketplace Applications on the Kubernetes clusters running On-Prem.
[Learning about Kubernetes? Read Why retailers choose to build on Kubernetes on the Google Cloud Blog.]
GKE On-Prem Architecture
There are 3 things that GKE On-Prem needs in your data center to function properly.
- VMware vSphere cluster running vSphere 6.5 (It’s the only supported version with GKE On-Prem version 1.0)
- F5 BIG-IP LTM to provide Loadbalancer type support for the Application services defined in your K8s clusters.
- Admin Workstation: The admin workstation is a bundle for all the different utilities needed to deploy GKE On-Prem clusters that can be deployed on a laptop that runs VirtualBox or a different management cluster.
VMware vSphere provides two main components to make the On-Prem experience better for users.
The vSphere Cluster API and vSphere Cloud Provider plugin to Kubernetes gives users the ability to seamlessly deploy virtual machines to be part of the admin and user Kubernetes clusters and the ability to provide persistent storage to the application pods or containers running on top of your Kubernetes clusters respectively.
The vSphere Cluster API adds the following constructs to enable easier deployments for the underlying virtual machines:
Machine
Machines are VMs running on top of your vSphere Infrastructure.
They are equivalent to Kubernetes pods. All the virtual machine level details like the CPU, Memory, Storage, and Base OS requirements can be defined using the Machine Definition.
Machine Set
Similar to Replica Sets where you define the desired number of Machines needed to support your K8s cluster and the Machine Set controller will run a reconciliation loop to ensure that the desired state matches the current state.
Machine Deployment
Similar to Deployments, you can perform rolling updates to the underlying base Operating System for your Machines (VMs) using the concept of Machine Deployments.
Machine Class
Similar to the Storage Class definition, this enables you to specify all the environment-specific configuration parameters in one place, rather than you having to define it with every Machine Definition.
Using vSphere Cluster API, you can deploy, manage and scale your virtual infrastructure similar to how Kubernetes deploys, manages and scales your containerized applications.
vSphere Cloud Provider
vSphere Cloud Provider plugin is built into upstream K8s, so you will not have to worry about deploying any additional CSI plugins to provide persistent storage to containers. vSphere Cloud Provider plugin enables users to consume storage from any data store that is mounted on vSphere (NFS, VMFS, or vSAN). When you use a vSAN Datastore, you can extend all the Storage Policy Based Management (SPBM) capabilities to the storage class definition inside Kubernetes. Persistent volumes defined by K8s are created as VMDK files inside the vsanDatastore and then mounted to the virtual machines that run your pods.
Deploying GKE On-prem
When you deploy your initial GKE On-Prem cluster using the Admin Workstation, you deploy an admin and a single-user cluster.
- The Admin Cluster is responsible for running all the control plane components for GKE On-Prem.
- When you want to deploy additional user clusters, you use gkectl to talk to the Admin cluster, which will then talk to the vCenter APIs to deploy additional virtual machines for the new User Cluster.
- User Cluster is where you deploy your applications.
- The Master nodes for the User Cluster also participate in the Admin cluster.
When you buy Anthos licenses, you pay only for the vCPUs consumed by the VMs in the User Cluster. This architecture is referred to as Kubeception.
Managing all your On-Prem clusters from the Google Cloud Console
Google Cloud provides a service called Connect that helps you connect your On-Prem clusters back to Google Cloud.
It allows you to manage your clusters, deploy applications, workloads and apply autoscaling policies to the pods running on-prem.
The Connect Agent needs a service account key that has the correct permissions to connect to Google Cloud, and it runs as a K8s Deployment inside your cluster. Once connected, it will share information about the cluster, workloads and any metadata with the control plane running in Google Cloud.
You don’t need additional VPN or Cloud Interconnect connections for GKE On-Prem unless you want to access any additional GCP services.
That’s a high-level overview of GKE On-prem (Anthos on VMware). I hope this helps you. Thank you for reading. See you soon!
