Why KOPS Updating Cluster Is Awesome
Here is yet another reason why I think the Kops tool for creating and managing a Kubernetes’ cluster lifecycle is so great.
I needed to do three things to the cluster:
1) Set the kubelet timeout to something higher than the default 2 minutes. This cluster is in ap-southeast-1 and pulling docker images is really really slow from there. Some time it takes longer than 2 minutes and the Kubelet will time out.
Well luckily kops exposes all of the Kubelets flags so you can tune it however you want. Here is the Kubelet documentation showing me I need to use the --runtime-request-timeout flag. This maps to the kops doc on what this parameter should be in my kops configs:
runtimeRequestTimeout: 10m0sThis is documentation working to its finest here! Everything lined up exactly how I would expect it to.
2) Change the instance group size on a few instance groups
3) I needed to add a new kops instance group
So i make my changes and run the command to check what kops would change without actually applying it to the cluster:
kops --name prod.ap-southeast-1.k8s.managedkube.com update clusterIt returns back to me this very handy and life saving diff on what it will exactly do.
1) I can see here that it will add these instance groups:
Will create resources:
AutoscalingGroup/ondemand-zone-a-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com
MinSize 1
MaxSize 5
Subnets [name:worker-zone-a.prod.ap-southeast-1.k8s.managedkube.com id:subnet-58c7a93f]
Tags {Project: cloud, k8s.io/cluster-autoscaler/node-template/label/k8s.info/hasPublicIP: false, k8s.io/cluster-autoscaler/node-template/label/k8s.info/instanceType: m4.large, k8s.io/cluster-autoscaler/node-template/label/k8s.info/application: infrastructure, k8s.io/role/node: 1, Name: ondemand-zone-a-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com, k8s.io/cluster-autoscaler/node-template/label/prod.ap-southeast-1.k8s.managedkube.com/role: scale-zero, Owner: kubernetes, k8s.io/cluster-autoscaler/node-template/label/k8s.info/isSpot: false, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: ondemand-zone-a, KubernetesCluster: prod.ap-southeast-1.k8s.managedkube.com, Purpose: kubernetes-ondemand-node, CostCenter: kubernetes-saas}
Granularity 1Minute
Metrics [GroupDesiredCapacity, GroupInServiceInstances, GroupMaxSize, GroupMinSize, GroupPendingInstances, GroupStandbyInstances, GroupTerminatingInstances, GroupTotalInstances]
LaunchConfiguration name:ondemand-zone-a-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com AutoscalingGroup/ondemand-zone-b-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com
MinSize 1
MaxSize 5
Subnets [name:worker-zone-b.prod.ap-southeast-1.k8s.managedkube.com id:subnet-9455cbdd]
Tags {CostCenter: kubernetes-saas, Owner: kubernetes, k8s.io/cluster-autoscaler/node-template/label/k8s.info/isSpot: false, k8s.io/cluster-autoscaler/node-template/label/k8s.info/application: infrastructure, Name: ondemand-zone-b-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com, KubernetesCluster: prod.ap-southeast-1.k8s.managedkube.com, Project: cloud, Purpose: kubernetes-ondemand-node, k8s.io/cluster-autoscaler/node-template/label/prod.ap-southeast-1.k8s.managedkube.com/role: scale-zero, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: ondemand-zone-b, k8s.io/cluster-autoscaler/node-template/label/k8s.info/hasPublicIP: false, k8s.io/cluster-autoscaler/node-template/label/k8s.info/instanceType: m4.large, k8s.io/role/node: 1}
Granularity 1Minute
Metrics [GroupDesiredCapacity, GroupInServiceInstances, GroupMaxSize, GroupMinSize, GroupPendingInstances, GroupStandbyInstances, GroupTerminatingInstances, GroupTotalInstances]
LaunchConfiguration name:ondemand-zone-b-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com AutoscalingGroup/ondemand-zone-c-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com
MinSize 1
MaxSize 5
Subnets [name:worker-zone-c.prod.ap-southeast-1.k8s.managedkube.com id:subnet-53715d15]
Tags {CostCenter: kubernetes-saas, k8s.io/cluster-autoscaler/node-template/label/k8s.info/isSpot: false, k8s.io/role/node: 1, Name: ondemand-zone-c-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com, KubernetesCluster: prod.ap-southeast-1.k8s.managedkube.com, k8s.io/cluster-autoscaler/node-template/label/prod.ap-southeast-1.k8s.managedkube.com/role: scale-zero, Owner: kubernetes, Project: cloud, Purpose: kubernetes-ondemand-node, k8s.io/cluster-autoscaler/node-template/label/k8s.info/application: infrastructure, k8s.io/cluster-autoscaler/node-template/label/k8s.info/hasPublicIP: false, k8s.io/cluster-autoscaler/node-template/label/k8s.info/instanceType: m4.large, k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup: ondemand-zone-c}
Granularity 1Minute
Metrics [GroupDesiredCapacity, GroupInServiceInstances, GroupMaxSize, GroupMinSize, GroupPendingInstances, GroupStandbyInstances, GroupTerminatingInstances, GroupTotalInstances]
LaunchConfiguration name:ondemand-zone-c-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com LaunchConfiguration/ondemand-zone-a-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com
ImageID 595879546273/CoreOS-stable-1745.7.0-hvm
InstanceType m4.large
SSHKey name:managedkube_automation_1 id:managedkube_automation_1
SecurityGroups [name:nodes.prod.ap-southeast-1.k8s.managedkube.com id:sg-c649a7bf]
AssociatePublicIP false
IAMInstanceProfile name:nodes.prod.ap-southeast-1.k8s.managedkube.com id:nodes.prod.ap-southeast-1.k8s.managedkube.com
RootVolumeSize 128
RootVolumeType gp2
SpotPrice LaunchConfiguration/ondemand-zone-b-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com
ImageID 595879546273/CoreOS-stable-1745.7.0-hvm
InstanceType m4.large
SSHKey name:managedkube_automation_1 id:managedkube_automation_1
SecurityGroups [name:nodes.prod.ap-southeast-1.k8s.managedkube.com id:sg-c649a7bf]
AssociatePublicIP false
IAMInstanceProfile name:nodes.prod.ap-southeast-1.k8s.managedkube.com id:nodes.prod.ap-southeast-1.k8s.managedkube.com
RootVolumeSize 128
RootVolumeType gp2
SpotPrice LaunchConfiguration/ondemand-zone-c-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com
ImageID 595879546273/CoreOS-stable-1745.7.0-hvm
InstanceType m4.large
SSHKey name:managedkube_automation_1 id:managedkube_automation_1
SecurityGroups [name:nodes.prod.ap-southeast-1.k8s.managedkube.com id:sg-c649a7bf]
AssociatePublicIP false
IAMInstanceProfile name:nodes.prod.ap-southeast-1.k8s.managedkube.com id:nodes.prod.ap-southeast-1.k8s.managedkube.com
RootVolumeSize 128
RootVolumeType gp2
SpotPrice
2) It tells me that these instance groups MinSize will be changed from what value to the new value:
Will modify resources:
AutoscalingGroup/spot-zone-a-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com
MinSize 0 -> 1 AutoscalingGroup/spot-zone-b-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com
MinSize 0 -> 1 AutoscalingGroup/spot-zone-c-m4-large-infrastructure.prod.ap-southeast-1.k8s.managedkube.com
MinSize 0 -> 1
3) Then finally it outputs the runtimeRequestTimeout flag that I wanted to add and it enumerates that through to all of the instance groups that it touched.
LaunchConfiguration/master-ap-southeast-1a.masters.prod.ap-southeast-1.k8s.managedkube.com
UserData
...
podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.0
podManifestPath: /etc/kubernetes/manifests
+ runtimeRequestTimeout: 10m0s
masterKubelet:
allowPrivileged: true
...
podManifestPath: /etc/kubernetes/manifests
registerSchedulable: false
+ runtimeRequestTimeout: 10m0s __EOF_CLUSTER_SPEC
... LaunchConfiguration/master-ap-southeast-1b.masters.prod.ap-southeast-1.k8s.managedkube.com
UserData
...
podInfraContainerImage: gcr.io/google_containers/pause-amd64:3.0
podManifestPath: /etc/kubernetes/manifests
+ runtimeRequestTimeout: 10m0s
masterKubelet:
allowPrivileged: true
...
podManifestPath: /etc/kubernetes/manifests
registerSchedulable: false
+ runtimeRequestTimeout: 10m0s __EOF_CLUSTER_SPEC
...
And finally at the end, it tells me if I want to apply these changes add the --yes to my command.
Must specify --yes to apply changesThis is absolutely wonderful and this is how I like to apply infrastructure level changes to my cluster. Changing any of these without being certain what it will do exactly can mean major downtime! I know this is mostly just a diff of some configs but it is presented out so well that every time I apply this type of changes it makes me smile =)
—
ManagedKube built k8sBot, a point-and-click interface for Kubernetes in Slack. Now, software developers and novice k8s users can get meaningful Kubernetes information with just one click in Slack, where you’re already talking with your team. With k8sBot, you can retrieve pod status, get pod logs, and get troubleshooting recommendations based on real-time information from your cluster’s Kubernetes API.
Learn more and start a free trial of k8sBot at managedkube.com
