Xandr-Tech
Published in

Xandr-Tech

Going the Extra Mile with Our k8s Setup

Kubernetes logo over network of interconnected light nodes
Photo by Conny Schneider on Unsplash; The Kubernetes logo is a registered trademark of the Linux Foundation

The challenge

The problem, illustrated

App dependency chain sample
  • static tide throughput.
  • idle resources on the previous app, while the next app in the chain is struggling.
  • backlogs being created on every app when the magic number is no longer valid and demand spikes occur.

Proof of Concept time

Illustration of pod scaling vertically to meet demand
Illustration of pod scaling horizontally to meet demand

Autoscaling based on CPU

// source from: https://gist.github.com/sqren/5083d73f184acae0c5b7// higher number => more iterations => slower
function intenseCalculation(baseNumber) {
console.time('mySlowFunction');
let result = 0;
for (var i = Math.pow(baseNumber, 7); i >= 0; i--) {
result += Math.atan(i) * Math.tan(i);
};
console.timeEnd('mySlowFunction');
}
--------------------------------------------app.post('/work', async (req, res) => {
intenseCalculation(5)
res.status(200).send('ok');
})
------------------------------
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: poc-k8s-vpa-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: poc-k8s-vpa-hpa-app-creat120
minReplicas: 1
maxReplicas: 100
metrics:
- type: Resource
resource:
name: cpu
target:
type: AverageValue
averageValue: 0.01
  1. Trigger command
for id in {1..1000}; do curl -X POST 'ENVIRONMENT_URL/work'; done;
kubectl describe horizontalpodautoscaler --context qa01nym2 -n creative poc-k8s-vpa-hpa
Output of kubectl describing horizontalpodautoscaler

Plot twist

kubectl describe horizontalpodautoscaler --context qa01nym2 -n creative poc-k8s-vpa-hpaWarning  FailedComputeMetricsReplicas  18m (x12 over 21m)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get object metric value: unable to get metric requests-per-second: Ingress on creative main-route/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registeredWarning  FailedGetObjectMetric         77s (x80 over 21m)  horizontal-pod-autoscaler  unable to get metric requests-per-second: Ingress on creative main-route/unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registeredkubectl get --raw/apis/metrics.k8s.io/v1beta1/namespaces/creative/pods/poc-k8s-vpa-hpa-app-creat120-5c794d9bb4-jxxsl  --context qa01nym2 -n creative | jqError from server (Forbidden): pods.metrics.k8s.io "poc-k8s-vpa-hpa-app-creat120-5c794d9bb4-jxxsl" is forbidden: User "REDACTED" cannot get resource "pods" in API group "metrics.k8s.io" in the namespace "creative"kubectl get --raw /apis/custom.metrics.k8s.io/ --context qa01nym2Error from server (NotFound): the server could not find the requested resource

Autoscaling based on custom metrics

  • We updated the code so it would push items to RabbitMq. This wasn’t technically required for the POC, since all that really mattered was that the metric was pushed.
  • We ensured that report metrics were sent to Prometheus.
app.post('/work', async (req, res) => {
const message = req.query.message;
// pushes a message to Rabbit MQ
await rabbitmq.publishMessage(
"my_queue",
JSON.stringify({
created_on: new Date().getTime(),
message
})
);
//sends a metric to prometheus
//metric name: total_messages_pushed
addTotalPushedData();

res.status(200).send('ok');
})
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: custom-metric-hpa
namespace: creative
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: poc-k8s-vpa-hpa-.Release.name
minReplicas: 1
maxReplicas: 10
metrics:
- type: Object
object:
metric:
name: messages_pushed_per_second
describedObject:
kind: Namespace
name: creative
apiVersion: v1
target:
type: Value
value: 2
apiVersion: v1
data:
config.yaml: |
rules:
- seriesQuery: 'total_messages_pushed'
resources:
overrides:
namespace: {resource: "namespace"}
name:
matches: "^total_(.*)"
as: "${1}_per_second"
metricsQuery: "rate(total_messages_pushed{<<.LabelMatchers>>}[5m])"
for id in {1..1000}; do curl -X POST 'ENVIRONMENT_URL/work?q=example'; done;
kubectl describe horizontalpodautoscaler --context qa01nym2 -n creative poc-k8s-vpa-hpa
Output of k8s horizontal pod auto scaler

What’s next?

  • Maintain a pool of resources to allow autoscaling across multiple apps, rather than hardcoding a set number of resources per application.
  • Achieve a moving tide throughput. Pods scale up or down on demand, with resources allocated accordingly.
  • Reduce and optimize our team’s resource fingerprint.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store