What is Terraform ?

Terraform is a powerful software from Hashicorp that allows you to write infrastructure as code in a declarative way to provision and manage any cloud, infrastructure, or service.

Image for post
Image for post
Terraform Workflow (Source: https://www.terraform.io/)

The terraform plan command (similar to git diff output) is used to create an execution plan. Terraform performs a refresh, and then determines what actions are necessary to achieve the desired state specified in the source code.

Image for post
Image for post
Output generate by terraform plan command (Source: https://github.com/dmlittle/scenery)

Once the execution plan looks good, terraform apply command is used to apply the changes required to reach the desired state of the configuration according to the plan.

Ops engineers are usually responsible of operating such infrastructure software, but what if we could change that and unblock developers to build even faster ? …

Kafka is a really powerful distributed publish / subscribe software that helps you build complex asynchronous applications. GumGum was an early adopter of this technology and is nowadays running hundreds of brokers across multiple clusters.

Kafka monitoring is a must

Kafka cluster operations are a thing on their own (scaling clusters in and out, recovering from a dead broker, reassigning partitions across the cluster…) but if you want to build performant client applications based on Kafka you want to pay close attention to your consumers:

  • Are they able to keep up with the incoming traffic (consumer lag) ?

Kafka Consumer Lag is an indicator of how much lag there is between Kafka producers and consumers. In other words, how far behind is your consumer compared to the latest produced message in the topic the consumer is reading from.


Florian Dambrine

Principal Engineer & DevOps enthusiast, building modern distributed Machine-Learning eco-systems @scale

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store