Open in app

Sign in

Write

Sign in

EXPEDIA GROUP TECHNOLOGY — ENGINEERING

Using Terraform to Provision Spinnaker on Kubernetes

How Expedia Group installs Spinnaker in the cloud

Manoj Tiwari
Expedia Group Technology

--

A couple pose for a photograph in front of the spectacular Pyramid of the Magician in Uxmal, an ancient Maya city of the classical period located in present-day Mexico.
Photo by Fernando Gago on Unsplash

Overview

Spinnaker frees developers up from having to worry about supporting infrastructure so they can concentrate on building code and delivering their apps. The development team may concentrate on developing the application while leaving the automated reinforcement of business and regulatory requirements to Spinnaker for operations provisioning. Spinnaker is a cloud-based, open-source continuous delivery platform originally developed by Netflix for the purpose of quickly and consistently deploying software modifications. It supports several continuous integration, build and deployment tools. It features integrations for configuring artifacts from Git, cloud services and other sources.

You can read more about Spinnaker at— https://spinnaker.io/

There are several methods available to install open-source Spinnaker:

  • Halyard — Halyard is a command-line administration tool that manages the lifecycle of your Spinnaker deployment, including writing and validating its configuration, deploying each of Spinnaker’s microservices and updating the deployment.
  • Kleat — Intended to replace Halyard, Kleat is a tool for managing Spinnaker configuration files. It is currently under active development, but it is ready for early adopters to try in development clusters.
  • Spinnaker Operator — This is a Kubernetes operator that deploys and manages Spinnaker using familiar tools. You can install a basic version of Spinnaker or use Kustomize files for advanced configuration.

At Expedia Group™️, we use Terraform in production to provision and manage Spinnaker at scale. Operating 18K+ application pipelines, we onboard hundreds of cloud provider accounts and integrate with a variety of custom plugins for different deployment targets.

After experiencing some problems with the aforementioned installation options, we concluded that they were not the best choice for us to manage the large-scale production environments. Instead, we chose Terraform to develop our own customized solution to manage Spinnaker provisioning at Expedia Group. We completely converted all resource setups into Terraform templates in our customized solution rather than using Halyard and Spinnaker-Operator. With Terraform, we can now easily and elegantly handle Spinnaker settings.

Terraform can provision, update and manage resources on any environment in any cloud, with some versioning. The human-readable configuration language helps you write infrastructure code quickly. Terraform’s state allows you to track resource changes throughout your deployments.

Spinnaker is handled by Terraform in the same manner as other Kubernetes and cloud resource deployments. You can have configuration templates or manifest resources for each Spinnaker service. Common modules can be used by other resources. This gives us advantages like having one centralized deployment for both Spinnaker and other cloud resources, all managed in one codebase. This makes it easy for collaboration within the team, and has no upstream dependencies, unlike Spinnaker Operator or other installation methods.

Now that we have the context, let’s dive right in to the fun part…

Schematic diagram showing how Terraform is used to provision cloud resources.
Fig: Deploy Spinnaker, K8s, Cloud resources with Terraform

The diagram above represents a high-level overview of a Spinnaker deployment on Kubernetes on a cloud provider.

  • Terraform templates: These templates contains TF configurations for all the Spinnaker Cloud and Kubernetes resources.
  • Terraform deployer: A deployer is how you execute your Terraform commands, e.g., via Jenkins job or another continuous integration tool to trigger Terraform.

Terraform configuration

Prerequisites

  • You have a basic understanding of Terraform.
  • You have a basic understanding of Spinnaker and different configuration files
  • You have a Kubernetes cluster up and running with basic configurations in place like namespaces , roles etc.
  • Cloud account with basic resource setup like VPC, subnets, etc.

The section below illustrates how we structure and manage our Spinnaker configuration. While it is not a comprehensive design breakdown, this article covers the necessary configurations and patterns so that readers can get an understanding of the approach. Since Terraform is versatile, one can extend the modules in accordance with their production setup and rearrange the configurations, as necessary.

Schematic diagram showing Terraform modules used to deploy Spinnaker.
Fig: Spinnaker Terraform module based structure layout

Let’s understand the above terraform configurations framework layout:

  • Modules: Terraform modules contain shared modules for Spinnaker service’s configurations and deployments. It also contains other shared cloud resources like DB’s, Roles, etc.
  • spinnaker-service-cluster module: The Terraform shared module contains the Kubernetes deployment and service resources, which can be source-coded and overridden in each service’s individual configuration file.
# deployment.tf
resource "kubernetes_deployment" "this" {
  metadata {
    ...
  }

  spec {
     ...

    selector {
      ...
    }

    strategy {
      ...
      rolling_update {
        ..
      }
    }

    template {

      metadata {
        ...
      }

      spec {

        ...

        container {

          dynamic "env" {

            content {
              
            }
          }

          image ...
          name ...

          liveness_probe {
            dynamic "http_get" {
              ...

              content {
                path ...  
                port ...
                scheme ..
              }
            }

            dynamic "tcp_socket" {
              ....

              content {
                port ...
              }
            }

            ...
            ...
          }

          port {
            ...
          }

          readiness_probe {
            dynamic "http_get" {
              ....

              content {
                ...
              }
            }

            dynamic "tcp_socket" {
              ...

              content {
                ...
              }
            }

            ...
            ...
          }

          resources {
            limits = {
              cpu...
              memory...
            }

            requests = {
              cpu...
              memory...
            }
          }

          security_context {
            allow_privilege_escalation = true/false
          }

          dynamic "volume_mount" {

            content {
              name ....     
              mount_path ....
              read_only ....
            }
          }

          volume_mount {
            name ...     
            mount_path ...
            read_only ...
          }

          volume_mount {
            name  ...
            mount_path ...
            read_only  ...
          }
        }

        termination_grace_period_seconds = <value> # in seconds

        dynamic "volume" {
          ...

          content {
            name ..
            config_map {
              name ...
            }
          }
        }

        volume {
          name ...
          config_map {
            name ...
          }
        }

        volume {
          name ...
          secret {
            secret_name...
          }
        }

        dynamic "security_context" {
          ...

          content {
            run_as_user...   
            run_as_group ...  
            run_as_non_root ...
            fs_group ...      
          }
        }
      }
    }
 }

}


# service.tf
resource "kubernetes_service" "this" {

  metadata {
    name ...
    namespace ...
    labels ...
  }

  spec {

    port {
      port ...
      protocol ...
      target_port ...
    }

    selector ...

    type = "ClusterIP"
  }

}
  • spinnaker-service-config: This Terraform shared module contains the Kubernetes config-map’s and secrets resources, which can be source-coded and overridden in each service’s individual configuration file to load the service configurations and secrets like spinnaker.yml and profile configs as configmaps and secrets mounted in deployments.
# configmap.tf, service container read service configs from configmaps mounted in deployments 

resource "kubernetes_config_map" "config_map" {
  metadata {
    name  ...
    namespace ...
    annotations { ... }
    labels { ... }
  }

  data  ...     # eg, data = local.data
  binary_data = {}
}


locals {

  # <!-- Based on Netflix config from https://spinnakerteam.slack.com/archives/CC101KSNP/p1601492412017300 -->
  log_levels {...}

  internal_service_urls {
    "clouddriver"        = "http://clouddriver.<namespace>:7002"
    "echo"               = "http://echo.<namespace>:8089"
    "fiat"               = "http://fiat.<namespace>:7003"
    "front50"            = "http://front50.<namespace>:8080"
    "igor"               = "http://igor.<namespace>:8088"
    "kayenta"            = "http://kayenta.<namespace>:8090"
    "orca"               = "http://orca.<namespace>:8083"
    "rosco"              = "http://rosco.<namespace>:8087"
  }

  data = {

    "spinnaker.yml" = yamlencode({

      global = {
        spinnaker = {
          timezone = "Etc/UTC"
        }
      }

      logging = {
        config = "/opt/spinnaker/config/logback.xml"
      }

      # These timeouts apply to *all* Java services
      hystrix = {
        command = {
          default = {
            execution = { isolation = { thread = { timeoutInMilliseconds = 60000 } } }
          }
          builds = {
            execution = { isolation = { thread = { timeoutInMilliseconds = 60000 } } }
          }
        }
      }

      okHttpClient = {
        readTimeoutMs ...
        connectTimeoutMs ...
      }

      services = {

        clouddriver = {
          baseUrl = local.internal_service_urls["clouddriver"]
          enabled = true/false
        }

        deck = {
          baseUrl = "<deck_domain/url>"
          enabled = true/false
        }

        echo = {
          baseUrl = local.internal_service_urls["echo"]
          enabled = true/false
        }

        fiat = {
          baseUrl = local.internal_service_urls["fiat"]
          enabled = true/false
        }

        front50 = {
          baseUrl           = local.internal_service_urls["front50"]
          enabled           = true/false
        }

        gate = {
          baseUrl = "<gate_domain/URL>/api/v1"
          enabled = true/false
        }

        igor = {
          baseUrl = local.internal_service_urls["igor"]
          enabled = true/false
        }

        kayenta = {
          baseUrl           = local.internal_service_urls["kayenta"]
          enabled           = true/false
        }

        orca = {
          baseUrl = local.internal_service_urls["orca"]
          enabled = true/false
        }

        rosco = {
          baseUrl = local.internal_service_urls["rosco"]
          enabled = true/false
        }
      }

      spinnaker = {
        baseUrl = {
          www = "https://<deck_domain>"
        }

        s3 = {
          enabled = true/false
        }
      }

      server = {
        "max-http-header-size" = <value>
      }
    })

    # Profile configuration section, Optional: you can also  use s3 to read profile configs as perhttps://spinnaker.io/setup/configuration/#using-an-s3-backend
    "spinnakerconfig.yml" = yamlencode({
      spring = {
        profiles = {
          ....
    })

    "logback.xml" = templatefile("path for logback xml file", {
      log_levels = local.log_levels
    })
    ...
    ...
  }

}

# secrets.tf, service container reads secrets from secrets mounted in deployments

resource "kubernetes_secret" "secret" {
  metadata {
    name ...
    namespace ...
    labels ...
  }

  data ...

  type = "Opaque"
}
# Example front50.tf [This is only for reference].

module "front50_role" {
  source                  = <module path>
   <override the values accordingly. You can extend this file or shared module as per configuration needs.>

}

module "front50_db_setup" { 
  source                 = <module.path>
  <Override the values>
}

module "front50_service_config" {
  source = <spinnaker-service-config module path>
  
  <override the values as per service need>
  
}

module "front50_service_cluster" {
  source = <spinnaker-service-cluster module path>
  <override the values as per service need>

}

Note: Deck doesn’t use the same configuration system as the Java apps, so it can’t use the “spinnaker-service-config” module.

# deck.tf 
locals {
  deck_config_data = {
    ... # config data like ssl certs, secrets, etc.
    ...
    "settings.js" = format("window.spinnakerSettings = JSON.parse('%s');", jsonencode({
      
      .... # put settings.js configurations here 
  }
}


resource "kubernetes_config_map" "deck" {
  metadata {
    name   ...
    namespace  ...
    annotations = { ... }
    labels = { ... }
  }

  data        = local.deck_config_data # loads from above configs
  binary_data = { ... }
}



module "deck_service_cluster" {
  source = <pinnaker-service-cluster module path>
  
  # Ensure that Gate is deployed before updating Deck
  depends_on = [
    <gate deployment>
  ]  

  <Override shared module configurations per service need> 



  # https://github.com/spinnaker/kustomization-base/pull/31
  set_security_context = false # Required because Deck can't run with a different UID

  service_environment = {
    API_HOST  = "http://gate.<namespace>:8084"
    DECK_HOST = "0.0.0.0"
    DECK_PORT = "9000"
    DECK_CERT = "/opt/spinnaker/config/deck.crt"
    DECK_KEY  = "/opt/spinnaker/config/deck.key"
  }

 ...
}
  • data_source: Terraform data_source resources to fetch the data of existing resources like VPC, subnets, K8S clusters, etc., that are used in other modules, resources and providers.
  • providers: Terraform providers are responsible for understanding API interactions and exposing resources. https://registry.terraform.io/browse/providers
# Examples providers used in spinnaker terraform modules
 
provider "<any cloud provider>" {
  ....
  }
}


provider "kubernetes" {
  ....
}

  • Ingress: Terraform resources for Kubernetes ingress to expose the Deck and Gate endpoint in the K8S cluster. The configurations depend on the ingress controller you are using and how you want to expose Deck and Gate endpoints. You may choose not to use ingress and expose the endpoints using the K8S load balancer service.
# Example ingress resource 
resource "kubernetes_ingress" "ingress" {
  metadata {
    namespace ...
    name      ...

    annotations = {
        ....
    }
  }

  spec {
    ....
    }
    tls {
      ...
    }
  }
}
  • variables and versions: Terraform variables and provider version configurations are used. Provider versions can be restricted, and variables can be defined in variables.tf or managed with Terragrunt (Optional). At EG, we use Terragrunt to manage the Terraform input configurations.

Benefits of managing Spinnaker with Terraform:

  • Manage any infrastructure and all resources with one tool
  • Easy to review and rollout any changes into production
  • Easy to track and manage Spinnaker services configurations
  • Easy to scale or extend the resources
  • No dependencies on any upstream or third-party solution.
  • Easy to customize as per requirement.
  • You can compose resources from different providers into reusable Terraform configurations called modules, and manage them with a consistent language and workflow.
  • Terraform keeps track of your real infrastructure in a state file, which acts as a source of truth for your environment.
  • Terraform allows you to collaborate on your Spinnaker infrastructure and deployment with its remote state backends.
  • Terraform’s state allows you to track resource changes throughout your deployments.
  • The human-readable configuration language helps you write infrastructure code quickly.
  • You can commit your configurations to version control to safely collaborate on infrastructure.
  • It gives much more control on what resources we want to install, adding additional metadata like tags, for example.

Although configurations vary depending on the needs and usage of each business, such as whether persistent storage is provided by DB or S3, how to manage cloud-driver provider configurations or any other Cloud/K8S /Spinnaker resources, etc., the fundamental logic for Spinnaker service configurations and setup more or less remains similar, if not the same.

#Happy Terraforming #Terraform for All the Things !!

Learn more about technology at Expedia Group

Kubernetes
Spinnaker
Terraform
Software Engineering
Cicd

--

--

Manoj Tiwari
Expedia Group Technology

Written by Manoj Tiwari

4 Followers
Writer for

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams