Building a Container Platform at Cruise

Karl Isenberg
Jun 5 · 6 min read
  1. Container Platform Security

Kubernetes

Kubernetes, at its core, is a platform for building platforms. It’s not a highly opinionated turn-key solution — it’s more of a foundational layer built for flexibility, which is exactly what we needed to support Cruise’s wide variety of workloads.

Environments & Tenants

When using Kubernetes, it’s tempting to assume you can get by with having only one cluster, and co-locating all your workloads to optimize efficiency and cost. However, it quickly became apparent to us that we still needed multiple clusters, even with our multi-tenant strategy. Co-locating containerized workloads gives us higher compute utilization, less operational overhead, and faster deployment. The trade off is that we sacrifice some resource, network, and tenant isolation. Sometimes, it makes sense to share, and other times it makes sense to isolate.

Infrastructure Boundaries

At the infrastructure layer, we configured GCP with vertical and horizontal boundaries:

  1. Horizontally: GCP projects within each environment allow for distinct permission management and visibility constraints between tenants. Projects make it easy for tenants to focus on what matters to them. Separation between tenants makes it harder for malicious actors to gain unauthorized access, and prevents tenants from accidentally impacting each other.
GCP Environment Matrix

Platform Boundaries

While GCP provides the primitives for infrastructure level isolation, Kubernetes provides the primitives for platform level isolation. At the infrastructure level, tenants can take advantage of GCP managed services such as CloudSQL, Stackdriver, BigQuery, Cloud Machine Learning Engine (CMLE), and Google Compute Engine (GCE). At the platform level, tenants can manage their container workloads, like applications, services, and jobs. Both layers help improve availability by increasing isolation, and provide boundaries on which to manage permissions, allowing for both coarse-grained and granular permission management. Both layers also make it easy for legal and security teams to audit who has access to what, and where each resource came from.

  1. Horizontally: Kubernetes namespaces act as tenant boundaries between workloads. This allows cluster admins to manage permissions, quota, and resources at the cluster level while delegating similar control on a smaller scale to tenants as namespace admins. Namespaces are used as security boundaries for role based access control (RBAC) both in Kubernetes and in integrated systems, like Vault and Spinnaker. Namespace are similar to GCP Projects in that they make it easy for tenants to focus on what matters to them.
GKE Environment Matrix

Platform as a Service

The overall Cruise Platform as a Service (PaaS) is a collection of integrated services, which enhance Kubernetes and GKE, and provide additional functionality for our engineers.

Cruise PaaS
  • Ingress
  • Observability
  • Deployment

To Be Continued…

Through this series of blog posts, we’ll cover the security, networking, observability, and other technical challenges we experienced while building the Cruise PaaS. Continue reading with our deep dive on container platform security!

Cruise

Cruise builds the world’s most advanced network of self-driving vehicles to safely connect people with the places, things, and experiences they care about.

Karl Isenberg

Written by

Container Platforms at Cruise Automation. Ex-Mesosphere. Ex-Pivotal. Gamer. Driver. Night-Time Sunglasses Wearer. LEGO builder.

Cruise

Cruise

Cruise builds the world’s most advanced network of self-driving vehicles to safely connect people with the places, things, and experiences they care about.