KubeCon ’19 Keynote: A centralised globally distributed platform
In May this year I had the privilege to keynote Day 2 at KubeCon + CloudNativeCon in Barcelona. It was a remarkable experience to be on stage and share Condé Nast International’s story of a global platform that embraces cloud native principles. This blog targets to provide an overview of the keynote and the platform at Condé Nast International.
Journey Preamble
With the headquarters based in London, Condé Nast International operates on an international scale in more than 11 countries across different geographies, including China and Russia. Our brand portfolio consists of 62 fashion and luxury oriented websites, that reach 301.5 million digital unique users monthly. This generates a total of 1.5 billion page views dividing into approximately 570 article views every second.
This is a considerable customer base and Condé Nast International targets to deliver the highest user experience for all existing and future customers. One step towards achieving this goal is the way we build and deploy our platform.
A centralised platform
In the past, the global teams acted like independent business units. At the time this made perfect sense as it provided the necessary environment for growth and development. However, from an engineering point of view, with evolution of different deployment mechanisms, it was clear that markets have a customised technology stack, with almost no shared resources between them. This includes the systems and platforms used to serve the live traffic and even the CMS tools used by the international teams to create and deliver the content. Another major inconsistency was the visual and design across brands, making it difficult to provide a unified experience for all our users.
With this in mind, 2 years ago, Condé Nast International triggered a digital transformation project that aims to rollout a centralised technology platform across the globe, whilst embracing the cloud native principles.
Platform Overview
All the websites hosted on the global platform are fronted by Fastly, our CDN service provider. Having a CDN provider reduces considerably the load on our backend systems, while optimising the content delivery time to the users.
Supposedly, an editor published a new article that has not yet been cached by the CDN. In this case a “x-cache: MISS” response header will be returned and the request will be forwarded to Traefik through an ELB. Traefik will route the requests to the multi-tenant global web-application, fetching the desired article and supplying the content to the readers.
Traefik is our ingress controller that manages the external access to the services within the Kubernetes clusters. In the current setup, we have 2 dedicated load balancers per namespace (for public and restricted traffic), that routes requests to name-based virtual hosts.
Kubernetes bootstrapping
Kubernetes clusters underpins the entire infrastructure at Condé Nast International. We are a CoreOS shop, using Tectonic installer for cluster deployments and etcd for our key-value store of our cluster data.
Tectonic allows the bootstrapping of a fully self-hosted Kubernetes control plane, while plugging in into different cloud providers to consume compute and networking resources. As such, we avoid vendor-lock in, while deploying self-hosted, multi-master, multi-workers, 100% open source Kubernetes clusters.
At this point in time, we manage over 100 instances in AWS to host 5 Kubernetes control planes. Due to uneven customer traffic throughout the day, we are using auto scaling groups in AWS , to scale up and down the number of nodes based on custom metrics.
Most of the showcased infrastructure is deployed using Terraform, having infrastructure as code (IAC) at the basis.
Our platform is serving live traffic since October last year, when we first launched GQ France. Since then, we successfully migrated 13 other websites, making a total of 14 websites. Monthly the platform is serving more than 80 million articles to our customers.
Global Distribution
As mentioned previously, Condé Nast International operates in areas, which a quite geographically dispersed. We need to consider the global distribution of our platform.
For example, from China and Russia alone we have around 19 million unique visitors monthly, which represents 17% of the digital readership on the global scale. This only highlights the need for Condé Nast International to be prepared technically to meet the customer demand in these areas.
Also, platform deployment in China and Russia spotlights the requirements for a cloud agnostic infrastructure. When the project for a unified platform was kicked off, there was not a clear vision of a cloud provider that would be used to host the platforms in all regions.
Another reason for cluster distribution was the origin latency. If all editorial content would be served from a single region and the cache would not be available, then the time to pull from origin would introduces a noticeable latency for the end user. This is not satisfactory for our business values.
Condé Nast International aims for market proximity and highest standards for our consumers. Hence, it was a business decision to replicate the current tech stack across 5 different regions, making a total of 9 Kubernetes clusters.
- 3 cluster in Frankfurt — targets European markets
- 2 clusters in Dublin for hosting internal tool
- 1 cluster in Tokyo which targets Asia Pacific markets
- 1 cluster in N. Virginia which targets Mexico market
- 2 cluster in China
We do not have a replica of our tech stack in Russia, as there is no major cloud provider that we can use for our infrastructure. The current strategy is to serve the content from the European servers, while using a Russia oriented CDN provider. This will ensure the service availability of our content in Russia, while investing minimal engineering effort. However, this solution will be re-evaluated if the time to serve content to our users is unsatisfactory for our business.
Self-service CI/CD
One of the reason to roll out a unified global tech stack was to further empower local and international teams. Hence, the cloud platforms team are strongly advocating for a well-defined, solid and self-service CI/CD process.
To deploy to the Kubernetes clusters, the teams will require a dockerized version of their application. These will be stored in Quay.io, our choice of Docker image registry.
Secondly, a template helm chart is provided for application deployment. A configuration file can be supplied by the teams to override default helm chart values (e.g. amount of replicas, CPU, memory, HPA etc). These settings allow transparency and flexibility to custom build services in each region.
Once these 2 dependencies are available, a CircleCI pipeline will be triggered to deploy the application in the production environment. The pipeline can be entirely adjusted by the dev teams, so it is common practice to inject application specific workflows if required.
Every team is allocated a namespace, where they have admin privileges. We make use of resource quotas to prevent use cases of resource over-utilisation in the clusters.
It is worth mentioning that the RBAC policies currently set allows view-only privileges to ALL users. As such, developers have access to all resources in the cluster, except the secrets in other namespaces, and core components of the cluster. This provides a great incentive for inter-team collaboration and knowledge sharing. As for the cloud platforms team or admin team, it enforces the best practices for interaction with the cluster as all changes will be applied via pipeline.
Summary
Condé Nast International is truly on the path for digital transformation. The project is far from completion, however so far we have achieved to build a centralised platform that is scalable, fault tolerant and distributed across different geographies around the globe. However, what makes our tech stack so successful, is the robust CI/CD process that further emancipates our local and international teams and their product.
Also, my keynote from KubeCon Europe ’19 is available on YouTube. Watch it here!
