Adobe Customer Journey Management’s Journey into the World of GitOps

Authors: Seung Kim and Marco Massenzio.

Jaemi Bremner
Oct 29 · 8 min read
Image for post
Image for post

The topic of CI (Continuous Integration) and CD (Continuous Deployment) is one of the most hotly debated across organizations and development communities. CI/CDs have many different solutions to, and opinions by, organizations, developers, and communities.

By and large, however, everyone agrees that at a minimum, a CI/CD framework should provide:

  • Orchestrated code-to-deployment process

This blog details Adobe Customer Journey Management (CJM) CI/CD discovery, architecture, and our learnings.

Adobe Customer Journey Management (CJM) CI/CD Overview

Adobe’s Customer Journey Management (CJM) is designed from the ground-up, to be a cloud-native and container-based multi-tenant system. We have chosen Kubernetes as our execution environment, as it is well-supported within Adobe. That decision was motivated by the following considerations:

  1. Kubernetes allows developers to focus on logical entities, such as services, pods, and stateful services while abstracting from hardware details and Cloud provider specifics.

Consistency across services, when managing the build artifacts, and launching applications, was a requirement.

Thus, the requirements of our Adobe CJM CI/CD can further be specified as:

  1. CI is responsible for handling the process of creating well-defined and well-tested deployment artifacts (Docker Images) and all necessary configurations, resources, and states specific to an application version are embedded in the docker image.

Further, CI/CD process must guarantee correctness at every step in the code-to-artifact pipeline and that all validations are correctly evaluated (e.g. tests pass without failures) during the process of promoting the artifact from Development to Stage to Production.

Beyond the functional requirements, we also desire the build and deployment process to be as cheap and repeatable as possible. Here “cheap” really means what it says, as we use Public Cloud resources, and any efficiency gain is reflected in actual financial savings.

Requirements and Architectural Principles

We architected the system following the principle of “Separation of Concerns” so that the responsibilities are split accordingly:

  • Jenkins, to build and test the artifact, produce the container image

However, the design is such that, if required, each component could be substituted for an equivalent one. For example, we experimented with using CircleCI as our build/test toolchain.

After running some prototypes, the team realized the following:

  1. Kubernetes is useful but has a steep learning curve, so it may be beneficial to provide frequently used operations as a set of features in the CD process.

a. No more divergent and scattered information among git, spinnaker, Kubernetes, and any other related systems.

— All information always flows from Git. Everything else is derivative of git.

b. No manual intervention for deployment

— No manual changes via Spinnaker UI

— No changes via kubectl CLI commands

c. This also includes multi-region deployments, handled via git commands.

Our original visions, which we have achieved, are:

  1. Git as the source of truth, for code and deployment
Image for post
Image for post
Figure 1: Adobe CJM CI/CD Process
Image for post
Image for post
Figure 2: Overview of CJM Deployment

Our Learnings

Git as the source of truth, for code and deployment

Traditionally, coding has been handled by software engineers. Build and release was the realm of dedicated QA and Release engineers. This separation of concerns has led to divergent practices for the management of code and deployment scripts, and often to territorial ownerships between the various teams. Understanding and managing scripts and processes takes time and effort.

By extending what had become known as “Infrastructure as Code” to the area of build/release/deploy and codifying the necessary steps, as well as using git not only as of the sole source of truth but also as the common lingua franca across domains, the GitOps model aims to remove such barriers and contribute to a more open development environment.

Allow consistent access patterns for managing artifacts

Managing artifacts consistently (“convention over configuration”) allows us to address many concerns implicitly:

  1. How to ensure all artifacts can be treated in a consistent manner at the time of deployment

Our approach to CI/CD promotes the following conventions:

  1. Docker images as CI artifacts and use of a common “base” Docker image provides consistent app-specific runtime within docker-ecosystem;

HashiCorp Vault for secrets management

Secrets are necessary to interact with certain systems and services: they may be needed for Jenkins-based integration tests, and will most definitely be needed at deployment time.

However, entering them manually every time would be impractical (and would entirely break the premise behind CD) or, possibly worse, storing them along with the source code in a git repository is an absolute security no-no.

We chose to make secrets management declarative, instead.

  1. HashCorp Vault manages runtime secrets

Having only declarations stored in Git, CJM safely separates and maintains valid secrets data stored in Vault, and ensures always up-to-date secrets are published to Kubernetes.

Use Jenkins for managing Common Services/Objects in Kubernetes

CJM utilizes Jenkins as a means of managing common services and secrets.

As many common services are stateful, such as Prometheus and MemcacheD, we can safely maintain valid service states via rolling deployments. This approach allows service upgrade without interruption, while not having to worry about traffic management issues, which are transparently handled by Spinnaker.

Equally for Kubernetes Secrets, unlike app-specific configuration maps, secrets hardly require versioning. We just want what is in Vault to be presented in a consistent manner to our deployment environment.

Jenkins in this manner lays down the foundational dependencies for eventual application deployments.

Provide easy Helm chart generation via our templating engine

In a typical REST application deployment, we need the following features/objects.

  1. FQDN (fully qualified domain name) and URI mapping

Since these features are app-specific, they must be bundled in the application Helm chart. Most of these declarations are commonly reusable features. Copying/pasting YAML files lead to inconsistency and are highly prone to error; not to mention, any subsequent changes/additions need to be replicated (manually) across several code repositories, adding to the opportunity for errors, misconfiguration, and out-of-sync content causing issues during deployment (or, worse in Production).

Instead, we opted for the following:

  1. Create commonly used features as injectable YAML template files.

Ultimately, developers only need to focus on providing the following.

  1. docker-image and tag

All other information, such as ImagePullSecret, common Environment variables, and network policy are hidden away from the developer’s view.

Provide easy integration with Spinnaker

Spinnaker has many great features when it comes to deploying applications. Managing traffics during an upgrade is a good example. However, building Spinnaker pipelines can be challenging, as it requires detailed domain knowledge.

Also, pipelines can be easily modified via UI, which are impossible to track, and hard to replicate (typically by providing screenshots and documentation, which quickly gets stale and is hard to maintain); further, any accidental change can cause major consequences. Spinnaker’s limited change management functionality does not help much in this respect.

Others copy and paste the generated Pipeline JSON into Git. However, the initial generation of pipeline is not declarative; someone needed to create them manually, and the flow of data originates from Spinnaker — not from Git.

And even if it worked (which it doesn’t), any time one sees a process requiring manual copy & paste, a healthy dose of skepticism is in order: it is almost universally a symptom of lazy thinking and poor (or non-existent) automation.

In Production, it is virtually assured to come back and cause significant system outage and customer disruption at one point or another, and usually at the most critical moment, when one can least afford it.

Instead, our CD solution defines a frequently used set of pipeline phases (fetch Helm Chart -> bake -> deploy) which is stored in Git: this allows app developers to define only deployment clusters and environments (equally stored in Git).

Spinnaker pipeline JSON files are then auto-generated during deployment time and pushed from Jenkins via Spin CLI.

The end result is that there is no need to manage Spinnaker pipelines via the UI directly; the latter is used mostly for debugging purposes, as a “read-only” control pane.

What’s Next

Adobe has a vast amount of different technologies and practices it has acquired over the years. Some are from individual contributors. Some are from acquisitions of existing teams/companies. Along with them, their unique culture and knowledge have combined into Adobe.

Our team’s CI/CD main goal is to provide a safe deployment environment, where the basic functionality “just works,” so that all applications onboarded on our large-scale distributed customer journey management solution can enjoy flexibility in development, and benefit from well-orchestrated CI/CD processes, to help developers understand where and how their applications are.

We believe this approach also has a positive side-effect in promoting the DevOps mindset and is rooted in the GitOps principles.

Follow the Adobe Tech Blog for more developer stories and resources, and check out Adobe Developers on Twitter for the latest news and developer products. Sign up here for future Adobe Experience Platform Meetups.

References

  1. Adobe Customer Journey Management

Adobe Tech Blog

News, updates, and thoughts related to Adobe, developers…

Jaemi Bremner

Written by

Experience Technologist. Developer Advocate for Adobe Experience Platform. Passionate about technology, architecture, fashion, and design. Twitter: @jaeness

Adobe Tech Blog

News, updates, and thoughts related to Adobe, developers, and technology.

Jaemi Bremner

Written by

Experience Technologist. Developer Advocate for Adobe Experience Platform. Passionate about technology, architecture, fashion, and design. Twitter: @jaeness

Adobe Tech Blog

News, updates, and thoughts related to Adobe, developers, and technology.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store