Adobe at Spinnaker Summit 2018: Shredding Stateful Systems with Spinnaker
The following is a summary of our talk at Spinnaker Summit 2018 on Oct. 8.
The Adobe Experience Cloud is built to bring together data and content to deliver exceptional experiences. We help many of the world’s largest and most trusted brands to inform, engage, and delight their customers.
Data is the underlying language of these experiences. And the data scale we must manage is truly exceptional — with more than 250 billion customer touchpoints every single day.
Doing this takes a lot of infrastructure — across a lot of data centers—all over the world. In fact, our systems are spread across two third-party cloud hosting providers as well as Adobe-owned data centers.
We need powerful tools to manage all of these systems, and one of them is named Spinnaker. Spinnaker helps us manage large fleets of edge computing, data collection, and delivery systems and respond to variable demands for our services — lots of scaling up and down (though mostly up!).
In the talk we gave at Spinnaker Summit, we presented on one particular aspect of how we use Spinnaker: leveraging it to manage stateful systems.
Usually, Spinnaker is used to manage what we call stateless and immutable infrastructure, which never changes from when it is deployed to when it is torn down. However, for our edge systems, we also need to store state — the log of customer interaction with our services — locally on the Spinnaker-managed infrastructure and we can’t afford to lose any of that precious data when the system is torn down.
So, we invented Shredder for EC2: a new side-car open source service that runs on those local systems, which are managed by Spinnaker and stateful in nature. Shredder allows us to safely tear down these edge systems, and clean up and collect the stateful data they contain, without losing a single precious bit.
What challenges did we face before deploying Spinnaker?
We believe development operations, commonly known as DevOps, must be considered in building a platform fit for use by enterprises in the Experience Era. DevOps capabilities in a platform enable the monitoring and optimization of:
- Collecting new data and segmenting for the next best action according readily available new data from real-time interaction with the digital properties and surfaces.
- Sending this new real-time information from a centralized cloud infrastructure to the edge regions to reduce compute latency, accelerate insights, then deliver the experience.
- Publishing aliasing information into Adobe’s Pipeline Services in order to be available to other experience delivery applications and services.
- Publishing data to authorized second- and third-party data consumers for experience delivery.
- Operational burden… hours of work with multiple engineers (rolling window deployment)
- Unguaranteed consistency
- Difficult rollbacks for our incidents
- Unscalable infrastructure, limiting size and future growth
We needed to architect Adobe Experience Platform with DevOps capabilities to:
- Provide consistent configuration through an immutable infrastructure.
- Deploy easily with fast rollback capabilities.
- Deliver continuously through self-serviceable mechanisms.
Why and how did we deploy Spinnaker?
We chose Spinnaker because it is an open source, multi-cloud, continuous delivery platform that allowed us to release software updates with high velocity and confidence. We can control from a single Spinnaker console to bake and deploy resources in our cloud accounts. We create applications, which in turn can contain multiple pipelines and tasks. We group multiple applications into a single project.
Adobe Spinnaker infrastructure is:
- Ported from Ubuntu to CentOS7
- Deployed using Terraform and Puppet
- Provides high availability for Spinnaker services
- Architected with two environments: stage and production
- Integrated with our services (Prometheus, Slack, Grafana, and Jenkins)
We will be using Spinnaker in Adobe Audience Manager (AAM) for multiple instance types for baking AMIs, performing blue/green ASG deployments, creating autoscale policies, and destroying infrastructure.
Since we use Puppet as our main configuration management tool, most of our apps and configurations are stored there. We have created a “base” application in Spinnaker PROD that is used to bake the AMI by using the Puppet base role. This AMI will contain all mandatory applications for our platform. Using such pre-baked AMI will help us in deploying applications, even if we are not going to use a puppet client upon deployment since it will contain all needed apps to integrate with AAM infrastructure.
What were our steps with Spinnaker?
Here are the steps we took with Spinnaker:
- Dynamic tool for instance lifecycle—what we call Provisione
- Dynamic configuration based on region and environment
- No package installation at bootstrap
- Limited and fault-tolerant external calls
- ASG lifecycle hooks
- Hybrid health check ELB/EC2
- Infrastructure and configuration validation — ensuring consistency
- Dynamic waiting for service health check
- Service warmup
- Automatic node replacement for failed bootstrap
If you missed us at Spinnaker 2018, you can still check out our presentation here.
If what we are sharing is exciting to you, join us. Adobe has developer and data scientist opportunities.
 Adobe Spinnaker Presentation: https://spark.adobe.com/video/cE3qNxKYj9FO5
 Adobe Experience Cloud Audience Manager primer: https://spark.adobe.com/video/cE3qNxKYj9FO5
 Spinnaker: https://www.spinnaker.io/concepts/
 Adobe Shredder for EC2: https://medium.com/adobetech/adobe-cloud-platform-open-sources-shredder-for-ec2-f43fe21b1b11