Applying the Eventual Consistency Model to Salesforce DevOps
When we talk and write about OrgFlow, we often refer to our eventual consistency based approach to Salesforce DevOps. While this term neatly sums up a big part of what makes OrgFlow unique, it’s not exactly self-explanatory.
I mentioned this term in our previous article, and promised a more detailed explanation on the topic.
So as promised, in this article we will do a deep dive into the ideas behind OrgFlow’s innovative eventual consistency based approach to Salesforce DevOps, and explore how it enables a simpler and more automatic, self-healing deployment process that makes Salesforce DevOps easier, faster and safer.
Comparing Salesforce orgs to Git branches
In any non-trivial software development context, development in isolation is a critical component. Isolation in this context means the production environment remains unaffected by developers’ changes until they have been completed, properly tested and are ready to go live. Isolation also means that multiple developers on multiple teams can simultaneously work on multiple features, without affecting or disturbing each others’ efforts.
For org-based development, Salesforce uses sandboxes to facilitate development in isolation. Most other development platforms use branches — in Git or in a similar version control system — to facilitate this. Let’s talk a bit about the similarities and differences between sandboxes and branches.
Consider the lifecycle of a Salesforce sandbox. From a metadata standpoint, every sandbox is created as a clone of your production org — either directly from production, or indirectly from another sandbox which was itself created as a clone of your production org.
Once created, the sandbox naturally starts to diverge from the production org, either because changes are made in production, or because changes are made in the sandbox, or both.
Each org in your Salesforce account (production as well as sandboxes) is a representation of the same metadata code base. At any point in time, each org may contain a different version of the same metadata code base, but they are still conceptually the same code base. The divergence is temporary; there is an implicit expectation that the orgs will, at some point in the future, once again converge, either through deployments or sandbox refresh operations.
This is very similar to Git works. A repository is a container for a single code base, in some programming language. The repository can have multiple branches which are temporarily diverging versions of that single code base. Each branch is created as an alternate version of your main branch, either directly or indirectly. Branches are typically not expected to exist forever. There is an expectation that they eventually converge, by means of merge operations.
As you can see, Salesforce sandboxes and Git branches are quite similar at a conceptual level. But Git branches have some important features that Salesforce sandboxes lack:
- For every branch, Git has a full detailed historical record of changes, to every line of every source file.
- Using this history, Git knows the exact point in time when any given file started diverging, and exactly what that file contained at that point.
- Using this knowledge, Git can perform a three-way merge operation to understand what changed where.
- Using this understanding, when merging one branch into another, Git can automatically merge any changes that are not in conflict with one another (for example because they happened on different lines). And just as important, Git can detect when changes do conflict, such as when the same line in the same file was changed in two different ways.
This translates to some big differences in functionality.
In Git, a merge from one branch into another is completely safe. No change is forgotten because Git has a record of every change, so all changes to all files are always merged. And nothing gets overwritten by mistake because Git detects any conflicts and prompts the user to resolve them.
In Salesforce, by contrast, a deployment from one org to another using a change set is basically a blind partial copy-paste operation. The user has to know which components were changed, and must remember to include them in the change set. And if one of those components was also changed in the target org, that other change will be lost, overwritten by the change set.
OrgFlow brings all the advantages of Git branches described above, to Salesforce sandboxes, by acting as the “glue” that integrates Salesforce with Git. Your Salesforce metadata is mirrored in a Git repository. Each Salesforce org (production or sandbox) mirrored in a branch in that repository, and OrgFlow keeps each org and its Git branch in sync as an environment. Metadata changes flow through Git and are merged between branches, instead of deployed directly between Salesforce orgs.
Eventual consistency
Eventual consistency is a general computer science concept, used to describe an architectural principle for propagating data changes throughout a distributed system. It’s not typically used in the context of Salesforce metadata deployments, but nevertheless, it is a very appropriate term to describe how OrgFlow approaches the propagation of metadata changes between Salseforce orgs.
To paraphrase from Wikipedia:
Eventual consistency is a consistency model used in distributed computing that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. Eventual consistency is widely deployed in distributed systems, and has origins in early mobile computing projects. A system that has achieved eventual consistency is often said to have converged.
The term, such as it is, describes pretty accurately how OrgFlow functions.
But besides that, the two words also highlight two contrasts between OrgFlow and other tools. “Eventual” because unlike other tools, OrgFlow does not required that the desired result be 100% achieved immediately. “Consistency” because whereas other tools focus on pick-and-choose deployments, OrgFlow actually strives for full consistency across environments.
Let’s look at each aspect in a bit more detail, starting with the latter.
The “consistency” part
As we discussed in the previous section, Git strives for full consistency across a repository. A merge operation in Git takes all the changes (technically all commits) from one branch and integrates them into another. There is no way in Git to merge just some of the commits, or just some of the files, because that would break the linked chain of commits and likely yield an inconsistent result with uncompilable code. (There is a feature called cherry-picking in Git, but it‘s only used in very advanced cases where you intend to discard the source branch.)
In most development platforms outside the realm of Salesforce, where Git is standard practice, this “all-or-nothing” notion of consistency is accepted almost as an axiom.
Developers who use Git typically always strive to keep the branches from diverging too much (by continuously integrating them using forward merge operations) and to keep the lifetime of branches short (by merging changes downstream as soon as they are ready to be integrated). By integrating early and often, developers avoid big complicated merge operations with lots of difficult-to-resolve conflicts.
Yet in the Salesforce ecosystem — even among teams who do use Git-based version control and practice DevOps — pick-and-choose deployments, resulting in partial consistency at best, are more often the norm:
“We need tight control over what goes into a deployment”
Traditionally, Salesforce teams either manually record the changes they want to keep so they can be included in a deployment later, or they base their deployments and Git commits on some form of visual compare from which they pick what to deploy.
But trying to be selective about what you deploy is a source of failures, because things depend on other things. Some tools, such as Gearset, try to compensate for this by using elaborate dependency tracing, deployment validation warnings and tweaks — and they do a good job at it. However, changing a deployment payload “in transit” means that, after deployment, the environments are out of sync, which leads to more work and other potential issues down the road.
We believe Salesforce development should be no different in this regard from Git-based development on other platforms — after all, it all comes down to (metadata) source code in branches in Git repository. We believe the Salesforce DevOps workflow functions much better if it’s designed like a standard Git workflow and embraces the notion of consistency.
OrgFlow enables this type of workflow for Salesforce. By default, when you ask OrgFlow to flow metadata changes between Salesforce and Git, or between different environments, OrgFlow uses standard Git functionality to do it, which means all detected changes are flowed and full consistency is the goal. As an example: if you merge from sandbox A into production, and then from production into sandbox A, then the metadata in the two environments will be identical (except for any undeployable components; more on those later on).
The workflow is not quite as rigid as it sounds, however:
Firstly, not all changes made in a sandbox need to be kept and merged downstream. Some changes are made unintentionally, others as part of experimentation. OrgFlow does in fact support a visual comparison view when flowing changes from an org into its Git branch, where you can selectively exclude such changes and have OrgFlow revert them to reestablish consistency between environments:
Secondly, you choose which subset of your metadata OrgFlow manages in the first place, by configuring the contents of the .orgflowinclude file in the root of your Git repository. This configuration is very powerful and flexible, and allows you to include/exclude metadata at package, namespace, type or component name levels, with full support for glob patterns and wildcards on all levels. Any metadata ultimately covered by your include file will be included in change detection and flow; all other metadata is ignored and untouched by OrgFlow.
The “eventual” part
In the Salesforce deployment engine and metadata API, deployments are essentially a transactional operation, in the sense that a deployment is either completely successful, or completely unsuccessful (rolled back). There is actually a flag in the API meant to allow partially successful deployments, but for the vast majority of deployment payloads, this flag does not work as intended. And even if you could do a partial deployment, what would you then do with the changes that could not be deployed?
This all-or-nothing restriction is naturally reflected in change sets, as well as in all third-party Salesforce deployment and DevOps tools, which all utilize the metadata API. As a result, Salesforce professionals have learned for years to think of a deployment as an operation that must be very tightly controlled and validated to ensure it will succeed in its entirety:
“Every deployment needs to be 100% successful”
This is probably the main reason why Salesforce teams try to limit deployments to contain as few components as possible, and why existing tools encourage them to do so. It’s also the reason why some tools (most notably Gearset) go out of their way to validate the deployment payloads and ensure maximum chance of success for every component.
When we first started sketching out OrgFlow on a whiteboard, we had already been pondering the idea for a while, for a different and much more efficient approach:
Couldn’t we build a deployment tool that tries to deploy everything, records any failures reported by the API, and then retries the deployment again without the failing components, and keeps retrying until it reaches a successful deployment? And couldn’t such a tool keep track of the failed components and use the Git commit graph to keep the changes inside them safe while still allowing them to be updated and fixed, and while keeping them up-to-date with changes from elsewhere?
Such an approach would remove the need for Salesforce teams to be so “surgical” when choosing deployment contents, and instead allow a more automated workflow that strives to eventually achieve full consistency between environments.
We designed and built OrgFlow around this approach. The result: A simpler and more automatic, self-healing deployment process that makes Salesforce DevOps easier, faster and safer, and lets teams spend less time thinking about deployment concerns.
Armed with this new approach, we can begin to challenge some of the conventional wisdom. Deployment errors happen — they are a fact of Salesforce life. The following may sound a bit radical: they don’t always have to be painstakingly avoided at all cost.
In OrgFlow, a deployment is not a one-time all-or-nothing operation. Instead, OrgFlow continuously strives to achieve consistency between your environments over time. OrgFlow keeps track of undeployable components as a first-class concept in a state store, and knows how to keep them safe and up-to-date using clever branching techniques behind the scenes. If one or more components fail in one deployment, it’s not the end of the world: they can either be fixed later and included in the next sync, or you can choose to ignore them. It’s ok.
And in case you’re wondering: You don’t have to wait until after a deployment to find out how much of the changes could be deployed. OrgFlow fully supports a “check only” mode where the same repeat algorithm is used, but where each deployment is a validation deployment only and the merged metadata is not pushed back into your repository. If you decide that enough of your changes were successfully deployed, and whatever components failed can either be deployed later or safely ignored, then you can proceed to perform a real deployment.
Summary
- Deployments in Salesforce and other third-party tools are traditionally an all-or-nothing operation
- Teams have therefore had to be very selective about deployment contents, which has made it impractical to strive for full consistency
- OrgFlow introduces a new workflow based on eventual consistency
- Using this approach, deployments no longer need to be 100% successful every time; failed components are automatically excluded and can be fixed and retried in a later deployment (or ignored if desired)
- As a result, Salesforce teams can adopt a workflow more similar to standard Git-based workflows common in other development platforms, and spend less time on deployment concerns
Don’t take our word for it — try it out for yourself and see OrgFlow’s eventual consistency approach in action. Find out more about OrgFlow on our website, or jump straight in and download it today. OrgFlow is free to try for 2 months — no usage limits, no strings attached, no credit card or billing information required.
