Introducing OrgFlow — True Git-based DevOps for Salesforce

Welcome to the very first article in our brand new publication about Salesforce DevOps in general, and our new product OrgFlow in particular. My name is Daniel Rosenberg and I’m the managing director of OrgFlow, which is a relatively new company that I co-founded together with my partner Chris Payne in early 2020.

Our mission as a company is to try and change the way Salesforce teams think about DevOps and deployments, into something that more resembles the practices and techniques that are commonplace in the rest of the software engineering industry today.

Salesforce professionals have learned over the last 5–10 years to think of deployment and DevOps a bit differently from those working with other platforms, mainly because of limitations in available tooling. We want to challenge some of those conceptions by introducing new tooling that enables a new type of workflow.

In this article we will explain why the Salesforce ecosystem needs an entirely new DevOps tool, and why we decided to build it.

Deployments in Salesforce

It’s not a secret, so I’ll just say it: although Salesforce is a very powerful and customizable platform, when it comes to versioning and deploying those customizations, Salesforce leaves a lot to be desired. For traditional org-based development (i.e. not package-based development) Salesforce gives customers a few things out of the box:

1. The SOAP-based metadata API, through which metadata customizations can be retrieved and deployed.
2. Change sets (built on top of the metadata API) which allow you to pick a set of components in one Salesforce org and essentially copy-paste them into another Salesforce org.
3. The Salesforce CLI which is capable of retrieving and deploying metadata to and from a local directory using the metadata API, as long as you tell it exactly which components to retrieve and deploy.
4. Recently, a source tracking API has been added to the mix, which allows you to query a sandbox for a list of metadata components that have been changed during a given timeframe.

And that’s pretty much it.

Now, if you are a developer or DevOps engineer with any amount of experience from other modern development platforms, languages and tools, you will quickly notice that some things are mysteriously missing. Unlike most other platforms, the built-in development and deployment story in Salesforce has no concepts of:

• change detection
• branching
• merging
• pull requests or similar review functionality
• conflict detection and resolution
• version history

As a result, if you’re using the built-in functionality of the platform, not only do you need to remember and manually identify every component your developers and admins changed in order to include it in a deployment; more importantly, if you develop in more than a single environment, every deployment also risks overwriting one set of changes with another. Not to mention how tedious and error-prone the process is, or how quirky and unpredictable the metadata retrieval and deployment engine can be.

When our team first got into the area of Salesforce DevOps, we came at it from two different backgrounds:

On the one hand, we had many years of experience doing contract work around DevOps, software development and architecture on other platforms. Industry-standard version control, automation and deployment techniques had been second-nature to us for a long time.

On the other hand, we had just spent a year-long project doing Apex, VisualForce and Lightning development on a large Salesforce implementation for a client, and had been shocked to learn there was an entire ecosystem around Salesforce where the benefits of Git version control and open standards-based CI/CD process automation, that we have relied on in the rest of the software industry for the past 10+ years, were just not commonly used.

Building a Salesforce DevOps solution

About three years ago, as a follow-up to the Salesforce development project we had worked on, we were asked by the same long-time client to help them design a Salesforce DevOps solution.

As a team with several seasoned DevOps engineers and developers with experience from both Salesforce and other platforms, we soon had a clear vision for what we wanted our DevOps pipeline to look like. We knew that changesets would be useless, and we knew that Salesforce DX was not the solution for us (more on that below). So the team spent months together with our customer, carefully evaluating other third-party deployment tools like Copado, AutoRABIT, Gearset, ClickDeploy, Flosum and others.

Many of these offerings are good tools in many ways, but for our purposes, we found that they were lacking in some important ways:

• Some are too all-inclusive, trying to be closed-off one-stop-shops for Salesforce Deployments by providing everything to everyone, and in doing so, end up reinventing functionality that’s already done much better by existing CI/CD tools, and forcing teams to use a single compromise for every part of their DevOps workflows.
• Some are quite rigid in the types of workflows they can support. One example of this is not allowing changes to be made directly in the customer’s production org.
• Most are very web UI-oriented, which means they are optimized for humans clicking buttons, but not so great for automation, scripting and integration.
• Most treat Git version control either as completely external to the tools’ workflows (for example, as just one type of source or target for a deployment) or as a completely embedded store behind the scenes that customers can only interact with through the tool itself.

The first point above is perhaps the most profound. Teams with experience from other development platforms are accustomed to using popular industry-standard tools and CI/CD platforms to drive their DevOps pipelines — tools such as GitHub, Azure DevOps, Jenkins, TeamCity and many others. And they typically already have them set up in their companies and are already using them for a bunch of scenarios where other, more domain-specific components are typically expected to hook into these CI/CD platforms in one way or another (either by providing a CLI that can be scripted against, or by providing a REST API).

Yet, in the Salesforce universe, that has not been practically attainable until now, neither with the built-in functionality of the platform, nor with any of the third-party products. We wanted our client to be able to take the investments and skillsets they already used to do DevOps on other platforms, and employ them the same way to build their Salesforce DevOps pipelines. It soon became clear to us that the existing offerings were not designed for that.

We simply needed something different.

Something that’s easy to use hands-on for a deployment manager, but that can also be scripted, automated and integrated into the CI/CD platforms DevOps teams already love and use. Something unopinionated that doesn’t impose restrictions on the kinds of workflows teams can build. Something that puts all the goodness of Git at the core of the process, not just in the periphery as an after-thought. Something that makes sense to seasoned developers and DevOps engineers.

So we built OrgFlow.

True Git-based DevOps for Salesforce

OrgFlow is a tool that opens the Salesforce platform up to modern software development, version control, deployment and automation techniques. It is essentially the glue that integrates any industry-standard Git and CI/CD tooling with Salesforce, and helps teams build DevOps workflows that make Salesforce deployments easier, faster and safer.

Let’s talk about some of the things that make OrgFlow different:

In OrgFlow, each sandbox is a Git branch, so customizations always flow through Git instead of directly between Salesforce orgs. This fixes many of the platform’s deployment shortcomings. No change is forgotten, no change gets lost, no change gets overwritten. Conflicts are either automatically resolved, or clearly surfaced for the deployment manager to handle, using their favorite merge tool. And because we normalize all metadata, deployment managers only have to bother with real conflicts — not the ones caused by formatting differences or arbitrary reshuffling of XML elements.

We built OrgFlow as a user-friendly and automation-friendly CLI so that you can choose how you use it. Download and use it on your Windows, macOS or Linux machine. Integrate it into your favorite CI/CD platform. Run it in a Docker container. Or do all of the above — your engineering creativity is the only limit.

Unlike other solutions, OrgFlow embraces principles of eventual consistency. Deployment errors happen — they are a fact of Salesforce life. The following may sound radical: they actually don’t always have to be painstakingly avoided at all cost. If the tooling is clever about it, they don’t always have to block a deployment. OrgFlow keeps track of failed components as a first-class concept, and knows how to continuously retry and safely deploy the rest. If one or more components fail in your deployment, it’s not the end of the world: they can either be fixed later and included in the next sync, or you can choose to ignore them. It’s ok.

We plan to go into more detail about OrgFlow’s eventual consistency approach to deployments, as well as other unique features of OrgFlow, in upcoming articles.

What about Salesforce DX?

Because OrgFlow is delivered as a CLI and makes a big point of being scriptable, automatable and integratable, many people wonder how it relates to Salesforce DX, which seems to check those same boxes.

The answer is that OrgFlow and Salesforce DX are meant for two completely different development scenarios. Generally, customizations in Salesforce can be done using two very different models:

Org-based development

Org-based development is when you simply customize your org’s metadata to fit your needs, whether that means changing existing components or adding new ones, without worrying about the dependency tree or how your changes relate to all the other metadata in your org.

You either make customizations directly in your production org, or you create a sandbox as a clone of your production environment, make and test your customizations there, and then deploy them to your production org in some way.

This model is powered by the metadata API, and surfaced through the built-in change set functionality, as well as through ANT, parts of the Salesforce CLI, and several third-party tools.

Package-based development

Package-based development, by contrast, is when you build your customizations, usually using scratch orgs, into a self-contained shippable package that can be installed into any org.

This approach is mostly used when your customizations consist mainly of new metadata components, and can be grouped into cohesive units (packages) where everything in a package depends only on things inside the same package, or on other packages. A package can be developed in a scratch org and then installed into your production org, or it can published on AppExchange so it can be found and installed by other companies.

The package-based model is mostly used for greenfield development. If your team has the luxury of doing only package-based development, then Salesforce DX and the Salesforce CLI will most likely satisfy your DevOps needs and address most of the shortcomings of the platform’s built-in deployment story.

Package-based development brings many advantages, but for many organizations, fully transitioning to package-based development is unrealistic. In particular, for orgs that have been around for a long time (maybe even since before the concept of package-based development was introduced) and orgs with a significant amount of customization, it can be very difficult to break apart metadata customizations into cohesive sets that can form self-contained packages.

Best of both worlds

As mentioned above, the reality is that many teams — large and small — are doing org-based development for good reason, and we don’t think they should be forced to change this just because they don’t have adequate tooling to support them. The Salesforce CLI does provide some rudimentary support for working with sandboxes and the metadata API, but not nearly enough to really support true Git-based DevOps.

That’s where OrgFlow comes in, providing the missing piece of the puzzle for those teams.

OrgFlow is designed for org-based development. In essense, OrgFlow provides similar capabilities for org-based development, to what Salesforce DX provides for package-based development. Additionally, OrgFlow goes much further and provides several benefits not achievable with Salesforce DX.

So what now?

After a year of intensive production use and continuous refinement, we were delighted to see how well our vision and the tooling we built actually worked in real life — and so were our customers.

And we figured, if it was useful for our team and our existing customers, it would probably be useful for others, too. After all, our Salesforce team can’t be that unique; there must be other teams out there who share the same sets of priorities when it comes to building their Salesforce DevOps solutions. So we spent another couple of years turning our solution into a high-quality, enterprise grade software product.

That product is OrgFlow, and it is now available for you to try.

OrgFlow is free to try for 2 months — there are no usage limits, no strings attached, no credit card or billing information required. If you or your team would like to give it a try, just head over to our website at www.orgflow.io where you can download OrgFlow and get started.

We also encourage you to get in touch with us by joining us on Slack. Our Slack community is the best place to connect with our team, and with other OrgFlow users. Join us to discuss OrgFlow-related topics, exchange ideas or get help with specific issues. Our team hangs out in these channels on a daily basis, and someone from us will always engage with anyone who needs help.

We look forward to helping Salesforce teams use OrgFlow to make their deployments easier safer and faster, and we are excited to see what kinds of DevOps workflows teams can build using OrgFlow.