A Package-based Development Approach for a Salesforce Enterprise Project

In large enterprise projects it is key to have a scalable DevOps approach. We would like to share our experience working on a centrally managed Salesforce project that is being rolled out to over 40 business units.

NatalyaBurova/istock

Introduction

The Salesforce application development lifecycle undergoes a constant evolution. In the past years we have seen major improvements in a range of Salesforce development tools and also an increasing willingness by the community to adopt well known DevOps best practices and traditional development techniques for their projects. In this article, we want to describe how we go about the package-based development and how we try to overcome the limitations of Salesforce that are still remaining. As usual, many approaches and recommendations from us are the result of our preferences and experience, there can and will always be different kinds of solutions or approaches to problems. If you are familiar with package-based development and unlocked packages, we would recommend skipping to the Project Scope section.

Package-Based Development

The goal of this article is not to explain Salesforce development models or aspects of Salesforce Developer Experience (SFDX) in detail, which countless blog articles and the official Salesforce documentation do admirably elsewhere, but rather to take you on a journey through the experience we made with our project. Nevertheless, we think a short introduction to some of the aspects mentioned above may be necessary in order to be able to comprehend some of our decisions.

When it comes to conventional Salesforce development approaches, there are three established development models currently in use, each coming with its own advantages and disadvantages

1. Change Set-based development

The first one is the Change Set-based development model, where the source of truth is a combination of metadata of the target environment and the metadata contained in the Change Set build. It’s typically being used in smaller organizations with small Salesforce teams and needs almost no technical expertise for operation. However, this model should be avoided at all cost, as it does not only support much fewer metadata types to be deployed between environments but also ignores well known best practices from traditional development models like source control.

2. Org-based development

A better alternative with a bit of a learning curve is org-based development, where the source of truth is the Production org. In org-based development you can make use of most of the tools that have been introduced with Salesforce DX and are complementary with package-based development. This allows switching to the latter when project complexity increases, the team grows or you want to distribute your Salesforce application.

3. Package-based development

Package-based development model, the last of the three distinct development models, is also the one we have used in our project. Put simply, the metadata in your repository can be divided into folders, these folders can then be subdivided further into package directories in your sfdx-project.json file. From these package directories you can create packages and distribute them to target orgs. Not only does this approach allow the use of modern development standards, it also comes with a lot of additional benefits like modular development process, management of complex dependencies, and much more.

Unlocked Packages

Unlocked packages are a newer type of packages which enable you to organize your metadata into a container and subsequently distribute it across 1 to n target Salesforce environments. As mentioned above, its source of truth is your VCS, which allows you to benefit from the advantages of source-driven development models.

Another advantage of unlocked packages is that you are able to customize and extend the capabilities of your Salesforce org without breaking any dependencies or impacting future upgrades. You also have more control over functionalities in your Salesforce environments by being able to customize the packages to meet your business needs. Another advantage is that it allows you to break up the logic into smaller independent modules, which can then be developed and tested separately.
This makes it easier to manage the whole development process and strongly reduces the risks of introducing bugs into the applications. Thus, maintenance of the application is easier to handle over time, as you can make changes to each package separately without affecting other packages.

Project Scope

Technical Objectives

Regarding the technical perspective, there were some core objectives we wanted to achieve. First and foremost, we wanted to set up a robust development process and be able to quickly react to any unexpected blockers. As the project was growing over time, it was important to set up for each package an environment that could handle the growing complexity of the requirements and increasing number of pipelines running in parallel. Last but not least, as the project was going to be distributed and installed on different instances, which would build upon our packages, it was important to have clearly defined and comprehensible metadata but also performant and extendable code.

Some of the technical key objectives within the project were:

1. Developing and maintaining unlocked packages according to the project’s requirements and specifications. The number of the packages increases progressively for new business units.
2. Ensuring that the packages were well designed, reliable and the mutual dependencies were clearly defined and considered during the development phase.
3. As the packages would be distributed to different business units, the code needed to be performant, the source of the metadata and the use of it needed to be understandable and we had to consider possibilities to extend the functionalities according to local requirements.
4. Making use of object-oriented programming (OOP) principles to allow the business units to be able to extend the code for their local implementations and benefit from its advantages, such as modular development, writing unit tests instead of integration tests.
5. Setting up a CI/CD pipeline to ensure a robust testing and quality assurance process and automate time consuming steps, while at the same time keeping redundancy of the pipelines low.

Software stack used in the project

Let’s explore next what technologies we’ve used in our project so far. An important disclaimer here should be that, throughout the project, we always worked with the technology stack that was requested by the client. As each technology comes with its own advantages and disadvantages, you should carefully consider what would be the most suitable for your project.

Source Code Management

To manage and version source code of our project, we’ve always been happy to use the most widespread technology available on the market — Git. Our platform of choice to store our Git repositories, which during the project was provided by the client, has so far been Atlassian Bitbucket. It’s a decent platform that hasn’t tormented us with any serious issues yet. It combines well with other development-related services that we’ve used, especially those produced by Atlassian.

DevOps Processes

The next service required by our team was setting up our DevOps processes — the automatic “pipelines” that are executed on a server and that operate on our codebase to achieve various goals like ensuring proper code quality, packaging the project, deploying it to development/testing/production environments, etc.

At first we ran with a Jenkins automation server. It communicated well with our Bitbucket repositories — all our Jenkins pipeline builds and Jira tickets related to a specific Pull Request were properly referenced in the Pull Request’s overview page. It provided a Low-Code way of defining pipelines by configuring them through a Web UI as well as Pro-Code method by writing Jenkins files with Groovy syntax.

Later in the project, we migrated to Atlassian Bamboo. Its compatibility with Jira and Bitbucket was obviously as good as — if not better than — Jenkins’, as all three are Atlassian products. Bamboo also provides decent Pro-Code methods of defining pipelines — namely Bamboo Specs — either through YAML or Java. However, logs of pipeline builds seem more noisy and less organized than in Jenkins — it’s been far more cumbersome to find a bug or a reason for a crash in a failing pipeline’s logs.

Meanwhile, we’ve been using Bamboo for a while and we’ve grown to accept it — we utilize its advantages and we’ve developed workarounds to counter its flaws. However, a migration to another automation tool stack might just be around the corner.

DevOps Utilities

To achieve our DevOps goals, we’ve been using a variety of utilities by either running them on our local machines or inside the pipelines. The most notable ones are:

• SFDXCLI — a command-line utility that lets us interact with our Salesforce orgs and manage our Salesforce DX projects and packages. It’s probably a must-have for any Salesforce DX project. With its help we’ve been able to validate and deploy our codebase onto various testing orgs, run tests on the orgs, create and install packages.
• PMD — a static code analysis tool that helps us to find potential bugs and code smells in our codebase as well as to enforce coding standards. We run it in our pipelines to minimize the amount of bugs and code convention violations introduced into the codebase with each Pull Request. It’s able to scan various Salesforce-specific languages like Apex and Lightning Web Components, and it can parse Apex test runs’ results together with code coverage reports.
• SonarQube — another code quality analysis tool. Bamboo has a built-in SonarQube plugin that lets us run SonarQube scans on our codebase from within the pipelines and visualize the results in the SonarQube dashboard as well as in the Bamboo build results. It’s compatible with PMD and therefore able to be fed with PMD’s results.
• Docker — a containerization tool that lets us run our pipelines in a customized environment. We’ve been using it to run our pipelines in a Docker container that has all the aforementioned utilities like SFDX and PMD installed.
• Node.js — a JavaScript runtime environment that we’ve used occasionally in our pipelines to run some custom JavaScript scripts when we required more complex logic than what would be possible with shell scripts.
• NPM — a package manager for Node.js. We used it to fetch required utilities at the beginning of our pipelines when Docker containers were not a viable option.

Software Stack Recommendations

Having thus explored the technology stack that we have used throughout the project, we can apply our experience to comparing the benefits and drawbacks of all the available technologies — those we’ve used as well as any other alternatives.

Before delving into which tool to use, the first question should always be Low-Code DevOps tools versus Automation tools. Low-Code, Salesforce-specific DevOps tools like Copado and Gearset don’t require much technical expertise and are very admin-friendly but at the same time limited in functionality and customization possibilities. On the other hand, automation tools like Jenkins and Bamboo require more technical expertise but provide more space for customization, provide more functionalities and — in the case of Jenkins — don’t come with any costs except a server for running the service. When a project increases in scale, complexity and specificity, often the simple, universal DevOps solutions provided by Low-Code tools are no longer suitable; the more technical, custom-tailored solutions that Automation tools offer are required.

CumulusCI and Github Actions are two further CI/CD tools that we have worked with in the past and would love to recommend. CumulusCI is an automation tool which is being developed and maintained by the Salesforce NPSP team and can be used in any project to automate the creation of scratch orgs, configuration of SFDX projects, managing of packages and deployment of metadata/packages to various Salesforce environments. It handles Salesforce development processes by giving the user many powerful tools to automate repetitive tasks in developing and deploying Salesforce applications. It has a great documentation which describes in detail how to set up and use CumulusCI commands with examples, and there are also Modules on Trailhead which describe in detail how to set up and work with CumulusCI. The CumulusCI suite comes with several tools and functionalities to enhance your development process:

• Snowfakery to generate fake Salesforce data. You can generate large amounts of fake Salesforce data by creating YAML files where you define your objects and the corresponding fields. It can also be used in combination with other automation tools like Jenkins to generate fake Salesforce data. For that, the tool only needs to be connected to the target environment; it is then able to generate data according to your needs. For example, after authorizing your org, you can create records defined in the Account.recipe.yml file by running following command

cci task run generate_and_load_from_yaml — num_records 10 — num_records_tablename Account — generator_yaml snowfakery/Account.recipe.yml — org <targetEnvironmentAlias>.

Snowfakery documentation is available here.

• Robot Framework is a testing framework which is integrated into CumulusCI, to create and automate end-to-end acceptance tests. After initializing your repository with CumulusCI, you can run

cci task run robot — org dev

and it runs through all of your robot tests. You can see in the terminal if the tests have passed or not. You also get an html output file which you can open with

 open robot/<projectName>/results/log.html

to see the results in your browser.

One big consideration is that much of the CumulusCI functionality only works in combination with Github — it does not support other hosting services like Bitbucket or Gitlab.

The second one Github Actions is a CI/CD software for automating workflows on Github, which allows you to build, test and deploy your code. You can either create your own Github Actions or make use of pre-built actions provided by the community to enhance your CI/CD pipelines. Such Github Actions can be used in workflows, which are defined in yaml and stored in your repository under .github/workflows/. A workflow can be triggered by several events, such as opening, closing or merging a pull request. The workflow then runs the jobs you have defined in the workflow and the steps within each job. Jobs run in parallel by default but this condition can be changed if you need jobs to be dependent on each other, e.g. if you need to trigger job2 after job1 has been successful. Some of the advantages of working with Github Actions are:

• It is beginner friendly and allows you to set up powerful pipelines with ease compared to automation tools like Bamboo.
• You can write rich workflows by using pre-built Github Actions from the community. It eliminates the need for maintenance and effort of developing tools.
• With dependabot you can ensure that actions you reference in your workflow.yml are always up to date, as the bot checks the actions reference against the latest version.
• You can run your workflows in one of the managed environments provided by Github so that you don’t need to manage the infrastructure yourself. We still recommend using self-hosted runners if possible because it gives you more control to adjust the runner to your requirements, whether it is from a processing power perspective — you may need more power or memory to run large jobs or making use of internally used software tools.

Overview of the Solution

Having explored the technology stack we have ended up with, as well as potential alternative technology stacks one could choose, let’s proceed to the solution that we’ve settled on for our project.

Package Structure

We have several packages, some are standalone packages with no dependencies, and some have dependencies to other packages. When a new business unit or feature is being introduced and its functionalities are not handled in any package, a new package is created. As a consequence, the amount of packages constantly increases, which makes it important to have an established process that should not only be able to handle the existing packages and their development processes but also be adaptable and extendable for new incoming packages and pipelines.

In Graph A you can see what our package structure looks like. We have a main package, which contains all necessary functionality for all business units. It’s important to understand that some packages will need to interact with each other in some way; to avoid any potential technical debt you should be mindful of this in designing your package structure. This Salesforce Architects article presents how to use dependency injection design patterns to manage such dependencies. Afterwards, we create a new package for each business unit that reflects their business process. This package usually will depend on the main package to remove redundancy. On top of that we also have standalone packages which don’t need any metadata from other packages. An example for when a standalone package seems suitable is when a standard Salesforce functionality like Salesforce Survey is requested.

Graph A — package structure

In the field, we have come across many different approaches regarding what kind of functionalities a package should cover and how large a package needs to be. We have heard from people who have built more than 30 packages for much smaller projects, and occasionally even up to 60 packages. Such approaches lead to complex dependencies where changes in one package could impact other packages and require the changes to be also reflected in these packages. In the worst case, if in development such anti-patterns as described in the article previously linked to have not been avoided, this would lead to nightmarish technical debt. This would not only introduce redundancy; an aspect many people overlook is that many packages have their own development lifecycle. Thus, an increasing amount of packages can lead to a large technical overhead e.g. by having to set up CI/CD pipelines for packages and documenting manual steps, which come with most packages as many features and metadata types still aren’t supported in unlocked packages and need to be implemented manually either before or after package installation. We would recommend having two main packages: one representing the technical artifacts, such as the logging framework, trigger handlers and so on, which other packages can make use of; and one containing the business logic, common processes, and data models, all of which secondary packages can then build upon. Secondary packages represent business units e.g. logic related to sales process, service process or campaign process. On top of that, we can build standalone packages for features like Survey or Chatbot which do not have any dependencies and can be distributed to business units that want to work with the features.

Branching Strategy

In the project each sprint takes two weeks, with each ticket in a sprint being handled as a feature. Developers checkout from the main branch to a new feature branch, with the name of the feature branch being the same as the ticket number. Each feature then needs to be pushed to the main branch, and after all the prerequisites defined in the CI are met, the feature is merged to main. This automatically triggers a deployment to the Integration Sandbox environment, where the changes are being tested. If the tests were successful, the feature is then marked as ready for packaging. If the feature changes don’t pass all tests, a new feature branch with the same name and a suffix will be created where the code can be altered. After each sprint a new release branch is created that contains all features to be packaged. Features which shouldn't be part of the release branch and are not going to be packaged are cherry picked and removed from the release branch.

Graph B — branching strategy

At the start of the project we followed the Gitflow Model with a main branch and a development branch representing the integration branch. We also had several branch types, e.g. feature, hotfix, bug and so on, to represent the type of the implementation. This complicated the process as it resulted in long living branches and gave rise to frequent merge conflicts that we needed to handle. The number of merge conflicts increased as the number of developers did, and more and more often developers got confused which branch type to use. This ultimately limited our ability to keep our git history clean. Thus, we slowly switched to a trunk-based development strategy. With this, we handle every new change as a feature, whether it is a bug or a hotfix or a feature. For each feature a new branch is created. We try to keep the branches as short lived as possible and run all branches through our CI pipelines during the pull request creation and the merger in order to maintain quality standards but also because it allows the team to process manual reviews faster as the changes are also smaller.

Last but not least, a trunk-based development strategy also helps us to keep our git history clean. As the feature branches are meant to be developed by a small group of people (in most cases only one person) and don’t have any knock-on dependencies, the contributors are free to perform destructive changes to their branch, e.g. perform rebases, resets and therefore also force pushes. This way, unnecessary commits can easily be cut out of the history and faulty commits can be amended before the branch is merged to the trunk whose history shouldn’t be tampered with.

DevOps Pipelines

It is part of our best practices on a project that we try to implement some form of CI/CD practices. According to Continuous Integration, all the changes to the codebase introduced by our developers on their feature branches should be continuously synchronized with the main branch. We also try to keep these codebase changes (as well as their corresponding requested User Stories) as small as possible, so that a merge to the main branch is easy and bears only a small risk of breaking any features or introducing bugs.

According to Continuous Deployment, the state of the codebase’s main branch should be continuously synchronized with the staging environment (Salesforce Org). In the staging environment, various acceptance tests can be performed by our Business Analysts to ensure new features are implemented correctly and no existing mechanics are broken.

Every Pull Request (PR) needs to pass our CI pipeline, where we have defined quality checks that need to be passed to gain high confidence in the quality and functionality of the metadata/code contained in the pull request. The PRs trigger our CI pipeline, where the first step is the creation of a scratch org. The scratch org is created according to settings defined in the file config/project-scratch-def.json, you can find the supported settings and features in the linked Salesforce documentations. Two tips here would be, first of all, to check if the feature you want to have in the scratch org is supported at all. The tricky part is that some features that are supported in scratch orgs are not documented yet, so sometimes you would need to write the name of the feature in the features section and try it out. Last but not least, always define a language in the file to avoid random nonsense errors that are related to not specifying a language. For all our projects, we set the language to “en_US” even if the project language is a different one.

Then we run some apex scripts we wrote to prepare the org for the incoming metadata. For example, there is an issue where the MarketingUser license sometimes is not assigned to the User, and if you have metadata that depends on the license, the deployment would lead to an error. In such cases apex scripts are a great help. You can just run the script in your shell scripts by running

sfdx force:apex:execute -f <pathToFile>

We have several apex scripts that help us in our pipelines to work around Salesforce limitations concerning the treatment of some features and bugs. They provide great flexibility in manipulating settings for our needs and also help us enhance our development process, which we will touch on later on.

Subsequent to that, the Metadata will be deployed to the newly created scratch orgs. It is important to always run all apex tests to ensure that the new metadata works properly in the given project context. We also recommend having some reporting in place to have an overview of the code coverage, though tests should be written to cover all functionality in the apex classes and not just for code coverage!

Graph C — CI pipelines

Then, we run the code through our code analyzers. Specifically, we use PMD for apex and ESLint for our LWC codebase. Any code quality issues will then be forwarded to the owner of the pull request to adjust his implementation according to the suggestions. This helps us maintain code quality standards and reduces the time we have to invest in manual review of the PR.

We have implemented a common formatting practice for our development process, which all developers have adopted locally. By this we also check if the code is formatted according to our standards by using the code formatter prettier. If that’s not the case, a prettier warning will be forwarded to the PR owner to do the adjustments. As we have multiple development teams, having a common formatting style helps us find bugs faster during code review. It also doesn’t interfere with source tracking because of different formatting settings, so that we know exactly which meaningful changes have been applied instead of getting bogged down with tracking irrelevant changes like spaces and so on.

Last but not least, every pull request is manually reviewed according to development standards and guidelines we have defined within the project. The review consists of two parts. First, we review the metadata. As we have different packages, we use different prefixes for each package to differentiate the components. We check if the API name of the metadata has the correct prefix, whether it contains spelling errors, if our logging framework is used correctly in the flows, and so on.
The second part is the code review. These are some of the main points we try to focus on:

• Complexity: can it be done in a less resource consuming way, for example by avoiding unnecessary nested loops?
• Design: can it be done in a more efficient way with less and readable code? Does it follow common best practices e.g. OOP development principles?
• Naming Convention: does the naming follow our internally defined naming conventions? Is this true for both variable names and method names? Also, do the variable and method names reflect their use cases?
• Comments: is the code properly commented and do the comments describe the functionality?
• Tests: do the tests cover all functionality? Does the developer write unit or integration tests? Are tests properly mocked?

We also try to realize Continuous Delivery by continuously packaging various branches of our codebase into Salesforce Second-Generation Unlocked Packages — the same packaging form we deliver to our clients. In Graph C you can see that we run the package creation pipeline once for each pull request. Also, we validate for each pull request if the metadata within the pull request runs smoothly in a packaging context or throws errors. This enables us to handle any packaging errors during the CI process, so that we don’t face any sudden surprises on packaging days after the sprints are done. Since we decided to add a package creation pipeline to our CI process, the time it took us to create packages has decreased significantly.

Graph D shows the whole current CI/CD process of our project. All steps are automated except the Package creation step during the CD process, as after each sprint there are some features which are not accepted and need to be isolated and removed from the release branch.

Graph D — overall CI/CD processes

Challenges

In the following, we describe some common challenges you usually face in Salesforce projects, especially when working with a package-based development model. Salesforce projects can become very tricky really fast. There are some hard limitations to the software which we described in the above sections, which force you to think about workarounds and adjustments to streamline your processes. These challenges also tend to vary substantially, you face different challenges in smaller projects than in big enterprise ones, and different development models also often tend to have their own challenges.

Automation of Time-Consuming, Repetitive and Error-Prone Tasks

Throughout the project’s life and regardless of the automation platform we were using at the time, there were many time-consuming, repetitive and partially manual tasks that we had to perform. To facilitate carrying out this set of frequently repeating chores, we developed our own suite of small command-line tools to automate these tasks.

These tools are mostly shell and Python scripts. Those related to the Salesforce platform would often add additional logic on top of the Salesforce CLI. They were mostly implemented with a view to being able to be used by any developer on any platform — therefore, we tried to stick to what was available in the Python 3 standard library and the POSIX (shell) convention. We also wrote comprehensive usage documentation for each tool. The tools were mostly used by the developers themselves, but we also ran them in our CI pipelines. At some point, this suite of tools had grown big enough for it to be turned into a separate versioned repository.

As the shell scripts became more and more numerous but still shared some common logic, we decided to extract this logic into its own POSIX shell library, which we then drew upon for the end-user scripts. The library contained POSIX-shell-compliant functions for manipulating strings, parsing GNU-style command-line arguments, but also constructs that enabled some sort of array data structure and function-local variables without breaking the POSIX shell convention. Compliance with the POSIX standard was important to us because we wanted to make sure that the scripts would work on any platform without any unexpected behavior. It would probably have been easier to sacrifice this compliance and use a more powerful scripting language like Python from the start, but that’s how our scripts evolved over time and we are happy with the result and experience we gained.

Several exemplary use cases of these tools are: removing obsolete Salesforce flows from an Org; setting up a new scratch Org with specific packages and metadata, based on the sfdx-project.json file and similar configuration files; populating some parameterized metadata with values before deployment, etc.

Last but not least, as you will be working in a terminal most of the time, it will make your life much easier to enhance your terminal environment. For example if your operating system is Windows, you can create a powershell profile and customize your powershell core by changing its appearance to make it easier to navigate through and use external modules like fzf and PSreadLine to enhance your workflow.

Managing Development Process of Multiple Developer Teams

Integrating developer teams, especially in Salesforce projects where you have traditional developers working alongside declarative developers (business analysts, BAs) is always a big challenge, especially in large projects with no use of Low-Code DevOps tools and a very technical devops environment. The goal is to increase the efficiency of the development teams and decrease the time to adjust and implement the development process. Here are some steps we have taken to achieve our goal:

• Quarterly presentations to explain to our BAs the development and deployment process we use and help them understand and integrate.
• Documentation of all our processes, from the creation of a JIRA ticket to the creation of a pull request with all necessary steps in between, to fasten the introduction of new project members.
• Automation of scratch org creation. When a scratch org is created, a new user with the credentials of the developer is automatically created and the developer is notified via email.
• Addition of a scratch org pooling functionality where new scratch orgs are created during nightly runs in order to be able to instantly provide developers with new scratch orgs.
• Automated enabling of Debug Mode in our scratch orgs with a simple apex script to support our developers in their development process.
• Planned implementation of a data seeding functionality for our process to fill scratch orgs with rich Salesforce data.

Managing Complex Dependencies between Packages

As time went by and new feature packages came into existence in our project, complex dependencies arose. Most of our feature packages depended on a core package that provided the behavior and data models that were fundamental to the business domain. Most of the packages including the core package also depended on external framework packages. Different versions of those packages depended on different versions of their dependencies. Feature packages also often depended on the core package’s unpackageable, deployable metadata.

One obvious mechanism we used to manage package dependencies was the sfdx-project.json file. As part of our project we required the specification of explicit dependencies of a given package we developed. However, we also leveraged the packageAliases mechanism to make our dependency specification more readable and maintainable. It thus became clear what package was referenced as a dependency without having to look up the Subscriber Package Version ID. It was also cleaner to look through the git history and commit new package versions — the name of a dependency package did not change, only its version.

Apart from the sfdx-project.json file, we also developed an additional custom configuration JSON file for each feature package that put dependencies of a given package in a wider frame containing not only the package IDs but also installation keys required for their installation, paths to metadata directories to be deployed, as well as paths to anonymous Apex scripts to be executed. This file was subsequently interpreted by our custom deployment script, which was responsible for setting up a given feature package on an Org. This way we could easily keep track of all the dependencies of a given package, and also easily deploy them to any testing or production Org.

As many feature packages had dependencies not only on the core package but also on some of the unpackaged metadata yet to be deployed, we required a solution to cleanly reference the core package’s metadata from within the feature package’s repository. For some time we simply copy-pasted the metadata from the core package’s repository and committed it whenever we released a feature package version which was going to depend on a newer core package version. This however wasn’t an elegant solution — copy-pasting introduced a lot of noise in the git history and needlessly increased the repository’s size. It also required us to be careful when copy-pasting the metadata in order not to miss any file additions or deletions. Later on, we also added additional git submodules. We simply referenced the core package’s repository as a git submodule and referenced its unpackaged metadata directory from within the feature package’s configuration file. Whenever a newer version of the core package was to be depended on, we simply updated the git submodule to point to the newer version. Our Git history was cleaner, our repository size smaller, and our version bumps easy. We only needed to tweak our pipelines to check out the submodules properly, and to make our team members comfortable with git submodules technology.

Dependency on Utilities

There also was a different kind of dependency — DevOps utilities. The aforementioned suite of scripts at first only contained a handful of files stored in our packages’ source repositories next to the package’s manifest and metadata components directory. Scripts performing essentially the same actions but with minor nuances and/or specialized for a different feature package were committed to various source repositories. Many were copied from older codebases with ad-hoc modifications. Code duplication was enormous. Repeating behavior was mixed up with hardcoded configuration details. New improvements in the scripts’ behavior propagated slowly and polluted git histories.

It took time and effort to extract common behavior from the scripts and introduce parameterization, but we ultimately got there. These utilities slowly formed a generalized suite stored on a dedicated repository.

At first, we tried to reference the utilities’ repository as a git submodule in our codebases. However, this turned out not to have been the best idea. The utilities’ code evolved rapidly and we had a multitude of branches in a dozen codebases — we weren’t able to constantly bump all the submodules in all those places, and even if we had been, it would’ve introduced too much noise in the git histories.

Later on, the best idea turned out to be a simple and intuitive one — not to use git submodules at all and explicitly check out the utilities’ repository when needed. We tweaked our pipelines to do this whenever a script was to be executed there, and we also once checked it out on our local machines instead of cd-ing into a specific package’s directory, updating the utilities’ submodule and using a script there.

Implementing and later reverting the submodule approach in this scenario luckily wasn’t too costly, and it was a good lesson to learn — git submodules can be really useful but are not always the best idea.

Unpackageable Metadata Components

Throughout the development of our project, some metadata components in our codebase turned out to be unpackageable into second generation unlocked packages. Such cases were mostly discovered during package version creation — either when one of us tried to create a new package version or when the CI pipeline tried to do so on a feature branch. In either case, the SFDXCLI would throw an error about the metadata component being unpackageable.

In most cases, we couldn’t hack our way around this problem by simply removing the unpackageable components because they were required for our application to work — ultimately they had to be present in the target Org. This meant that they had to be set up either manually through the Salesforce Web UI or by the SFDXCLI, which could be done either by us personally, by an automated script or pipeline of ours, or by the customer’s technical team (which would have to receive appropriate instructions from us).

Deployable Components

For those components which couldn’t be packaged but could be deployed, we created a separate source directory in our codebase. We entered it after the main, packageable directory in the sfdx-project.json file. This way, we could leverage SFDXCLI’s capabilities to deploy multiple source directories at once when deploying metadata to a development Org with force:source:push command.

For all the orgs that require packages to be installed (e.g. staging and production orgs), we created a helper script that parsed the sfdx-project.json file and installed all the listed packages (including dependencies) as well as deploying all the listed source directories in the correct order. This way we could automate yet another part of the delivery process prone to human error.

This unpackaged metadata source directory was taken into account also in our CI pipelines. Thus, the components in it were statically analyzed, validated and included in the test runs.

As we later discovered, the sfdx-project.json file’s JSON schema has a property called unpackagedMetadata to specify a directory for

Metadata not meant to be packaged, but deployed when testing packaged metadata.

However, we could not make use of this property, because as described in the Salesforce documentation, it’s used only for second generation managed packages, and only for Apex test runs.

Undeployable Components — Manual Actions

Some components — or rather mostly some parts of components and some Org configuration — turned out to be undeployable. This meant that they had to be set up manually by us or by the customer’s technical team through the Salesforce Web UI. For such cases, for every feature package in our project, we created a separate Confluence page with a list of all manual actions to be performed before the application can be used.

As time went by, and with new package versions being released, new manual actions were added to the list and some of the old ones became redundant for the new package versions. The Confluence pages became hard to read because one had to manually skip many steps not applicable for their specific case (package version they were upgrading from and to) by reading notes written in a natural language, placed next to each step. To regain readability, we used some Confluence macros to create a table of actions together with a list of package versions each action was applicable for. This way, the reader could easily select the package version (in the table filter) they were upgrading from and to and see only the actions applicable for their case.

Moving Metadata Components between Unlocked Packages

In our project consisting of many separate feature packages developed on separate source repositories, there were cases when some metadata components had to be moved from one feature package to another. We applied two Pull Requests to two separate source repositories — one removing a component from the first codebase and the other adding it to the second codebase (one would probably consider using a monorepo, but it’s not an approach we chose this time).

In a scenario like this, upgrading the first package on some Org meant either marking the component “deprecated” or (depending on its type) removing it completely. Many components do not get deleted. When a component is not deleted, it is still referenced by the first package as “a component that it owns”. This becomes very problematic when we consider an Org that has both packages installed because when we want to upgrade the second package, we get an installation error — the component is still owned by the first package and cannot be “hijacked” by the second one.

In most cases this was only a minor inconvenience — one would simply manually unlink the given component from the first package in the Salesforce Web UI and then install the second package with no issues. However, at one point we faced a serious architectural decision — we wanted to merge several feature packages into one. This naturally meant that we would have to move a lot of components between packages. The decision was made and at first people had to manually click through several hundred components in the Web UI, which took several days of tedious work.

However, after some time we managed to automate this process. We wrote a Python script which utilized Selenium technology to launch a headless browser, log into the desired Org, and unlink all the components by clicking “Remove” buttons in the Web UI, one by one. As time went by, we added more features to this script, e.g. the ability to concurrently run multiple instances of the browser and click random “remove” buttons to minimize the risk of those instances colliding with each other. We also added a feature to automatically fall back to the correct web page if clicking some button redirected to an unexpected page.

This script was a huge time saver. No poor human should ever be tasked with such tedious, repetitive work.

Complex Pipeline Definitions

As time went by, our suite of pipelines increased in number and complexity. There were multiple pipelines achieving the same goal but for different feature packages, and the only differences were the credentials, repository references and some refactoring and/or nuances. To change some behavior in one type of pipeline, we would have to go through a dozen pipelines and carefully introduce the same change to it by clicking through graphical panels in Bamboo’s Web UI. Such cases were cumbersome, error-prone and tiring for us. To keep this suite maintainable and not let it get out of control, we had to employ more advanced solutions.

Back when our DevOps processes were on Jenkins, we utilized Jenkinsfiles, which is Jenkins’ idea of the implementation of the Configuration-as-Code, a source-code definition of pipelines. For correctness-checking pipelines, we put a Jenkinsfile in our feature packages’ source repositories next to the source code. For Continuous Integration, we put multiple Jenkinsfiles (one for each feature package) in a separate “pipelines” repository. These are two different approaches, each with its own pros and cons. For one, only the first approach was officially supported by Jenkins — it is able to read a Jenkinsfile for a source code repository it has checked out, and run the steps specified in the file. For the second approach, we simply explicitly pasted contents of these Jenkinsfiles into the Jenkins Web UI, and Jenkins would make pipelines out of that. The aspect of keeping the pipeline definitions in the repository was our way of not losing track of what were the freshest, most correct pipeline definitions. Every time we made improvements to these pipelines, we would commit the changes to the repository and paste the new Jenkinsfile contents into the Jenkins Web UI to update the pipeline. Both approaches were error prone, but we did not have an alternative at the time.

When we moved to Atlassian Bamboo, we were happy to see that Bamboo also had a solution for Pipelines-as-Code. Bamboo YAML Specs would be a straightforward continuation of our first approach with Jenkinsfiles. However, Bamboo Java Specs were what caught our attention. Initially, we started out defining our pipelines in Bamboo Web UI, but we also took some time to understand how Java Specs were different. Ultimately, we migrated our pipelines to Java Specs.

When you look at it, Bamboo Java Specs technology is simply a Java API for defining Bamboo pipelines, most prominently in the form of a Maven project. It does not matter whether it’s in the same repository as the source code or many repositories away. It can hold a definition of one or many pipelines. Bamboo is capable of automatically reading the Maven project (when changes are introduced) and updating all the pipelines defined in it, but this process can also be initiated from a local machine, as the main method of the project will publish the pipelines to a given Bamboo instance. So even when Bamboo’s automatic reading capability doesn’t suit your needs, you can still publish the pipelines from other JVM-capable systems. Moreover, as JVM is the project’s target, you can potentially use any other JVM language — not just Java — to define your pipelines if you’re brave enough to step into undocumented waters. We have not tried this, but it is certainly possible.

All the above reasons were very compelling to us, but probably the most important reason was that we could now define our pipelines in a Turing-complete language. YAML is just for describing data, while Java has variables, conditionals, functions, polymorphism, (moderately) strong static typing and all the other powerful features we could wish for. We immediately started defining our pipelines as parameterized functions and then calling them with different package-specific credentials and references. To fill gaps not covered by Bamboo’s library, we also started defining helper functions and data structures that used strong typing to ensure even more safety and were written to be easily testable. We kept the code as declarative as possible by balancing between object-oriented and functional style.

For some people this may sound like an overkill. We liked it, though, and it seemed to fit such a complex pipeline suite. However, we don’t rule out that in the future there may appear for us a more preferable solution that embraces simplicity in such a way that would render all this Java stuff overengineering. We are open to change.

Handling Technical Debt

Technical debt can be a nightmare for any project, as with time the complexity grows and fixes or reverting high level architecture decisions can be costly and to some extent even impossible. Especially the Salesforce ecosystem is one of constant change and evolution, development standards known previously can become obsolete very fast, either by change of the platform or an increased awareness of conventional development standards to make use of in Salesforce projects. When we first joined the project, our tests took up to one hour and the apex architecture was designed in a way that a number of core object-oriented programming principles were not followed. As an example, almost all methods were class methods, highlighted by the use of the static keyword. Therefore test methods could not be mocked, there was no clear separation between unit tests and integration tests, and records were being created and inserted into the database in all test classes. As a result, this led to steadily increasing test running times.

To handle this problem, we developed our own mocking framework using the Stub API interface from Salesforce, moved from class methods to instance methods and adjusted our development approach by making more use of OOP principles. We made more use of interfaces and abstract classes to get rid of redundancy and decrease dependencies in our code and moved to writing unit tests which helped us write fast running test methods.

We would recommend implementing a development process in accordance with conventional development standards as much as possible, making use of maintained open source projects like mocking frameworks and trigger frameworks instead of developing your own. Reinventing the wheel doesn’t make sense.
There are many good frameworks and resources out there to help you get a better understanding of these topics. One particular resource I would love to recommend is this blog by James Simone, who is also the maintainer of some great Salesforce open-source projects.

Environment specific data

In most projects, you will be dealing with different Salesforce environments and you will have components that need to have environment specific values if you want to deploy them to target environments. What we did in the past in such cases was to dynamically inject our values according to the target environments with shell scripts and then revert those changes later on. Since SFDX version 1.176 there is a new ephemeral string replacement functionality on push / deploy commands by which you can add a replacement property to your sfdx-project.json file and, within that property, specify what you want to replace for which environment in which file.

Conclusion

As a conclusion, we would suggest implementing a well thought-out DevOps solution regardless of the size of your project, as the benefits will always outweigh the costs. It is also important to consider the technical expertise of your team, only then can a DevOps process be implemented properly so that the team can manage and utilize it completely. In most cases, simplicity is better than complexity, and as time goes on and experience is gained, the DevOps process can be improved and refined.

As pointed out in this article, the technology is constantly evolving, new ideas and solutions continue to emerge. Salesforce tools change and their APIs receive new features, automation tools expand their capabilities and new innovative open-source projects are published. It is vital to the success of any project to be aware of this dynamic and try to keep up with it by exploring all those new possibilities and attempting to integrate them into your process.

The article was co-authored by Paweł Leśniak.