Find Bugs Earlier with Second-Generation Packaging

Salesforce.org is the social impact center of Salesforce, building powerful technology for, and with, nonprofits, schools, and philanthropic organizations. We’re part of Salesforce, and we work like an independent software vendor (ISV), building amazing products on top of the Salesforce platform to help enable positive change around the world. As a model ISV, we participate in the open-source community and demonstrate best practices in the Salesforce ecosystem.

Building a trusted development lifecycle

One of our key goals in defining our software development lifecycle (SDLC) is to shift risk as far “left” (early) in the process as possible. Bugs caught early are cheap to fix; bugs caught late are very expensive to fix; bugs caught in production are worst of all, because they impact our customers and damage trust.

We can summarize a testing and development process that maximizes our ability to mitigate this risk:

We test every change in a realistic environment that is equivalent to production, before the change is promoted beyond the initial development phase.

When we fail to meet this ideal, we shift risk “right” (later) in the process. And by doing so, we heighten that risk and its potential cost, ultimately even to the point of imperiling the trust of our customers.

Legacy processes, infrastructure limitations, and technical constraints all can hinder this idealized development practice. As we scaled our product lines and development velocity, we encountered key bottlenecks that limited our ability to shift risk left in feature testing and in end-to-end testing. To eliminate these bottlenecks, we turned to a novel application of the Salesforce platform’s newest delivery technology: second-generation packaging.

What are packaging, 1GP, and 2GP?

Packaging is a Salesforce platform technology that enables metadata to be packaged into a discrete artifact that may be installed in many orgs. Packaging is a solution both for migrating changes and for distributing products to customers. It underlies the AppExchange. Packaging comes in first generation (1GP) and second generation (2GP) flavors, which have different capabilities and limitations. While there’s lots of depth to packaging, a top-level difference between 1GP and 2GP is the source of truth. In 1GP, the source of truth is a packaging org, while in 2GP the source of truth is version control.

You can learn more about second-generation packaging in the Salesforce DX Developer Guide. The architect’s guide to Migrating Changes explores the pros and cons of many different migration and distribution strategies.

Identifying legacy development process risks

A representative ISV development process might look much like ours did in 2020:

• Developers and product managers collaborate to design and build new features. This development takes place in a feature branch in the version control system, separate from the production code delivered to customers.
• As a feature is developed, it’s tested along the way. At this stage, the team can only test changes to the product in an “unmanaged” form. The package that will really ship to customers hasn’t been created yet, which means the process does not meet the ideal state we talked about above. Since testing does not involve a real packaged version, there’s space for bugs to slip through.
• Well-tested features move forward and are integrated into the main branch, representing the production code base. Then, package versions can be built, allowing for regression testing in a production-like environment. For first-generation managed packages in particular, it’s critical to preserve the integrity of the main branch and the packaging org.
• Lastly, customers receive a released version of the package incorporating the new changes.

The limitations of this process are particularly severe for ISVs like us who support existing subscriber bases on first-generation packages. However, any SDLC that gates package builds or deployments to production-like environments upon a single branch is similarly limited: risk shifts right, and some bugs can only be found late in the development process.

Here, we’re missing the mark we set in our ideal process definition above. Because regression testing is serialized on merges into the main branch, we’ve also created a bottleneck that constrains our ability to scale development horizontally across multiple teams.

The challenges of end-to-end testing

A product can be more than a package. That’s why we practice what we call the Product Delivery Model, which centers the idea of a product as delivered to a customer, in all its complexity. One or many packages, unpackaged metadata, potentially off-platform services, and setup automation all fall under the umbrella of the product.

End-to-end testing means testing a change to one component of an application in the context of all of the other components of that application. Take the Nonprofit Success Pack (NPSP), our flagship product for nonprofit organizations, as an example. The product includes six NPSP managed packages, plus unpackaged metadata, plus the Elevate donation platform, plus two more managed packages that connect NPSP to Elevate. That’s a lot of moving parts, and a lot of potential impact from changes!

When we think about end-to-end testing of a complex product, as articulated in the Product Delivery Model, a higher-order source of risk emerges. When a feature includes development in, for example, both the Nonprofit Success Pack managed package and the Elevate NPSP Connector managed package, which depends on NPSP, how do we shift testing of that development left?

A look at the process exposes challenges that resemble those found in testing a single package, but at higher scale and with greater impact.

• Across all of the components of the product, development takes place in individual feature branches.
• Each package or component’s changes are tested in isolation. In some cases, especially with features that have cross-package or off-platform dependencies, it’s impossible to fully test functionality at this stage. Even worse, some types of dependency prevent even feature testing from taking place for extension packages until a full release is completed in core packages.
• Only once development is completed, merges are completed, and package versions are created can end-to-end testing take place. This entails assembling the latest versions of each component into an environment and performing end-to-end tests there. If any bugs are discovered, the process has to begin again — starting with the top of the dependency hierarchy and moving down.
• Lastly, once all of the tests are successful, the release and delivery process can proceed.

These limitations force us to serialize changes to our packages along the dependency hierarchy, with core packages going first, and extension packages following. And it pushes all of the risk of end-to-end testing rightwards, with problems in the first package potentially going unobserved until development completes on the last package. Not only does this challenge weaken our testing program, it also limits our development velocity!

Testing with second-generation packaging

One of the great innovations of second-generation packaging is the ability to create packages (the real artifacts that are delivered to customers) across multiple code lines or branches of development. First-generation packaging is limited to one primary code line, represented by the state of the packaging org. With second-generation packaging, we can create a package version from any state in our product’s development, including development taking place on multiple features in parallel.

Coupling this capability with pervasive automation enables us to shift more and more risk left across the development lifecycle.

Shifting feature testing left

We have a deep automation infrastructure built around CumulusCI, our open source framework for orchestration with scratch orgs, and MetaCI, our continuous integration service. We built second-generation packaging into our automated processes to help us shift testing left. Here’s what our development process for a first-generation package looks like after this transition:

• Development takes place in a feature branch in our version control system, separated from the production code that we are currently delivering to customers. As the feature is developed, CumulusCI and MetaCI continuously build second-generation package versions from every single commit — a real artifact similar to what we will ultimately ship to a customer, with the same namespace as our production package.
• We test the second-generation package version in continuous integration and via manual QA, giving us clear, consistent results reflecting what this work would look like when we ship in production. Using a real package at this level gives us a better picture of our customers’ experience and enables us to catch more bugs early in the development lifecycle.
• Once a feature has completed testing, we can move forward confidently with integrating the changes into our production (main) code base. Then, we create a beta version of our package, and execute a final regression test pass — one where we can be more confident that critical bugs have already been found early in the process.
• Lastly, once all final tests are completed, we create a release version of our package, and deliver it to our customers.

We’ve moved testing of our products to much earlier in the process using the same kind of environments that are used when they’re delivered to customers. This is a critical shift that makes it possible to catch whole classes of bugs during early development that previously would have gone undetected until much later. We’ve also shielded ourselves more effectively against creating 1GP beta package versions that don’t meet our standards and require more costly remediation, and we now have the ability to create packages for testing across many parallel development efforts.

The combination of second-generation packaging with CumulusCI automation and continuous integration unlocked tremendous value for us as an ISV, but we discovered, with critical insights from our development teams, that we could go even further with this process.

Shifting end-to-end testing left

To support parallelized development in our Product Delivery Model, we rebuilt CumulusCI’s dependency engine to take into account our new capability of building second-generation packages for every single change to every package. Then we discovered that the solution we built to help development teams scale horizontally also offers solutions to the end-to-end testing challenges that arise from the complex interaction between the SDLC, packaging, and dependency management.

Our revamped dependency engine enables our development teams to shift end-to-end testing left, into the initial development phase. We make it easy for them to build functionality in two, three, or more separate packages that interoperate, and then create integrated scratch org test environments that contain those changes as second-generation packages — before any work is merged, anywhere.

Here’s how it works:

• Across all of the components of the application (packages), development takes place in feature branches. Development teams use shared naming conventions — like feature/improve-customer-experience — across packages to correlate cross-package features in development.
• On each commit, continuous integration builds a namespaced, second-generation beta managed package version using the Skip Validation option. This yields a package artifact that’s very similar (but not identical) to the packages we ship to customers, and using this specific package type offers us more capabilities we’ll discuss below.
• Test builds create orgs that contain these package versions across a whole product. For example, if we have a feature/improve-customer-experience in development across packages A, B, and C, then our test builds will include the second-generation managed packages built from that feature branch, from all three packages.

Our use of an unusual package type — namespaced, second-generation beta managed package versions with Skip Validation — makes these capabilities fast and flexible, while still allowing these package versions to serve as an effective test representation of the real package artifacts we’ll ship to customers. These packages’ dependencies are validated at the time of installation, rather than the time of creation. As a result, we get the freedom to mix and match versions in an org to represent the state of development across any combination of branches and any number of packages.

This means we can create orgs suitable for performing end-to-end testing before we merge any changes, anywhere in the product. And it’s easy and automatic: a single command or CI build creates an org with the changes from feature/improve-customer-experience across packages A, B, and C, all in packaged form with production namespaces. This org offers a realistic, near-production environment for end-to-end testing.

We’re now able to catch bugs that previously might have stayed hidden until the final stages of end-to-end testing or prerelease testing, right at the beginning of the development process. This capability frees our teams to experiment and innovate without risk to our production code base, keeps bug fixes easy and inexpensive, and further protects our customers from the possibility of flaws in our software. And since our process works the same way for products we deliver with 1GPs and products we deliver with 2GPs, we get a consistent, high-value testing experience across all of our products.

By applying second-generation packaging, we’ve gotten much, much closer to the ideal with which we began:

We test every change in a realistic environment that is equivalent to production, before the change is promoted beyond the initial development phase.

We’re equipped to deliver the quality products that we expect of ourselves and preserve the trust of our customers. And what’s more, we’re preparing for the migration to second-generation managed packaging today. We’ve already built and tested our products using second-generation managed packages, our build pipeline is already humming in production, and our infrastructure is ready to adopt the latest platform technologies.

Conclusion

Second-generation packaging unlocks new capabilities that can benefit every ISV — even those that deliver first-generation managed packages! Automated testing and end-to-end testing using 2GP has brought numerous benefits to our development process:

• We’ve moved risk left by testing in customer-like environments before we merge any code.
• We’ve added a completely new capability: testing multiple dependent products, end-to-end, before cutting any releases at all.
• We’ve improved our ability to scale and parallelize development work across large, complex package suites.
• We’re more prepared for future migration of first-generation managed package products to second-generation packaging.

Learn more about CumulusCI, second-generation packaging, and automating the development lifecycle:

• Documentation about how this process works
• CumulusCI documentation
• Build Applications with CumulusCI on Trailhead
• Package Development Model on Trailhead
• Second-Generation Managed Packages in the Salesforce DX Developer Guide
• Flexible Version Management | 2GP Deep Dive on Youtube

About the Authors

David Reed is a Lead Member of Technical Staff in Release Engineering at Salesforce.org, where he works on the CumulusCI automation framework and products like the Nonprofit Success Pack, Education Data Architecture, and many others. David is 9x Salesforce certified and is a moderator at Salesforce Stack Exchange. He lives in Denver, Colorado.

Brandon Parker is a Senior Member of Technical Staff in Release Engineering at Salesforce.org. He currently works on the suite of tools that the Salesforce.org Release Engineering team maintains — including CumulusCI, MetaCI, and MetaDeploy. He lives in Portland, Oregon.