Creating Scratch org pools at Scale!
In this post, I will show you how to create scratch orgs pools using DX@Scale!
Rocket Logistics Scenario
To explain some DX@Scale concepts, I will introduce you to a practice scenario of Rocket Logistics, a global company that uses Salesforce and several salesforce products — Sales Cloud, Service Cloud, CPQ, and a few other managed packages.
They have five salesforce teams: Alpha, Beta, Zeta, Gamma, and Omega.
Both teams contain on average 6 salesforce developers each, and the new salesforce architect Xavier, that just joined that company received the first challenge, to improve their development process.
They have been using sfdx source format, but still struggling with scratch org provisions and long-duration pipelines.
In the first meeting with developers, they listed the top complaints about the current development process:
- Often scratch org creation limit has been reached, and developers have to send messages on Slack channel asking developers to review their scratch org usage and remove unused ones.
- Due to the fact they are using many manage packages, scratch creation is taking a long time to get created.
- Developers are using a bash script to create scratch orgs, but it is only supporting MACOS, however, some external consultants only use Windows O.S.
- Pull request validations are very, very slow! The current pipeline is creating scratch orgs, installing all the manage packages, deploying source code, and finally running the tests; as a result, a simple validation can take up to 3 hours to get validated.
Provisioning among teams and pipelines
Scratch org are limited based on devhub’s edition and additional scratch orgs are available as an add-on.
To mitigate the recurrent lack of scratch org, developers mentioned to Xavier that the company should buy more scratch orgs. However, after some research, he checked on devhub that developers had multiple active scratch orgs at the same time, possibly the issue wasn't the number of scratch orgs available, but the way teams were using scratch org allocation.
As a DX@Scale practitioner, Xavier took some actions to improve scratch org provisioning:
- Migrated project to DX@Scale — In order to use scratch org pools he had to do the initial setup. Migration was easier since DX@scale doesn't force to use only unlock packages, and temporarily they decided to use source packages.
- Created a scratch org pool per team — Before scratch pools, developers used to send messages to the slack channel, asking their colleagues to delete any eventual non-used scratch org; After a specific pool per team, teams owned it, and care more about resource utilization.
- Created scratch org for CI
- Set limit of scratch orgs per developer — Aligned with developers so that each member of the team could have max 2 scratch orgs active.
- Decreased scratch org expiration— As result developers will keep their environment as fresh as possible and reduce the risk of some scratch orgs getting lost for 30 days without use.
- Scratch org pool metrics-By configuring the native datalog integration, he had better insights on the configuration that every team required.
By configuring validation pipelines to use scratch org pools, Xavier was able to significantly improve the performance of merge request validation. The impact of this optimization is clear when looking at the timelines below, which demonstrate the amount of time saved.
Diving into DX@scale concepts
Type of pools
sfpowerscripts allows you to create multiple pools, and define specific configurations per pool.
They are divided into two main categories:
- CI/CD pool — Contains scratch orgs with all dependencies installed and an optionally customized source code.
- Dev scratch org pool — Contains scratch orgs ready to use: external dependencies installed, custom code deployed, data samples, and user configuration. They are used by developers and administrators.
- Snapshot pool — Contains scratch orgs only with packages installed, snapshot pools are used by other pools.
Snapshot pools
Snapshot pools are pools for pools.
If you rely on a lot of managed packages, you could save time for your scratch org pools by creating a snapshot pool.
Most of the time spent on scratch org creation is related to package installations, snapshot pools allow you to save time on scratch org pools creation by creating scratch orgs, and installing package installations.
This allows your regular scratch org pool pipeline works more efficiently, since they will get scratch orgs from the pool with all package installed, and the remaining work is just “delta” change deployments or any additional changes.
Metrics
Incorporating metrics into your DevOps process is crucial for achieving continuous improvement. By utilizing these metrics, you can gain valuable insights into your operations and drive progress. One effective way to track and analyze these metrics is by creating a dashboard on Datadog. With the metrics generated by sfpowerscripts, you can create a comprehensive dashboard that provides visibility into key performance indicators and helps identify areas for improvement.
GitLab Configuration
Sfpowerscripts is an sfdx plugin that can be used with most CI/CD tools. Some examples include GitHub actions, GitLab, and Circle CI. In this tutorial, I’ll show you how to set up a CI scratch org pool using GitLab.
To get started, fork the dxatscale-template repository from GitHub. This repository contains templates for GitHub Actions and Azure, as well as the files we’ll need for our GitLab setup.
Once you’ve forked the repository, navigate to the settings for your GitLab repository and configure the following variables:
- DEVHUB_ALIAS: This should be set to the name of your production org.
- DEVHUB_SFDX_AUTH_URL: This should point to a file containing the sfdx auth URL for your production org. You can retrieve this URL using the command “sfdx force:org:display -u my-devhub-alias -v -j > authfile.json”.
Install or deploy scratch org pool requirement
https://github.com/dxatscale/sfpower-scratchorg-pool)
Next, we’ll create a scheduled job in GitLab to run our scratch org pool. To do this, follow these steps:
- Go to the CI/CD settings for your repository and click “Scheduled pipelines”.
- Click “New schedule” and fill in the following details:
- Description: CI POOL
- Interval pattern: 0 16 * * * (this will run the pipeline every day at 4pm)
- Target branch: master/main
- Click “Save pipeline schedule”.
- Finally, create a variable named “TARGETTASKNAME” with the value “schedule-prepare-ci-pool”.
That’s it! With these settings in place, GitLab will automatically run a scratch org pool job for you every day at 4pm.
Thank you for reading this post! If you found it helpful, please consider subscribing to my blog for more Salesforce-related content. You can also stay updated on my latest videos and articles by following me on Twitter, YouTube, and LinkedIn.
I’m always happy to hear feedback or suggestions for future topics, so feel free to reach out to me on any of these platforms. Thank you again for your support!
https://twitter.com/dieffrei
https://www.youtube.com/dieffrei
https://www.linkedin.com/in/dieffrei/
