Why you should use scratch orgs — A Real Journey
If you want to make your team productive, give them a good development environment.
However, it can be challenging in salesforce, depending on your codebase and dependencies.
In 2007, When I began with salesforce development, one of the most annoying things was the effort to have a good salesforce environment to test/develop.
It was common to use shared sandboxes, with entire teams changing the shared components in the same environment.
Disclaimer — Scratch orgs are not a silver bullet, but this post aims to show the main reasons to use them.
Commons issues with shared sandboxes
- Obsolete Data — Data gets outdated quickly and doesn’t often get refreshed.
- Seed data — Developers can spend a lot of time trying to reproduce a scenario due to the lack of knowledge of a domain.
In large projects, usually creation of certain records like Opportunity, involves a lot of configuration.
It can get even worse when onboarding new team members. - Conflicts — When you get some issue (like trigger failing, apex not compiling), you never know if the issue is related to your changes or from others 6..7..10 colleagues.
Then what do you do? First, check if it is your error (1h? 2h?); Second, Send messages to the team to ask to fix it and wait for a response (1h? 2h?) and then finally fix it (1h, 2h), you will see how much you spend time only due to environmental issues. - Out of update environments — Sandboxes are very difficult to keep in sync; one day, you miss deleting a field, tomorrow a report, and after some months, a lot of non-relevant metadata/code is still there without any tracking — As a good practice, sandboxes have to be refreshed often.
ChatGPT opinion:
Therefore, Salesforce launched SFDX.
One of the main features is the possibility to create environments from “scratch” based on source code (source of truth).
You don’t need anymore to copy the environment from production for development purposes.
Key benefits of using scratch orgs
- Ephemeral environment — Yes, they don’t last longer than sandboxes. Scratch org lifetime can be from 1 to 30 days.
What is the benefit of it? It helps to ensure that developers always have the latest code version and are not dependent on any environment. - Source of truth — It is created from the source, which is great because it gives you a lot of flexibility. For example, you can have multiple environments quickly from different versions.
- Quick spin-up of the environment — There are no long queues to wait for your environment to be created (like when creating a sandbox). (only scratch org creation takes seconds, of course, without customizations and code)
However, unlike sandboxes, scratch orgs are created empty. It is your responsibility to prepare them for use.
Our journey
When I began using salesforce scratch orgs, The first approach to provide scratch orgs was creating a simple bash script similar to the one below:
However, this approach became very hard to maintain:
- The script was running on the local machine — Many things can happen during scratch org provisioning: Network Issues, Incorrect SFDX dependency, Code deployability, etc…
- Non-developers had difficulties to setup project requirements: sfdx plugin and requirements (sfdx, java, plugins), devhub authentication, GitLab authentication, CLI commands, etc…
- Long Durations — when the project began, we had only one external dependency (CPQ), but after some months, we got another five packages which took almost 2 hours to install.
- Lack of scratch orgs — Developers created a lot of scratch orgs and often forgot to delete them after use, quite often, we used to hit scratch orgs allocation limits. (Pro tip: Don’t use long expiration dates)
- External packages installation
If your org is dependent on any external package (managed, unlocked…), you might have this installed in your scratch org before pushing source code.
When you are using 1 or 2 packages is fine, but when you need to control multiple external dependencies and install them, it is easier to use a plugin such as Texei plugin, which can orchestrate all dependency installs. - Scratch didn't have enough data — In order to test any functionality, you need the configuration and maybe some test scenarios already available.
for instance, CPQ configurations require a lot of configuration; developers got frustrated, losing a lot of time doing it.
I’ve tried to use sfdx CLI standard command to deploy data, but it is just too basic! The first bottleneck you will find records that require record types.
Provisioning scratch orgs with excellence
Providing easy access to scratch orgs for all users
What helped us to allow to provide easy access for non-developers users was Hutte. It is a web application that allows the team to collaborate: developers, admins, product owners, testers etc...
For Admins — The benefit is that they don't need to install anything, log in to Hutte, grab a scratch org, do the changes and click on the button to generate a pull request. Brilliant!
For Developers — It fixed networking issues during scratch org creation; the scripts for scratch org creation didn't run locally anymore, which makes it possible for developers to have a new scratch org in seconds. (It is required to install an sfdx plugin to authenticate scratch orgs).
Nowadays, we use sfpowerscripts — This topic is so big that I’m writing a few posts about the migration to it.
Populating data
As an architect for a consulting company, one of the most annoying things was to wait for the availability of some domain expert to explain how I could replicate a use case and which configuration I needed in order to get my environment working.
To address this issue in our process, we decided to add sample data maintenance as a DoD requirement(Definition of Done).
To support that, we used SFDX JSON Bourne Plugin, which is a plugin to Import/Export salesforce records based on JSON files; it supports lookup/record type and several util configurations, like batch size, multithreading, etc...
The downside was that it only supports JSON, external ids are required for upserts, and you have to install a managed package in your org.
In my current job, we use SFDX SFDMU. It is my favorite! I was introduced to this plugin when we migrated to dx@scale; because it is used by sfpowerscripts to support data package.
It is mighty; it supports different file formats, data transformation, and advanced relationships and covers almost all our scenarios.
Installing dependencies
You don't want to make your scratch org creation scripts full of unreadable package ids, this information usually is already defined in packageAliases section in sfdx-project.json.
sfpowerscripts based on sfdx-project.json, detect the package dependencies and install them in the right order.
Scratch org allocation management among teams
Provisioning the correct number of scratch orgs among development teams and pipelines is not an easy task, and it is very important due to the daily limit allocations — It is possible to have additional scratch orgs on devhub; however, it implies an additional cost.
sfpowerscripts assisted us in mastering the process by providing metrics for a wide range of commands. This valuable information, such as the number of scratch orgs per pool and scratch org limits per devhub, is utilized to populate our Datadog dashboards.
What’s next?
Scratch orgs are a great option for salesforce development; however, they have to be provisioned in the right way.
Scratch org creation is a slow process, which can directly impact the team's performance.
How to speed up? How to allocate scratch orgs among teams?
Next post, I will explain the benefits of scratch org pools.
Don’t forget to subscribe & Share your thoughts!
