GameForce, Part 5. Setting up CI/CD for Salesforce using GitHub Actions

There is this thing that I like to call a CI/CD paradox — there is never a good time to set it up. In the early stages of the project, it might sound like an overkill to spend time on CI/CD configuration, while you still don’t have any real code to test. But then, suddenly, you have too many things at once — PRs that you need to check, new builds that you have to release, features that you want to write, a backlog that you need to maintain, and many many more. So obviously, the best time to implement the CI/CD for your project is right before the moment when your project is starting to get big but is still not too big to become a mess. Defining this moment is somewhat tricky (especially when you are working alone) and it depends heavily on a ratio of your threshold for doing monotonous work to the level of your aspiration to have a maintainable code.

To explain this concept, let’s actually start from the “aspiration to have a maintainable code” part. If there is something that I’ve learned from my years of working as a software developer — things tend to break down when least expected. A system of checks and balances is needed to not get stuck in a constant cycle of fixing old things while trying to work on new features. Unit testing is one part of such a system, and it is important not only to run new tests, but to also re-run old tests to see that nothing was broken in the meanwhile. But this is where the “threshold for doing monotonous work” comes into play — the thing is that adding new features inadvertently would require adding more unit tests. As a result, with each new feature, re-running all the tests would gradually become more tiresome and time-consuming, meaning that at some point you will stop doing that, allowing bugs and issues to creep into your code.

For me, this moment has come somewhere at the end of the first milestone, when I was almost done working on the backend and was writing a third part of the blog series, hoping that I will find someone willing to contribute to the project. Currently, my repository has 13 different unit test classes and somewhere around 54 unit tests, which doesn’t sound like a lot, but as I’ve said earlier — I do not only run tests for the feature that I am working on at the moment but all other unit tests as well. Additionally, any new potential contribution would require me to checkout the branch with changes, create a scratch org, deploy everything, and finally run the tests to make sure that nothing is broken even before I would start code reviewing it. All of that adds up and becomes a tedious and repetitive procedure. And when there is something tedious and repetitive it is time for the automation.

To be honest, I wasn’t going to write an article about CI/CD configuration. My plan was to write something about why it is important and what benefits it brings. But, while I was investigating the subject, I realized that a lot of articles about this topic aren’t detailed enough. Authors tend to cut corners, and as a result, available tutorials are incomplete at best or even contain obvious security vulnerabilities. So, strap in, since it will be one more article about how to configure CI/CD for Salesforce using GitHub Actions.

Defining CI/CD triggers for the project

Up until this point, there was no automation in my repository and obviously no releases, and as a result — no need for a complex branching strategy. The development flow was pretty straightforward and looked something like this:

But, as I’ve said before, to make sure that I’m not breaking anything, I had to run a full set of tests each time I was going to merge anything to the main branch. So the actual development flow, from my perspective, looked more like this:

“Manual checks” on the diagram are the first obvious candidates to be replaced with automation. To do so, we don’t have to change anything in our current branching strategy. We just have to replace “manual checks”, with “auto checks”(unit tests, jest tests etc.) executed by GitHub Actions. And as we can see, “manual checks” are always happening after PR was created or something new was added to it (some fix was pushed to the feature branch while PR was still). So the obvious trigger for our pipeline would be PR creation or modification:

The same logic would also apply, in case someone wants to contribute to the project and forks it since they would still have to create a PR to add their changes to the repository.

This is exactly what the “CI” part of CI/CD is about — making sure that frequently merged changes from multiple developers don’t break our codebase unexpectedly.

Now let’s move on to the “CD” part. Continuous Delivery (or Deployment, if you prefer) is the automated process of deployment of the software to production or staging environments. There are 2 main processes, that fall under the “CD” category, that you might want to automate from the get-go. The first one is the deployment of changes to your dedicated test environment so that QA engineers can test the latest changes before they get released. And the second one — creating a release package once you have enough changes to be called a “new version”. Sure, in an ideal world, you would probably release new changes with each new issue being completed, but in reality, it usually makes sense to combine and release multiple issues together. The problem is that when you have a single main branch that collects all the changes, it might get hard for your automation to differentiate between releasing things, or just deploying new changes for testing. This problem can be easily solved, by introducing a new “dev” branch. All new feature branches will originate from the “dev” branch, and once something gets pushed to the branch — it will trigger the automation that would deploy changes for testing. Once you are ready to release a new version of your code — you would have to create a new PR from “dev” to “main”, which would re-run auto checks again. After this PR gets merged into the main — it would trigger the creation of a new package.

To sum it up, our CI/CD automation should work somewhat like this:

1. Whenever PR to the “dev” or “main” branch gets created or update — run auto checks (unit tests, LWC Jest test, etc.)
2. Whenever PR is merged into the “dev” branch — deploy it to the test environment
3. Whenever PR gets merged to the “main” branch — create a new version of a package and release it.

Now, once we understand “what we want to do”, let’s move on to “how are we going to do it?”.

Configuring CI/CD

I’ve noticed that people unfamiliar with a configuration of CI/CD usually treat it as some sort of a sacred ritual, that only chosen ones can do. And I think there are two reasons why this is the case:

1. People don’t have an understanding of how CI/CD actually works from a technical standpoint
2. And even if they do, they are not comfortable doing things using CLI

Let’s tackle those issues one by one. Imagine that you have a person willing to do monotonous work for you. The only problem is that that person has zero understanding of your project, and responds only to messages that you write him in a chat that doesn’t support anything other than text. You can try to write instructions to him like “open your browser, go to this URL and click the button on the right side of a screen, etc.”, but I bet you that they will get confused quite quickly and will mess things up. Alternatively, you can try to write him a set of CLI commands, that he can copy and paste one by one, and since there will be no room for interpretation on his side of things, this approach should be way more successful. The only problem is that you are limited to things that can be done with CLI commands, which is not a big deal when it comes to validating code quality or packaging the code. This is exactly how CI/CD works. But, instead of a person, we have a server of some sort, that executes CLI commands and checks the results. Specific triggers for actions (like PR creation to branch “X”) and the list of commands are defined in so-called YAML files. If we get back to our analogy, you can imagine that YAML is a set of instructions that you would send to our “hypothetical person” via chat.

Okay, so now, since we have a grasp of a concept, let’s get familiar with YAML and how it works. Here is what a basic YAML file might look like:

name: PR Checks
run-name: A set of checks and validations for each PR to dev or main
on: 
  pull_request:
    branches:
      - dev
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: STEP 1
        run: |
          CLI command 1
          CLI command 2
          CLI command 3

      - name: STEP 2
        run: |
          CLI command 1  
          CLI command 2

And as you can see, it is fairly easy to read. This set of commands will be executed on pull request creation (or modification) to branches “dev” or “main”. Commands will be executed on a server running Ubuntu (sure, I know that technically it is a container, but I will be calling it “server” for the sake of simplicity), so all CLI commands should be written so that they can be executed using a built-in Ubuntu terminal. Notice that commands are grouped into “steps”. This is not exactly necessary, but it splits commands into logical chunks, making it easier to monitor the execution. Here is an example of how “steps” are represented by the GitHub:

Assuming that the YAML format is somewhat clear now, we need to populate it with actual CLI commands that we want to execute. Do you remember when I told you that our “hypothetical person” has zero understanding of our project? Well, this also means that he doesn’t have tools to deploy the source code, and doesn’t even have the source code. So the first set of commands should fix that:

name: PR Checks
run-name: Checks running whenever new PR is created
on: 
  pull_request:
    branches:
      - dev
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      # Checkout the Source code from the latest commit
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      # We need to install NPM so that we can install SFDX CLI tools
      - name: Install NPM
        run: |
          npm install  
      # Isntalling SFDX CLI
      - name: Install the SFDX CLI
        run: |
          npm install @salesforce/cli --global

The next thing that we need to do before we can create a scratch org and run the tests is to login to the DevHub. But the problem is that it usually involves running the “sf org login” command and then opening the browser window and inputting username and password. Which wouldn’t work since we don’t have a browser, we only have the CLI. Fortunately, there is a different way to authenticate with Salesforce using only CLI — “sf org login jwt”, which uses certificate-key pair for authorization. Before we will continue, there is a thing that I have seen multiple times in different tutorials that needs addressing:

CAUTION. If you ever see a CI/CD tutorial that uses username and password or stores key-certificate pair directly in your repository — IT IS UNSAFE, DON’T DO IT LIKE THAT!

Hopefully, this is clear and we can continue.

To use “sf org login jwt” we need to do 3 things first:

1. Generate certificate-key pair using OpenSSL
2. Create a Connected App on our DevHub to store the certificate
3. Store key securely in GitHub secrets

Generating a certificate-key pair is super simple, and you can do this by following this official Salesforce tutorial: Create a Private Key and Self-Signed Digital Certificate

Creating a Connected App is a bit more tricky. You should do that by following the official Salesforce tutorial, but be careful, because it is quite lengthy and it is easy to miss a step and as a result, you won’t be able to authorize. If everything is done correctly — you will have a Connected App, where you have uploaded the certificate (.crt file) that you’ve just generated and that allows authorization of system administrator users (“System administrator” profile has to be added to the Profile section of the Connected App on step 20 of the tutorial). Alternatively, you can create a dedicated “CICD Profile” and user, but I don’t think it makes much of a difference when it comes to a personal pet project.

At this point I would recommend you to try to authorize your Salesforce org using your local development machine, just to make sure that everything works as intended. The authorization command would look like this:

sf org login jwt --json --alias cicd 
--username *** Your username *** 
--keyfile  *** Path to .key file that you have generated *** 
--clientid *** Consumer Key of the Connected App ***

If this works, it means that Connected App is configured properly and your certificate and key match. Congratulations! The hardest part is behind.

Now, we need to store the username, certificate key, and consumer key securely in the GitHub, so that our CI/CD can access them. And, okay, this is important so I will repeat it once again:

CAUTION. If you ever see a CI/CD tutorial that uses username and password or stores key-certificate pair directly in your repository — IT IS UNSAFE, DON’T DO IT LIKE THAT!

So please, never store your secrets directly in your repository. Instead, you need to go to the settings page of the repository. Go to the “Secrets and variables” section and add them as “New repository secret”

Create three new secrets:

• SF_CICD_SERVERKEY — copy the contents of the .key file that you’ve generated
• SF_CICD_USERNAME — username that will be used during authorization
• SF_CICD_CONSUMERKEY — Consumer Key value copied from the Connected App

Once this is done, we can get back to configuring our CI/CD and authorizing with a DevHub:

name: PR Checks
run-name: Checks running whenever new PR is created
on: 
  pull_request:
    branches:
      - dev
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      # Checkout the Source code from the latest commit
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      # We need to install NPM so that we can install SFDX CLI tools
      - name: Install NPM
        run: |
          npm install  
      # Isntalling SFDX CLI
      - name: Install the SFDX CLI
        run: |
          npm install @salesforce/cli --global  
      # We need to use a key value stored as GitHub secret and 
      # create a .key file since 'sf org login jwt' command expects a path 
      # to .key file as a parameter
      - name: Create server key file
        run: |
          touch server.key
          echo -e "${{ secrets.SF_CICD_SERVERKEY }}" >> server.key  
      # Authorization with DevHub. Notice that all critical values are
      # used from the GitHub secrets storage and aren't public
      - name: Authorize DevHub
        run: sf org login jwt --json --alias DevHub 
                --set-default --set-default-dev-hub 
                --username "${{ secrets.SF_CICD_USERNAME }}" 
                --keyfile /home/runner/work/game-force/game-force/server.key 
                --clientid ${{ secrets.SF_CLIENT_SECRET }}

Notice how ${{ secrets.**NAME_OF_THE_SECRET** }} notation is used to access GitHub secrets.

And once we are authorized with a DevHub we can create a scratch org, push our code to it, and run unit tests like this:

name: PR Checks
run-name: Checks running whenever new PR is created
on: 
  pull_request:
    branches:
      - dev
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      # Checkout the Source code from the latest commit
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      # We need to install NPM so that we can install SFDX CLI tools
      - name: Install NPM
        run: |
          npm install  
      # Isntalling SFDX CLI
      - name: Install the SFDX CLI
        run: |
          npm install @salesforce/cli --global  
      # We need to use a key value stored as GitHub secret and 
      # create a .key file since 'sf org login jwt' command expects a path 
      # to .key file as a parameter
      - name: Create server key file
        run: |
          touch server.key
          echo -e "${{ secrets.SF_CICD_SERVERKEY }}" >> server.key  
      # Authorization with DevHub. Notice that all critical values are
      # used from the GitHub secrets storage and aren't public
      - name: Authorize DevHub
        run: sf org login jwt --json --alias DevHub 
                --set-default --set-default-dev-hub 
                --username "${{ secrets.SF_CICD_USERNAME }}" 
                --keyfile /home/runner/work/game-force/game-force/server.key 
                --clientid ${{ secrets.SF_CLIENT_SECRET }}
      # Pushing sourche to Scratch org
      - name: Push source to Scratch Org
        run: sf project deploy start
      # Running tests to make sure nothing is broken 
      - name: Run unit tests
        run: sf apex run test --wait 30 --test-level RunAllTestsInOrg --code-coverage

And that is how you configure CI/CD for Salesforce using GitHub actions.

Before I will end this article, I need to confess — at the beginning of the article, I promised to give you the full CI/CD setup, but this setup is far from complete. There are a couple of reasons for that — this article is already way too big, and I am getting bored by this whole CI/CD thing and want to switch to working on other issues, and the current setup is already way better than running all the checks manually. But it doesn’t mean that I am done with CI/CD for this project. I still need to configure the “CD” part once I will be ready to actually release something. Also, there is a matter of Scratch org limits for free development orgs — I can only have six active Scratch Orgs at once, and can only create three orgs a day, so there is no way I can create a new Scratch Org each time I am updating PR. Also, it would be nice to delete a scratch org once PR gets merged. I’ve even created issues for those improvements(issue 52, issue 53, issue 54), so if someone wants to play around with CI/CD and help me out a bit — you are welcome. I will also describe those further CI/CD improvements in future articles, so stay tuned!

Part 6. A pop-up dilemma

Part 4. Backend

GameForce GitHub