Maintaining ecological validity in a multi-tiered infrastructure using CircleCI contexts

In application development, you will want to deploy to multiple environments (development, testing, staging and production, for example) as the project progresses through a software development lifecycle. When designing software as a service, a condition of a twelve-factor app is that each environment should be as close to parity as possible.

If an integration deployment is successful and passes fitness tests (unit, integration and performance) then deploying to production can be made with confidence — making the continuous integration process repeatable and predictable.

Keeping parity also ensures that deployment issues can be found and mitigated early on in your development lifecycle — this is commonly known as ecological validity.

The extent to which the conditions simulated in the laboratory reflect real life conditions.” — https://www.interaction-design.org/literature/book/the-glossary-of-human-computer-interaction/ecological-validity

By the end of this article, you should be able to:

• Understand the purpose of CircleCI contexts and how it helps with configuring a twelve-factor app.
• Use contexts to consistently deploy to a multi-tier infrastructure, ensuring parity between environments.
• Keep configurations amongst tiers as consistent as possible.

How not to do it

The way most continuous integration systems work is to use environment variables with prefixes (e.g.: DEV_ or PROD_). Then using a separate set of references, referencing the variable that you want to use. CircleCI doesn’t allow this because it does not support environment variable interpolation:

# Interpolation does not work
...
 
workspace
  production:
    environment:
      AWS_ACCESS_KEY_ID: $PROD_AWS_ACCESS_KEY_ID
      AWS_SECRET_ACCESS_KEY: $PROD_AWS_SECRET_ACCESS_KEY
 
...

While it is possible to circumvent this issue using a shell command (e.g. in Bash) using prefixed namespace environment variables is a practice which should be avoided:

export ENV=DEV
export DEV_KEY=abc123
export KEY="$(echo "${ENV}_KEY")"
echo ${!KEY}

CircleCI contexts

What are they?

Contexts are a way to execute a single group of steps with different environment variables (which can give different results). They are available organisation-wide, so the same context can be used on multiple repositories/projects.

You may want to briefly skim through the CircleCI documentation on contexts. It also contains important information on access permissions (which is outside the scope of this article).

Implementation

For this article, we are representing ACME (A Company that Makes Everything). It doesn’t matter which language we use (Node.js, PHP, etc.) or the target we are deploying to (AWS, Azure, Heroku, etc.) — the concepts remain unchanged. This article targets AWS, which requires the following two environment variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Setting up contexts

You can manage your contexts by:

1. Signing in to CircleCI and managing your Organization Settings.
2. Click Contexts on the sidebar menu.

Since there are two environments targeted for publishing in this scenario, there will be two contexts:

• acme-context-dev for development
• acme-context-prod for production.

Two contexts, one for development and one for production.

You can then go into your context by clicking its name and adding any required environment variables. For each context, this example will use the following:

• APP_ENV — a friendly environment name (development, production, etc.),
• AWS_ACCESS_KEY_ID — the environment’s AWS access key ID,
• AWS_SECRET_ACCESS_KEY — the environment’s AWS secret access key,
• DEPLOY_BUCKET — the AWS S3 bucket which will receive the artefacts.

An example implementation of environment variables in a context.

Optionally document the contexts in your configuration

Begin your .circleci/config.yml file by documenting the contexts and environment variables that are in use throughout your application. While this is optional, it simplifies future maintenance.

# Required contexts:
# - acme-context-dev
# - acme-context-prod
#
# Environment variables (per context)
# - APP_ENV (e.g.: development, production)
# - AWS_ACCESS_KEY_ID
# - AWS_SECRET_ACCESS_KEY
#
# Environment variables (per project)
# - AWS_REGION
 
version: 2.1
 
jobs:
  build:
    steps:
      - checkout
      - run: echo 'build steps here'
  publish:
    steps:
      - run: echo 'deploy steps here'

This defines two (very) simple jobs: build and publish in keeping with the twelve-factor app build-release-run requirements. Your integration will include business-specific steps (e.g.: caching dependencies, build and tasks, etc.).

Set up your workflow

Define a workflow in your .circleci/config.yml. This will control the order in which the jobs are executed and inject the context to each workflow.

...
 
workflows:
  version: 2
  build:
    jobs:
      - build:
          name: build
 
      - approve-dev:
          type: approval
          requires:
            - build
  
      - approve-prod:
          type: approval
          requires:
            - build
  
     - publish:
          name: publish-dev
          context: acme-context-dev
          requires:
            - approve-dev
  
      - publish:
          name: publish-prod
          context: acme-context-prod
          requires:
            - approve-prod

This example describes five jobs:

• The build job — runs on every commit push.
• The approve-dev and approve-prod jobs, which wait for user intervention before continuing.
• The publish-dev job (inheriting publish jobs from earlier) which only publishes when approved from the CircleCI interface.
• The publish-prod job (also inheriting publish jobs from earlier) which only publishes when approved from the CircleCI interface.

Contexts are only attached on publish jobs — another principle of 12-factor apps as the build process shouldn’t be affected by environment variables.

This structure allows the publishing process to be written for multiple tiers in a unified way using different endpoints, credentials and variables to propagate the artefact —helping to remove the likelihood of human error.

Executing

Now that the contexts have been configured for each environment and the workflows have been set up, commit and push your git repository and (provided all your hooks are set up properly) your build should run.

Conclusion

You should now have a consistent way of propagating your code between each of your environments. You can make as many contexts as you like but remember: contexts are organization-wide, so one context can optionally be used on multiple projects.