How to use Honeycomb Build Events

Published in

Cazoo Technology Blog

4 min readMar 31, 2021

Intro

As we race through 2021, tech companies get more and more complex. We get better at distributing and separating our code. Gone are the days of the monolith; microservices are here to stay. It will typically take hundreds of microservices or more to run a medium to large tech company. Being able to understand how effective you are at deploying those microservices becomes more powerful. Honeycomb Build Events will give you insights you’ve never had before on your deployment pipelines.

Environment Variables

The orb requires some environment variables to be set in your CircleCI context or Environment Variables. At minimum:

BUILDEVENT_APIKEY - your Honeycomb API key, available on the accounts page
BUILDEVENT_CIRCLE_API_TOKEN - a Personal CircleCI API Token, available from the api tokens section of your CircleCI account page

How to use it

Basic use of the Orb

A simple trace with just two spans, one for the watch and one for start trace — What you’ll get in honeycomb

Add the orb

orbs:
  buildevents: honeycombio/buildevents@0.2.7
	...

2. Add the job definitions that will trace the build

executors:
  linuxgo:
    parameters:
    working_directory: *working_directory
    docker:
      - image: circleci/golang:1.14# Start HoneyComb buildevents trace
start_honeycomb_trace: &start_honeycomb_trace
    executor: linuxgo
    steps:
      - buildevents/start_trace# HoneyComb buildevents trace watches for job to finish then ends trace
honeycomb_trace_watch: &honeycomb_trace_watch
    executor: linuxgo
    steps:
      - buildevents/watch_build_and_finish

3. Add the following new jobs (or inline them if you prefer)

jobs:
  start_trace: 
    <<: *start_honeycomb_trace  watch_trace: 
    <<: *honeycomb_trace_watch

4. Add the jobs to the start of your workflow

workflows:
  version: 2
  pull_request_plan:
    jobs:
      - start_trace: 
          context: cazoo-tools
      - watch_trace: 
          context: cazoo-tools
		...

This will give you very basic tracing. “How long did each of my workflows take?”

Breakdown by job

A much more detailed breakdown of all the steps in the workflow — What you’ll get in honeycomb

You need to nest your job inside the following:

steps:
  - buildevents/with_job_span:

with_job_span should be the only command in your job's steps section. It starts a span that will record the length of this job. It has its own steps command, in which all the things you want this job to do reside.

More info here

Old version:

terragrunt_init: &terragrunt_init # This is what it used to look like
  run:
    name: terragrunt init
    command: |
      source $BASH_ENV
      cd environments/$ENVIRONMENT
      terragrunt init

New version:

terragrunt_init: &terragrunt_init # This is the nested version
  steps:
    - buildevents/with_job_span: 
        steps: # buildevents/with_job_span expects to see a child, `steps`
          - run: # so we've had to add that too 
              name: terragrunt init
              command: |
                source $BASH_ENV
                cd environments/$ENVIRONMENT
                terragrunt init

This will add a breakdown by individual jobs within a workflow

Breakdown by individual command

Even more detailed breakdown with tracing for the individual commands within steps — What you’ll get in honeycomb

The orb gives us the buildevents binary on our path. This allows us to access the underlying go library which includes a function: buildevents cmd

This will execute any given command and trace it. More info here

E.g.

Old Version:

terragrunt init

New Version:

buildevents cmd $CIRCLE_WORKFLOW_ID $BUILDEVENTS_SPAN_ID terragrunt-init -- terragrunt init

$CIRCLE_WORKFLOW_ID and $BUILDEVENTS_SPAN_ID are automatically populated for you. No need to worry about them.

The third argument to cmd is the name of the trace. In this case, terragrunt-init. This is what will be displayed in honeycomb. This doesn't need to match the command, but it's probably smart to make it reflect the code you're executing.

The fourth argument is separated by -- and it tells cmd to assume the rest of the line is a self-contained executable command.

buildevents cmd will take the rest of the line as the command to execute. It may interfere with the execution if you have a multi-line command

Weirdnesses to be aware of

The watch times out after 20 minutes. This is overridable. If you have spans that are delivered later, they will still be included but the overall metric for duration will be inaccurate

We expect the trace details to start after maybe 20 seconds because circleci has to pull down docker before it can even begin tracing. So you may have a bunch of dead space at the start of each of your jobs.

For internal circleci jobs, we aren’t able to trace them e.g. persist_to_workspace

We have no way of explicitly tracing approve steps

You can only have 1 buildevents/with_job_span per job. You cannot nest them or put multiple at the same level.