Troubleshooting Spinnaker/Rosco Bakes

Rosco is Spinnaker’s open source image bakery service. Rosco uses Packer to bake images and each bake request results in a local job being spun up to run a Packer command.

When a bake fails and you’re not sure why, a good first step is to try to rerun the Packer command directly without going through Spinnaker. All bake information is persisted via Redis so we need to first determine what Redis key to query.

To determine the exact Packer command resulting from a Bake Stage:

Click on the View Bakery Details link in the execution view of your Bake Stage:

Note the uuid included in the bakery details url:

Use this uuid to query Redis for the details of the bake (first showing all hash keys here, and then getting the value associated with command):

$ redis-cli hkeys 27f57ea9–09ca-4965-ba19–69d4ced63ea1
1) "bakeStatus"
2) "bakeLogs"
3) "region"
4) "createdTimestamp"
5) "bakeRequest"
6) "bakeKey"
7) "command"
8) "updatedTimestamp"
9) "roscoInstanceId"
$ redis-cli hget 27f57ea9–09ca-4965-ba19–69d4ced63ea1 command
"packer build -color=false -var gce_project_id=rosco-oss-2 -var gce_zone=us-central1-f -var gce_network=default -var gce_source_image=ubuntu-1204-precise-v20161020 -var gce_target_image=mypackage-all-20170110180918-precise -var gce_account_file=/home/duftler/.spinnaker/rosco-oss-2.json -var repository=\"http://dl.bintray.com/spinnaker/ospackages ./\" -var package_type=deb -var packages=mypackage -var configDir=/home/duftler/build/rosco/rosco-web/config/packer /home/duftler/build/rosco/rosco-web/config/packer/gce.json"

Before running the packer command we just need to replace \” with . We can retrieve the command, remove the escaping of the double-quotation marks, and execute the command, all at once as follows:

eval `redis-cli hget 27f57ea9–09ca-4965-ba19–69d4ced63ea1 command | sed 's/\\"/"/g'`

If the packer command doesn’t work properly on its own, it will not work properly when called from Spinnaker. Isolating the packer command like this removes many moving parts and makes the troubleshooting cycle much more efficient. Make sure to run the Packer command on the same machine on which Rosco runs, and as the same user that runs Rosco itself. This is important so that you have the correct network access and other permissions available.

You will likely find the other hash keys useful for troubleshooting as well. For example: bakeRequest to retrieve the original request that initiated the bake and bakeLogs to retrieve the packer logs.

Common Issues

  • Path to Packer templates is wrong
  • User permissions for Packer templates or its dependencies are wrong
  • Cloud provider credentials can’t be resolved or don’t grant permissions required to produce an image
  • deb/rpm repository is unspecified or incorrect
  • Very long Packer jobs timing out (this can be configured via the rosco.jobs.local.timeoutMinutes setting)

Next up…

Explanation of Rosco’s deduping logic