Docker 1.12/Kubernetes simplified health checks and container ordering with Goss
In this article we’ll go over some common health check and dependency challenges when using docker standalone or with Kubernetes and how they can easily be remedied with Goss and Goss docker-images.
Edit: I recently released a new tool Miniswarm, in this section of the tutorial video I show Goss health checks being used with Docker Swarm.
So.. What is Goss?
Note: Unless you’re familiar with Goss v0.2.0, I recommend you read this section.
Goss is a YAML based serverspec alternative tool for validating a server’s configuration. It eases the process of writing tests by allowing the user to generate tests from the current system state. Once the test suite is written they can be executed, waited-on, or served as a health endpoint.
More info can be found on the Goss webpage.
Example goss.yaml test file:
The above goss.yaml file can also be generated by using the following commands on a configured system:
goss add dns mysql# "a" is short for "add"
goss a addr mysql:3306
goss a command '/goss/checks/db_login.sh'
Once we have a test suite, we can:
- run it once
goss validate- keep running it until the system enters a valid state or we timeout
goss validate --retry-timeout 30s --sleep 1s- serve the tests as a health endpoint
goss serve &
curl localhost:8080/healthz# JSON endpoint
goss serve --format json &
curl localhost:8080/healthz
Docker Challenge: Running more than one HEALTHCHECK command
Docker 1.12 introduced the concept of HEALTHCECK, but only allows you to run one command. By letting Goss manage your checks. The goss.yaml file can define as many checks or reference as many scripts/commands as you like.
FROM alpine:3.4# Install Goss
RUN apk add --no-cache --virtual=goss-dependencies curl ca-certificates && \
curl -fsSL https://goss.rocks/install | sh && \
apk del goss-dependencies# Add our checks
ADD db_login.sh /goss/checks/db_login.sh
ADD goss.yaml /goss/goss.yaml# Use goss to run all our other checkes
HEALTHCHECK --interval=1s --timeout=6s CMD goss -g /goss/goss.yaml validate
You can view the container health using the following commands:
# View health check status
docker ps container_name# Print out the text of the last 5 checks
docker inspect -f '{{ range .State.Health.Log }}{{ println "======\nStart:" .Start }}{{ .Output }}{{end}}' container_name
Docker Challenge: Delaying container startup until dependent services are up and running
If your application can’t gracefully handle missing dependent services, wrap the CMD in a script that calls Goss validate with a retry.
# Alternatively, the -r option can be set
# using the GOSS_RETRY_TIMEOUT env variable
CMD goss -g /goss/goss.yaml validate -r 5m && exec my_command..Kubernetes Challenge: Monitoring 3rdParty applications that do not provide a /healthz endpoint
Create a multi-container pod and use Goss as a /healthz endpoint. Set the /healthz endpoint as a readiness and/or liveliness probe to control traffic and/or container status.
To create a healthz container:
- Create goss.yaml and optionally a checks folder in the following structure:
.
|-- Dockerfile
`-- goss/
|-- checks/
| `-- db_login.sh
`-- goss.yaml- Use the goss:onbuild image to build our healtz image:
FROM goss:onbuild
CMD ["serve"]Example of a mysql deployment with /healthz endpoint:
Kubernetes Challenge: Ensure container waits for dependencies to before starting up
Use the goss:onbuild image to create an image that waits for certain conditions to be met (perhaps another service’s /heathz endpoint). Set it as an init-container. For more information on init-containers, see the Kubernets docs here and here.
Example Dockerfile:
FROM goss:onbuild
CMD ["validate", "--retry-timeout", "5m"]Parting thoughts
Check out Kelsey Hightower’s great talk at Monitorama to see some of the above patterns in action.
If you found this article informative, please recommend it to others by clicking the heart.
