Building a Dashboard for a data processing pipeline with the Stackdriver Dashboard API
This is part 2 of a 2 part series.
- Part 1: covers identifying recommended Stackdriver Monitoring metrics to use for a data processing pipeline
- Part 2: covers how to using the Stackdriver Dashboards API to implement the charts and dashboards described in part 1 from a JSON template.
First, why automate?
One of the common requests that I hear from the DevOps and SRE teams using Stackdriver Monitoring is that they want to automate dashboard deployment like they do with most infrastructure or environment set-up. It’s the popular the config-as-code approach to reduce human toil! Another common request that I hear, this time from dashboard users, is that users want to be able to share the configuration for a dashboard that they have developed so that a colleague can use the dashboard as a template. The recently released Stackdriver Dashboards API provides the solution to both of these use cases.
Stackdriver Dashboards API
The Stackdriver Monitoring API provides a resource called projects.dashboards which provides a familiar set of methods: create, delete, get, list and patch. These methods follow the REST semantics, accept JSON payloads and are consistent with other Google Cloud REST-based APIs.
The rest of this post requires an understanding of the details of how dashboards, charts and metrics are used in the Stackdriver Monitoring console. You can read the Creating charts section in the docs to find all the details.
Building the dashboard JSON payload the easy way
I mentioned that the Dashboards API provides both a create and a get method. Because building a dashboard in JSON from scratch requires detailed knowledge of the API data model and corresponding JSON syntax, a much easier approach is to build the dashboard in the Dashboards section of Stackdriver Monitoring and then use the API to expore the JSON representation of the dashboard. Once you have the JSON, you can then use the create method to create another dashboard based on the JSON. I strongly recommend this approach as an easy way to get started.
The dashboard JSON payload
In order for you to create a dashboard via the Dashboards API, you need the define several bits of JSON payload data for which you can use to call the projects.dashboards.create method. Here’s an example JSON payload:
- displayName — the human-readable name of the dashboard
- gridLayout — the container for the widgets
- widgets — the container for the chart items
- dataSets — this includes the details used to gather the specific data in a timeSeriesFilter object including the metric name, metric filters and how the metric is aggregated.
- xAxis, yAxis — definitions affecting the presentation of the axes
- chartOptions — definitions affecting the mode of the chart
Using projects.dashboards.create
There are many ways to call the projects.dashboards.create method. In this example, you can easily use the “Try this API” functionality directly in the API documentation to test out the API calls*.
*Please note that you should already have a Stackdriver Monitoring Workspace defined and the GCP project id for the project that contains the Workspace.
I have already created a JSON dashboard template including all 6 of the charts described in part 1 of this series. You can use this JSON payload as a template for your own dashboard.
- To create the dashboard, click the blue “TRY IT!” button to open the “Try this API” feature on the right-hand side of the projects.dashboards.create method.
- Enter a value for the parent input form in the pattern “projects/YOUR_PROJECT_ID” replacing your own GCP project id that contains the Stackdriver Workspace where you want to create the dashboard for the “YOUR_PROJECT_ID” string value.
- Highlight the default values in the Request Body input form and then copy/paste the JSON dashboard template
4. Click the “EXECUTE” button at the bottom of the page. If all goes well, you should see a green HTTP “200” response code along with the JSON description of the dashboard that you just created.
5. Open the Dashboards section in the Stackdriver Monitoring console to review your newly created dashboard. Find the “Data Processing Dashboard Template” and click the name to open the dashboard.
If you have already deployed Pub/Sub, Dataflow and BigQuery resources, you should already see values in the dashboard.
Practical use cases for the dashboards template
I started this series with 2 common use cases:
- DevOps/SRE team members creating or updating dashboards
- users sharing dashboard templates
So, how can you use the Dashboards API for the use cases?
In the first case, dashboard JSON templates can be constructed, by exporting an existing dashboard from the Stackdriver Monitoring console, and then the JSON may be checked into a source repository like github or Cloud Source Repos. From there, you can deploy the dashboard by calling the Dashboards API as a part of your standard deployment process by pulling the JSON template file from the repo.
In the second case, you can create a dashboard from an existing dashboard from the Stackdriver Monitoring console and then use the Dashoards API to export the JSON template. You can then share that template either via repo or however you normally share files with your colleagues.
In both cases, having a pre-built template dashboard to start should make it easier to build and share Stackdriver Monitoring dashboards.
Conclusion
In this series, I have detailed a reasonable set of metrics and charts for a data processing pipeline and then described a detailed approach for deploying the dashboard template using the Dashboards API.
