QuickSight Dashboard Promotion

Migrating dashboards across AWS accounts

Introduction

Many data-driven organisations use QuickSight dashboards to provide visualisations and deliver insights to their users. However, migrating a dashboard from a development AWS account to another can be a chore.

A typical workflow involves BI engineers connecting to the production QuickSight instance to create analyses and to publish them as dashboards for use by the rest of the business as appropriate.

Sometimes, when data sources or data products are still in development, it won’t be available in the production account yet. In these cases, BI engineers may want to start developing dashboards in the development (or other non-prod) account and work closely with the data engineers to make changes as required.

When promoting a data product into production, it is good practice to promote any dashboards related to it at the same time (alongside the data pipelines and other related components). However, since BI engineers typically create dashboards by hand using the QuickSight web UI, promoting these to higher environments via automation is not as easy as, say, a data pipeline defined as code. The dashboard definitions are exportable but the underlying data sets reference data sources in different AWS accounts. These data sources would typically be isolated and not accessible cross-account (and not hosting production data anyway!).

We have created a process that solves this problem. It helps BI engineers save time by preventing them from having to recreate the dashboards by hand in each account (which would potentially introduce inconsistencies in the visuals or calculations), and to prevent them from having to configure the data sources in each account (which means they would need access to the credentials). The process leverages the QuickSight APIs to promote dashboards across accounts and it updates all of the underlying references to resources.

Anatomy of a QuickSight dashboard

So, what does a QuickSight dashboard look like under the hood? The QuickSight API defines a dashboard in terms of the:

• Name
• ID
• Definition (the placements of visuals and fields)
• Theme
• Data Sets

Digging deeper into the output, the data sets contain references to the underlying data sources in the DataSetIdentifierDeclarations key. We need to update this in order to point to the equivalent data source in a different account. This key contains a list of all the data sets and creates a mapping between an Identifier (used to reference the data set throughout the dashboard definition) and the DataSetArn.

When migrating a dashboard to another account, we must update the value of the DataSetArn and, because the dashboard definition references the associated Identifier, everything should fall into place. In reality, there are a few extra steps before we can get to this point, so let’s have a look at what’s involved…

Migrating the dashboard

Before migrating a dashboard, we must deploy the other components of the data product into the target account. Essentially, we need the data source and its data to exist in the target account.

We can now begin migration by following these high level steps using the QuickSight APIs:

1. Get the dashboard definition.

Make the describe-dashboard-definition call for the chosen dashboard in the source account.

2. Get the data set properties.

For each data set in the dashboard definition, the describe-data-set call can be made to reveal the underlying data source.

3. Recreate the data sources.

For each underlying data source, make the describe-data-source call in the source account to get the data source properties. Depending on the data source type (for example Redshift or RDS), there will be different DataSourceParameters to update which are specific to each source type such as hostname and port.

The Credentials will also need to be set (if applicable) and the VpcConnectionProperties (if applicable). The remaining data source attributes such as the Name, DataSourceId and Type can remain the same.
These updated values should be used as parameters for the create-data-source API call in the target account.

4. Recreate the data sets.

Now that we’ve got the data sources created, we need to create our data sets. First we need to get our data set attributes using the describe-data-set call we made in step 2 in the source account.

Then, we need to update the DataSourceArn value to point to the new data source ARN we created in the previous step. This value is hidden in the data set attributes under: PhysicalTableMap.<PhysicalTableMapId>.RelationalTable.DataSourceArn.

The remaining attributes DataSetId, Name, LogicalTableMap, ImportMode, FieldFolders and DataSetUsageConfiguration can remain the same as the original values. Only the PhysicalTableMap needs to be changed.

These updated values can now be used as parameters for the create-data-set API call in the target account.

5. Recreate the theme.

The hard work is out of the way now, we just need to get the theme definition using the describe-theme call in the source account.

Then simply pass the ThemeId, Name, BaseThemeId and Configuration values to the create-theme call in the target account.

6. Recreate the dashboard.

The final step is to recreate the actual dashboard we’re migrating. Take the dashboard definition from the first step and update the DataSetIdentifierDeclarations values. The Identifier will remain the same and the DataSetArn must be changed to the ARN of the data sets in the target account created in step 4 above.

Now use the create-dashboard call and pass the DashboardId, Name, ThemeArn and Definition to create the dashboard in the target account.

We now have a dashboard and all of its dependent resources copied from one AWS account to another.

Miscellaneous notes

This process was created to migrate dashboards from a development AWS account to the production AWS account as the production account wasn’t deployed yet during the development of the data product. Keep in mind that typically you would want your BI engineers to simply work on their analyses on QuickSight in the production account and publish as dashboards when ready.

The script to facilitate this work was written in Python using the boto3 package. Most of the steps are automated but some user inputs are required such as specifying the data source parameters like hostname, credentials and VPC connection ARN for the target account. This was suitable for our requirements but could be improved by performing a lookup in Parameter Store which maps the sources across environments to allow for parameter and credential lookup without user input. This could be useful in cases where this process is being run more regularly and you don’t want the engineers migrating dashboards to have access to credentials.

You cannot migrate dashboards which have errors in their definition. The Errors key will be present in the output of the first step above only if errors are present. These must be resolved before the migration can take place otherwise the create-dashboard call in the target account will fail.