Analytics Hub - GCP’s Data Exchange Solution
In this article, we will explore basics of Analytics Hub and also share how to use it to exchange BigQuery datasets within or outside of an organisation — with ease.
We all know that the existing options for data sharing are really expensive. Moreover, it requires some data sharing setup to be in place and still often end up violating security or governance protocols due to lack of proper auditing capabilities. In addition to these challenges, there are limited options for data sharing, collaboration and scaling. This is where Analytics Hub comes to our rescue!
What is Analytics Hub ?
Analytics Hub is a data sharing platform created to ease the task of sharing larger data assets within organizations to address the challenge of data reliability and cost at scale. The data you need is rarely stored in one place, often data is scattered across multiple sources and in various formats.
Analytics Hub not only allows for the curation and sharing of a large selection of analytics-ready datasets globally, it also enables teams to tap into the unique datasets only Google provides, such as Google Search Trends or the Data Commons knowledge graph.
Analytics Hub allows you to share the analytics-ready datasets in three different options (private exchanges are default setting)
- Public — Listing shared publicly to all Google Cloud Users
- Private — Listing shared directly with you or a group you belong to
- Within Organization — Listing shared with your organization
Key Components of Analytics Hub
- Data Exchanges — Data exchanges bring together the data publisher and data consumer. Inside a data exchange there can be many listings which will allow the user to uniquely identify the data.
- Listings — Listings are mainly used to used to uniquely identify the data. Listing is nothing but a reference to the dataset which the user has published inside a data exchange.
- Data Publisher — Data publisher is the one who publishes datasets into Analytics Hub as a listing within an exchange, which can be searched on the listings page of Analytics Hub(if shared publicly).
- Data Subscriber — Data subscriber is the one who wants to use the shared dataset in their own BigQuery project. Data Subscriber can search for the required dataset by using various filters like category, location, provider and listing type.
Next up, let’s do a quick hands-on.
a. Prerequisites of using Analytics Hub
To use Analytics Hub one must enable the Analytics Hub API, but before that they must have following permissions in place
- View APIs in Cloud console — serviceUsage.services.get/list
- Enable APIs— serviceUsage.services.enable
After adding above mentioned permissions one can enable the Analytics Hub API by following the below mentioned link(please modify the project name).
https://console.cloud.google.com/marketplace/product/google/analyticshub.googleapis.com?q=search&referrer=search&project=<project-name-here>
Click on enable to enable the Analytics Hub API.
Hooray!! You are all set to use Analytics Hub.
b. Create Data Exchange and publish a listing
Step-1 : You can find Analytics Hub option on the left hand side of BigQuery page
Step-2.a : Click on create exchange option
Step-2.b: Fill in the exchange related details
Step-3(Optional) : Set up permissions for the exchange. More details related to exchange permissions are available below
- Administrators: these users are assigned the Analytics Hub Admin role.
- Publishers: these users are assigned the Analytics Hub Publisher role. For information about the tasks that publishers can perform, see Manage listings.
- Subscribers: these users are assigned the Analytics Hub Subscriber role. For information about the tasks that subscribers can perform, see View and subscribe to listings.
- Viewers: these users are assigned the Analytics Hub Viewer role. Viewers can view listings.
Once the exchange is created you would be able to see the exchange on the page of Analytics Hub
Step-4: Click on the exchange listed on the Analytics Hub page and create listing
Step-5.a : Fill in the details related to publishing a listing
Step-5.b : Create documentation about the listing using markdown language.For a quick reference to the Markdown syntax Click Here.
Step-6 : Hit save and you are done. Your listing is created.
c. Use Existing Listing in your project
Step-1: Go to the Analytics Hub page and click on search listing.
Step-2: Search for a listing which you want to use in your project using search options and filters.
Step-3: Select the searched listing.
Step-4: Click on add dataset to your project to add the dataset.
Step-5: Select the project and provide a name to this linked dataset. Click save.
You’re done. You would be able to see the linked dataset under your project which you mentioned in the previous step.
That’s it! Thanks for going through the blog, see you next time!
Additional References :
