Magda is an open-sourced data catalog system that provides a single place where all of your organisation’s data can be catalogued, enriched, searched, tracked and prioritized.
We’ve just published Magda v0.0.57, which is the first production release in the last 8 months. However, we also packed a lot of good features which I believe is worth the wait :-)
Kubernetes API 1.16 Support
Since version 1.16, Kubernetes API stopped serving many deprecated API versions in favour of newer and more stable API versions. Consequently, earlier version Magda won’t support Kubernetes API 1.16 or above as we still used some unsupported APIs in our Helm charts.
In v0.0.57, we’ve rewritten most of our Helm charts and migrated to new Kubernetes APIs. Therefore, you can now be sure Magda will run perfectly on Kubernetes clusters with API version 1.16 or higher.
For users who are on older Kubernetes API versions, as long as your cluster is on Kubernetes API version newer than 1.14, Magda v0.0.57 is compatible with your cluster as well.
For new users, you can start to use v0.0.57 straightaway and refer to this document for installation guide.
For existing users, you can refer to our migration document to migrate your existing cluster to v0.0.57.
With v0.0.57, our Helm charts have been upgraded to Helm 3 compatible in order to utilise latest Helm features and provide more stable & streamlined deployment experience.
Furthermore, we now encourage a “chart-based” deployment approach (versus the previous value file approach) to enable a more modular way of deploying & configuring Magda. You can find examples from here or here.
To help you to migrate your existing Helm 2 deployment, we prepared this migration document to migrate your Helm 2 deployment to Helm 3 with Helm release revision history being kept.
Minions & Connectors
As part of efforts of making Magda more modular and extensible, all built-in Minions & Connectors are moved out of our main repository and packed as separate Helm charts. Therefore, you can customise your deployment to only include the minions/connectors that you need. It also allows you to create/deploy your own customised minions/connectors much easier. You can find our official Minions & Connectors repositories as examples from here and here.
Minions & Connectors SDK
To help customized Minions / Connectors development, Magda has published a list of NPM packages to make developing new connectors/minions easier. They are:
- @magda/registry-aspects: all Magda built-in aspect definitions
- @magda/utils: Magda utility functions
- @magda/scripts: Utilities scripts. e.g. docker image building script
We also created a short document on how to build your own connectors or minions.
Registry API: More Powerful Aspect Query Supports
Our Registry records query API supports an
aspectQuery parameter that allows you to filter result by certain aspect field and its value. We previously only support an
:. In v0.0.57, we extend its functionality and add more operators, such as
:! “not equal”,
:? “pattern match”,
:> “greater than” and more.
Besides new operators support, we also added
aspectOrQuery parameter to allow you to specify an
OR logic. And
sortBy parameter to sorting result by aspect field values.
For the full list of new operators supported and new query capability, please refer to our API document here.
User Supplied OPA Policies
Magda uses Open Policy Agent as our central access control policy engine and build our authorisation system on top of it. If you are interested in customising the access control logic, since v0.0.57, we offer an option to supply your own policy files during the deployment. For more details, please check this document.
Internal File Storage
Since v0.0.57, we added the support of storing data files internally by Magda. Powered by open-sourced Kubernetes Native Object Storage MinIO, we now allow the user to store data files internally with Magda via our new storage APIs. Our metadata creation tool UI also provides options to users to allow them to store data file in Magda’s internal storage.
API Key Support
Many of our APIs require admin access. Previously it’s not easy to access those APIs programmingly or with automated tool. Since v0.0.57, we added API key support so that we can generate API keys that allow you to access the API on behalf of your account. To create a API key for your account, please check this document.
Authenticate User Locally
Magda provides wide range of SSO support. But there are some occasions when you will want to authenticate users locally. In v0.0.57, we added local authentication support. To create local users, please check this document.
Function as a service (FaaS) Support
Once you have v0.0.57 Magda deployed, you will find two more namespaces are auto-created as well. These two namespaces are suffixed with
-openfaas-fn, which are where Function as a service (FaaS) workload is managed and run. Powered by open-sourced project OpenFaas, we allow you to extend Magda’s functionality using more abstract function model in v0.0.57 Magda. As there doesn’t have to be a Kubernetes pod running unless the function is called, the FaaS model is a perfect fit for adding data processing capability to Magda.
In future, we will look at gradually replacing our existing Connector/Minion model with an event/message-driven function model (and opportunities to simplify our event Registry Service). For the long term, we may leverage it to build a platform where user-contribute data processing capbilitities (“functions”) can be created, discovered and consumed.
Metadata Creation Tool
In v0.0.57, we offer a more complete Metadata creation tool. To try it out, please add your self an
admin role and set your user record
isAdmin field to “true”. After login with your admin account, you will see the new interface from the home page.
Try it out
To try it out v0.0.57, you can access our Helm chart repo: https://charts.magda.io and access
magda chart with version
You can also follow the instruction here to quickly setup a new site.
For developers, you can access npm for our latest SDK version 0.0.57.