Magda v0.0.57: Kubernetes 1.16 Support and More

Image for post
Image for post
Photo by Joe Leahy on Unsplash

Magda is an open-sourced data catalog system that provides a single place where all of your organisation’s data can be catalogued, enriched, searched, tracked and prioritized.

We’ve just published Magda v0.0.57, which is the first production release in the last 8 months. However, we also packed a lot of good features which I believe is worth the wait :-)

Kubernetes API 1.16 Support

Since version 1.16, Kubernetes API stopped serving many deprecated API versions in favour of newer and more stable API versions. Consequently, earlier version Magda won’t support Kubernetes API 1.16 or above as we still used some unsupported APIs in our Helm charts.

In v0.0.57, we’ve rewritten most of our Helm charts and migrated to new Kubernetes APIs. Therefore, you can now be sure Magda will run perfectly on Kubernetes clusters with API version 1.16 or higher.

For users who are on older Kubernetes API versions, as long as your cluster is on Kubernetes API version newer than 1.14, Magda v0.0.57 is compatible with your cluster as well.

For new users, you can start to use v0.0.57 straightaway and refer to this document for installation guide.

For existing users, you can refer to our migration document to migrate your existing cluster to v0.0.57.

Helm 3

We use Helm to pack & manage our Kubernetes application package. Helm 3 is the latest release that comes with many long-awaited features such as 3-way Strategic Merge Patches and Removal of Tiller.

With v0.0.57, our Helm charts have been upgraded to Helm 3 compatible in order to utilise latest Helm features and provide more stable & streamlined deployment experience.

Furthermore, we now encourage a “chart-based” deployment approach (versus the previous value file approach) to enable a more modular way of deploying & configuring Magda. You can find examples from here or here.

To help you to migrate your existing Helm 2 deployment, we prepared this migration document to migrate your Helm 2 deployment to Helm 3 with Helm release revision history being kept.

Minions & Connectors

As part of efforts of making Magda more modular and extensible, all built-in Minions & Connectors are moved out of our main repository and packed as separate Helm charts. Therefore, you can customise your deployment to only include the minions/connectors that you need. It also allows you to create/deploy your own customised minions/connectors much easier. You can find our official Minions & Connectors repositories as examples from here and here.

Minions & Connectors SDK

To help customized Minions / Connectors development, Magda has published a list of NPM packages to make developing new connectors/minions easier. They are:

We also created a short document on how to build your own connectors or minions.

Registry API: More Powerful Aspect Query Supports

Our Registry records query API supports an aspectQuery parameter that allows you to filter result by certain aspect field and its value. We previously only support an equal operator :. In v0.0.57, we extend its functionality and add more operators, such as :! “not equal”, :? “pattern match”, :> “greater than” and more.

Besides new operators support, we also added aspectOrQuery parameter to allow you to specify an OR logic. And sortBy parameter to sorting result by aspect field values.

For the full list of new operators supported and new query capability, please refer to our API document here.

User Supplied OPA Policies

Magda uses Open Policy Agent as our central access control policy engine and build our authorisation system on top of it. If you are interested in customising the access control logic, since v0.0.57, we offer an option to supply your own policy files during the deployment. For more details, please check this document.

Internal File Storage

Since v0.0.57, we added the support of storing data files internally by Magda. Powered by open-sourced Kubernetes Native Object Storage MinIO, we now allow the user to store data files internally with Magda via our new storage APIs. Our metadata creation tool UI also provides options to users to allow them to store data file in Magda’s internal storage.

API Key Support

Many of our APIs require admin access. Previously it’s not easy to access those APIs programmingly or with automated tool. Since v0.0.57, we added API key support so that we can generate API keys that allow you to access the API on behalf of your account. To create a API key for your account, please check this document.

Authenticate User Locally

Magda provides wide range of SSO support. But there are some occasions when you will want to authenticate users locally. In v0.0.57, we added local authentication support. To create local users, please check this document.

Function as a service (FaaS) Support

Once you have v0.0.57 Magda deployed, you will find two more namespaces are auto-created as well. These two namespaces are suffixed with -openfaas and -openfaas-fn, which are where Function as a service (FaaS) workload is managed and run. Powered by open-sourced project OpenFaas, we allow you to extend Magda’s functionality using more abstract function model in v0.0.57 Magda. As there doesn’t have to be a Kubernetes pod running unless the function is called, the FaaS model is a perfect fit for adding data processing capability to Magda.

In future, we will look at gradually replacing our existing Connector/Minion model with an event/message-driven function model (and opportunities to simplify our event Registry Service). For the long term, we may leverage it to build a platform where user-contribute data processing capbilitities (“functions”) can be created, discovered and consumed.

If you are interested in trying it out, please check out this Magda FaaS function template. You can also find some existing features we created as FaaS Functions here.

Metadata Creation Tool

In v0.0.57, we offer a more complete Metadata creation tool. To try it out, please add your self an admin role and set your user record isAdmin field to “true”. After login with your admin account, you will see the new interface from the home page.

Try it out

To try it out v0.0.57, you can access our Helm chart repo: https://charts.magda.io and access magda chart with version 0.0.57.

You can also follow the instruction here to quickly setup a new site.

For developers, you can access npm for our latest SDK version 0.0.57.

Written by

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store