Using Solidatus to reduce noise in data lineage
When organisations deal with interconnected data systems, developers within the organisations will inevitably need to write software which implements the processes taking place in the relationships.
This could be picking up a set of data fields from system A and writing them to system B.
The task itself may be relatively straightforward, but from the process alone a developer cannot understand how it fits into the landscape containing system A and system B.
There may be thousands of data fields in systems A and B. There may then be hundreds more systems each with large sets of interconnected fields. Of all of this data, the developer is only interested in a small subset of fields in system A which will be read and then written to system B.
In terms of the model metadata, the developer needs:
- to view the data fields of system A and B to understand the landscape they are developing against
- to view only those data fields which are relevant
Solidatus has a flexible filtering engine which utilises its powerful query language to remove visual development noise.
Solidatus allows attributes, objects, and whole systems to be selectively filtered to the developer’s requirements. When filtered, transitions are propagated, so lineage is not lost in the visualisation. The query language allows users to craft filters to cater for a large variety of use-cases.
An example of what the query language can do is given below:
Show: isAttribute() and owner = ‘Daniel’
Hides all attributes except those with metadata key ‘owner’ with value ‘Daniel’
Hide: isAttribute() and $numIncoming = 0 and $numOutgoing = 0
Hides all attributes with no incoming or outgoing transitions