Data lifecycle management: Validity
The validity of the cat’s data
In my previous pieces I have been discussing a drawing of a cat made on a hill by the Nasca people in Peru. No one knows, but the creation of this drawing probably was a project. Maybe they designed the cat, the cat was created somewhere on a hill, the manager was not satisfied because it was too small, and they redrew it on this hill with some changes. The cat had a lifecycle of its own — a project lifecycle.
The construction of the cat was a big project, or it was a project establishing a series of drawings. In both cases later analysis, evaluation and reporting had to take place. To allow this, you can imagine that all data at any time had to be saved. I call this ‘validity’: for every data attribute and association it has to be clear, for the end user, what the value of it was at any point in time.
The use case is clear from an analysis and (functional) trouble shooting perspective: as a user, at any point in time, I can see exactly what the status of the data is or was at a previous moment in time, allowing me to see what data decisions and actions were made at that point in time.
For all administrative systems this use case is valid. I’m even sure it is valid for all systems.
Now many system providers claim the support of validity by pointing at a log-file. However, this is not a proper solution: a log-file supports trouble shooting data from a technical perspective, while the user needs the validity of data from a (functional) process perspective: the user does not need the data, but needs the information rendered on the same functional forms (with the same meaning) as it was originally.
In Excel, for instance, you need validity on two worksheets. So you do this by copying the exact layout to two other worksheets ‘history’, and you create a macro that, after changing something in the original sheets, adds the entire line to the proper history sheet and adds the date-of-change. Now, everything is reproducible for the user in the way he/she is used to it.
Versions
In my last blog I discussed ‘Versions’. They are a feature that can be used to save historical data, but it in most cases applied to changes that have a meaning also for (external) business partners. Not all data therefore will be part of a version. Also, not all entities will have versions, but they need validity. Validity will be applicable to all data, and all associations that have no other mechanism to address historical values.
Validity for attributes
This validity applies to attributes of an entity. All important attributes (all attributes that are directly visible to or entered by the users of the entity) are clustered in a separate History-entity with date validity.
Suppose the address entered for the company was ‘Mendix Mansion’, ‘3076 NX’, ‘Rotterdam’ but on 13–7–2019 someone discovered that the zip-code has to be ‘3076 MX’ then there will be two entityData-objects. One with the old data and one with the new data. The first object will not have a Moment_From, simply because we do not know when it started, the other object will not have a Moment_To: we do not know when the data in it will be replaced.
Please note that there is a more normalized way of constructing the validity, but then data will be spread over multiple entities and some logic will have to be applied to select the right data. The construction above will also pick one-record (entityData) based on the target moment selecting all data necessary at once.
Now, if the user is asked why on a letter from early 2019 there was a zip-code ‘3076 NX’ it can be perfectly explained by selecting the data with a target date early 2019!
Validity for associations
This validity applies to associations between entities. Every association (even 1:n associations) is replaced by a m:n association containing validity.
Suppose entityB contains addresses and entityA contains companies. On some addresses multiple companies may be registered, but also companies can change to another address. In this case for company ‘Mendix BV’ there is an association object with Moment_To 13–7–2019 relating to the address with ‘3076 NX’ and another association object with Moment_From 13–7–2019 with ‘3076 MX’.
If the user works with a target date early 2019 only the ‘3076 NX’ object is selected.
Why would the business care?
Using validity offers functional business values:
·It create’s a timeline for every attribute and every association (User’s can find and access a history, track changes and explain decisions that were taken based on the data itself);
· It generate’s functional audit-trails from the application data alone;
· It record’s a proper history of every object.
· As stated before flexibility is increased by having validity on ALL entities.
Using generic validity features
It makes sense then, when adding new entities to an app, to create the validity constructions for it, including the microflow to select the data and the microflows to add new versions of the data.
Using a generic validity feature does come with advantages:
· It only needs to be developed and tested once. If it works, it can easily be added to another entity: it therefore increases the flexibility of the application;
· Functionality to maintain and access validity of data only has to be developed and tested once, and can be copied easily for new entities: it reduces time-to-market;
· Correction of a mistakes can easily be done by a microflow selecting the original data object and update it with the current timestamp as Moment_From and no Moment_To.
· Most importantly: together with versioning a complete lifecycle of the entity is created, therefore (functional) trouble shooting, error-correction, audit-ability and more is improved;
Mendix implementation
I would like Mendix to consider adding this feature to the platform: in my opinion every entity and every association needs validity entities.
Although it is not a technical issue, it certainly should be part of the functional data foundation of most administrative applications.
My implementation — Using basic Mendix features, I could not find a generic solution, so I’ve had to replicate the same solution for each entity and then change it to support that new entity. This does result in some extra work but, it’s less work than ultimately having a different solution for all the instances.
This piece is a part of a series and is a continuation from:
Read more
From the Publisher -
If you enjoyed this article you can find more like it at our Medium page or at our own Community blog site.
For the makers looking to get started, you can sign up for a free account, and get instant access to learning with our Academy.
Interested in getting more involved with our community? You can join us in our Slack community channel or for those who want to be more involved, look into joining one of our Meet ups.
