In my experience I’ve found that there are groups that embrace the truth and those that try to hide from it. I prefer working with the former.
Let me explain.
Whenever you connect data from disparate sources into a unified view, you’re going to show people things they’ve never seen before. This is the good news and the bad news. You’re going to see things you didn’t know were possible. Just remember, this is why you asked me to help connect your data! We will connect it and you are going to see everything in your data: the good, the bad, and the ugly!
These data problems and discrepancies aren’t visible in their own individual silos, but once technical and organizational boundaries are overcome and you bring data together into a unified and cohesive structure, the reality of how data is managed within your organization will be brought out of the shadows and become clearly visible for all. Enjoy.
The truth shall set you free! But for this to work, and to help move your organization in a positive direction out of the dire data mess it finds itself in, you’re going to have to be willing to swallow and digest some large chunks of truth about what your company’s data looks like.
Whenever I’ve helped to successfully bring data together from silos into a singular view, I’ve found organizations fall into one of two camps. One camp will pause, and somewhat reluctantly but still responsibly accept the reality and push onward and upward in the data unification journey. The other camp instantly goes into CYA (Cover Your A$$) mode. This CYA camp presses onward in connecting data, but progress will be sluggish and may even fizzle out, as daily data management falls back into “business as usual”.
As an example, over the years I’ve had the opportunity to work with 2 major insurers who experienced a similar problem with latitude and longitude values in their data but varied drastically in their approach to coming to terms with what we uncovered.
With one insurer, we were working with doctor’s notes to provide a dashboard examining unified patient outcome data. The data included lats and longs for treatment facilities and these were being visualized as pins on a map. I was querying data and found offices that appeared to be located in the middle of a desert in Africa. I asked about this and they instantly went into CYA mode.
I was told I must’ve done something wrong! If not me or my team, then it was insisted upon that the product I used to integrate the data must’ve screwed their data up! They were adamant that what they were seeing couldn’t be THEIR data. But the data never lies!
When we bring data together, we start by putting the data together into a unified structure. We never lead by munging any data values. I want you and your organization to see what the world looks like if we bring your data together as-is. The first step in solving a problem, is admitting you have a problem, and by showing you the reality of your own data, we can start to acknowledge the issues and begin to implement a plan for repairing your data processing pipelines.
In this instance, I pointed to the raw sources of data we’d brought together and how one system was identifying lats and longs as ( 0, 0 ) but another system providing the actual office addresses had the offices defined as Cleveland, Ohio. When they saw this, they investigated and had to acknowledge that one of their upstream systems was persisting incorrect values. They didn’t like this at all though. So, Problem Solved — They removed the map from the dashboard as they couldn’t bear to let their leadership or other teams see what we had uncovered. They added a bar graph and decided to show counts for something else instead.
I was disappointed in their decision, as I think identifying the errors across disconnected sources is a good thing. But above the silos of data, are silos of people. Unfortunately, sometimes, the politics of the organization win over the greater good. But thankfully that’s not always the case.
I experienced almost the same exact scenario with a property and casualty insurer a couple of years later working with claims data for home insurance.
We brought the data together, and showed the team on a map how their organization had offices for insuring agents showing up in the middle of an ocean off the coast of Africa. They sighed, acknowledging the reality, and admitted this was indeed the case. They knew they had an issue with an upstream system incorrectly enriching some records. They’d tried to speak to others in the organization about it before, but their pleas for data cleanup had not been addressed. They hoped now that an unbiased third party was able to come in, with no prior knowledge of their data, and demonstrate the issue to them, they could clearly communicate the problem to other teams and begin to make a decision whether to clean up the data in the upstream system, or in the system where we were now integrating their data.
This group was a pleasure to work with. They didn’t hide from the reality. Because of it, they were able to make a quick start towards positive and constructive action to fix their data management issues.
The issue with the CYA groups, as far as I can tell, is people tend to take the data discovered as a reflection of their personal work. The reality of the data’s condition can make people feel insecure about their position in the company and they seem to think that bad data is a reflection of something they or their team did. It could be, but that’s hardly ever the case.
In large organizations, with complex technology stacks acquired and built upon over decades, there is no way any one person or team is responsible for the current state of all that data. When we integrate silos of data, some of which you’ve never even seen before, you’re going to see things you haven’t. This is just the nature of the data unification beast.
I’ve experienced this a few times over the years with different groups and different data sets and have found that the quicker an organization can come to terms with the reality of their data, the quicker we can launch toward delivering results, grounded in reality, that deliver real, measurable benefits to the business. You really can’t keep doing the same thing and expect to get different results. Healthy organizations understand this.
