Avengers: Data Governance
Ok, ok so this may not be the next film in the Marvel franchise but how similar to the real world is the Avengers? Let’s have a look….
I’m Duncan and I’m a Senior Sales Engineer at Snowflake. Opinions expressed here are solely my own and do not represent the views or opinions of my employer.
This weekend I got my geek on and rewatched films from the Marvel Cinematic Universe (MCU). For those who don’t know, the MCU is an American media franchise and shared universe centred on a series of superhero films produced by Marvel Studios. The films are based on characters that appear in American comic books published by Marvel Comics. Characters such as Thor, Black Widow, Captain Marvel, Captain America, Iron Man etc etc….
After finishing watching Endgame I was struck by how much the fate of the universe depends on a strong team with a focus and goal. The creation of the Avengers was no accident. One of the early films in the timeline, Avengers Assemble, sees the creation of “The Avengers”
“There was an idea, Stark knows this, called the Avengers Initiative. The idea was to bring together a group of of remarkable people to see if they could become something more. To see if they could work together when we needed them to, to fight the battles that we never could.” — Nick Fury, Director of S.H.I.E.L.D
The Infinity Stones of Data Governance
Before the universe began, six singularities existed. The Big Bang then formed concentrated crystals out of them and sent the six stones hurtling throughout the universe.
Each stone represented a different aspect of the universe (Space, Mind, Reality, Power, Time, and Soul). These stones remind me very much of some of the underlying principles of Data Governance (DG) a.k.a Governance Stones:
- Space i.e. Is it in the right place? giving us the “Misfielded Stone”
- Mind i.e. Was it done right? giving us the “Process Stone”
- Reality i.e. Is the data real ? giving us the “Made Up Stone”
- Power i.e. Who did this? giving us the “People Stone”
- Time i.e. Is it up to date? giving us the “Timeliness Stone”
- Soul i.e. Is it personal and or sensitive? giving us the “GDPR Stone”
Taking any one of the Governance Stones out of picture causes a huge problem for organisations just as it did for the Avengers when trying to reset the timeline. Two of the main characters in Avengers Endgame had a lengthy discussion where the one said taking an Infinity Stone from the past would doom that branched reality. So while removing one of the Governance Stones may not doom our reality we do need to consider what happens if data is neither complete, accurate or appropriate? What happens if we go against policy and share sensitive data? Removing any one of these stones can have serious financial, and sometimes worse, penalties.
This is why DG is such a key component in the way companies operate. Smart companies don’t see DG and a laborious overburdening set of things that must be obeyed and hinder transformational projects but instead use it as bedrock for best practice to go beyond compliance and drive performance. A data governance framework (DGF) enables the business to define and document standards and norms, accountability, and ownership.
DG should be embedded in the company’s Data Management Strategy and should span all aspects from data privacy and accountability to training new hires on treating data as an asset. Most smart organisations implement a DGF. A DGF is a logical structure for classifying, organising, and communicating complex activities involved in making decisions about and taking action on enterprise data.
All organisations need to be able to make decisions about how to manage data, realise value from it, minimise cost and complexity, manage risk, and ensure compliance with ever-growing legal, regulatory, and other requirements. Management and staff need to make good decisions — decisions that stick. They need to reach consensus on how to “decide how to decide.” They need to create rules, ensure that the rules are being followed, and to deal with noncompliance, ambiguities, and issues.
In short, they need to do more than manage data; they need a governance system that sets the rules of engagement for management activities. Small organizations, or ones with simple data environments, may be able to succeed in these goals through an informal system of governance. They may not even be aware of when they are switching between making management decisions and broader governance decisions. On the other hand, larger organisations, or ones with more complex data or compliance environments, generally find that they need to step back and agree upon a more formal system of governance.
In short, a DGF is like putting all the Data Governance Stones in a protective case, probably stored in Odin’s (Thor’s father) treasure room to make sure they’re properly looked after.
Having a DGF is not enough, It’s only the start — the guiding principles. One needs to have a plan on how to execute against the framework
Develop Your Plan for Data Governance
Saving the world from the likes of Hela and Thanos may require stealth and misdirection, but victories (and momentum) come from picking off the low-hanging fruit and sending in the Hulk to lay waste to everything within reach. Victory for the Avengers relies on the right plan of attack before a hammer is ever thrown or the smashing starts. For example:
- If a company’s goal is to improve operational efficiency and/or improve the customer experience then there needs to be “fit for purpose” data supporting this. Customers are no longer satisfied with blanket marketing attempts to sell them products they are not interested in. They demand highly personalised and relevant experiences, and this requires marketing to become more data-driven than ever.
- If risk mitigation and compliance are the goals, this requires putting a data governance strategy in place to find and protect hidden data in a company’s repositories. Here, data governance best practices should define how sensitive data is controlled (cleaned, converted, redacted) and protected so that it no longer poses a risk.
- For most businesses, competitive advantage is gained by increasing speed-to-market. In these instances, data governance best practices involve automatically extracting the necessary values from documents and modifying the metadata so that the content can be used to fuel analytics engines that will identify trends and opportunities for new product development.
All the data hidden around the organisation has the potential to deliver true competitive advantage, but to leverage the data, it must first be assembled by removing borders and access barriers, identifying and eliminating the ROT [redundant, obsolete and trivial data] and transforming it into actionable data.
The key to a successful Data Governance Strategy is in keeping it outcome focussed. There is no point in trying to change the entire culture in one go. It will fail. There is no point trying to roll out all the policies under the sun. It too will fail. The notion here should — Start Small, Think Big, Scale Fast.
So we have some guiding principles (aka The Governance Stones) and we have a plan. All we need now is the team to bring it to life…
Assemble Your Diverse Data Governance Team
Alone, the individual Avengers are formidable, but as a team they are invincible, because they are able to leverage their different, but complementary skill sets. Iron Man and War Machine rule the skies. Black Widow and Hawkeye play on stealth. Thor and Hulk are an indestructible power duo. A data governance strategy is no different. You need to play to the strengths of the team and deploy accordingly.
Furthermore, a successful data governance strategy brings together all the stakeholders who can provide different types of expertise and insight in a digital transformation project — harnessing the power of complementary skill sets in disparate departments. This creates a multifunctional group that is greater than the sum of its parts
Anticipate The Next Data Enemy
The Avengers saved the planet in the Assembled movie. The battle was won but the job is not done. The MCU seems like it’s a continuous, endless battle. This is true for Data Governance, always be refining and revising to world around us. Be on your guard data citizens!
Pick the Right Weapons
When I say weapons, what I really mean is Technology. To be clear, there is no such thing as a Data Governance Solution — no single vendor in the marketplace today provides EVERY capability needed. Every customer I’ve been involved with on this front ends up deploying a combination of technologies to support and underpin the execution of the policies and processes that make up the DG strategy and DGF.
Celebrate the Wins but Stay Focused
At the end of the first movie, the Avengers are a wreck. Half of New York lies in ruins, but they have saved the world. This is when Iron Man, crawling out of the rubble says, “There’s a shawarma joint about two blocks from here. I don’t know what it is, but I wanna try it.” After each battle, get up, dust yourself down and take the time to celebrate their victories and capture lessons for the next sequel.
This last piece is really important. Keep the energy and enthusiasm high — its contagious!
Shameless plug number 1: Snowflake provides some great capabilities in the Data Governance space — download the free eBook:
Shameless plug number 2: I’m presenting at AWS Summit in London on the 27th on Snowflake and AWS Sagemaker
Shameless plug number 3: Snow Summit is happening!
Snowflake Summit is taking over Las Vegas from June 13–16, 2022! This will be where the entire global Data Cloud network comes together to collaborate and learn. Join us for a week packed with the latest Snowflake announcements, hundreds of sessions for all levels of expertise, hands-on experiences, and much more.
