The Why of Data-Driven Organisations
I’m a data scientist here at Skills Matter and I’ve been brought on as part of the larger company vision to become a data-driven organisation. This places us along with pretty much every other company under the sun right now, which is by no means an exaggeration: a 2016 report by EY found that 81% of respondents thought data should be at the heart of all decision-making.
The term ‘data-driven organisation’ is very much in vogue and seems set to replace ‘startup’ as the most overused term in tech. The best definition is that it refers to a company with a culture of using data to inform business decisions, where data is open and available to as many employees as possible. Due to the number of moving parts involved, companies find themselves placed along a wide range of existent data-driven implementation, with very few actually basking in data nirvana. For us at Skills Matter, it’s still early days.
An aspect of my job, or at least what I like to tell myself, is to advocate for data and data related initiatives here. Part of this involves being able to discuss some of the big questions people may have around data-driven organisations: the what, how, and fundamentally the why of data-driven organisations.
Starting with these kinds of questions in mind I went down a rabbit hole of articles, books, and tweets to find eloquent discussions around this topic.
What became a particular point of confusion for me was how to mesh together the joint aims of:
1. Engaging with more technically challenging questions
2. Opening access to data, company-wide
These are typically the two best communicated aspects of a data-driven organisation.
At face value, these two aspects seem to be at odds with one another. But perhaps they have some things in common.
Hierarchy of Needs
What initially inspired this article was another by Monica Rogati titled ‘The AI Hierarchy of Needs’. Rogati writes about how the aspects of data science and machine learning that tend to make the headlines (such as deep learning and A/B testing) can only be approached once the more unglamorous groundwork is laid (such as data collection and storage infrastructure). This becomes what she terms the ‘data science hierarchy of needs’ — a somewhat tongue in cheek reference to Maslow’s hierarchy of needs (why are these kinds of hierarchies the go-to illustration for discussing data science initiatives?)
In fact I’d say this comparison with Maslow’s hierarchy of needs is in itself quite revealing as to the motives of a lot of companies in this area, not to mention the data scientists and analysts. Progress up the hierarchy is a journey to do more aspirational work as it is seen as more desirable from the perspectives of all parties, not simply because of the technical dependencies of a higher layer on a lower layer.
In becoming a data-driven organisation, you should be thinking of the resulting generated business value at every step of the process.
Business Value
Why do we care about moving up this pyramid in the first place? One answer is to be able to deal with more interesting problems and to address more technical questions. This tends to also allow companies to ask more predictive and prescriptive questions about their business, such as what would happen in a given scenario or what recommended actions to take in a given scenario.
But why do we want to ask more technical questions in the first place?
I’ve come to understand that this is to create greater business value through richer data collection and analysis.
In becoming a data-driven organisation, you should be thinking of the resulting generated business value at every step of the process. This is where things clicked for me, and where I could begin to equate the multiple ideas contained within the definition of a data-driven business.
Before going any further, I should probably define what I mean by business value, which is a fairly hand-wavy concept. It tends to be thought of as referring to the combination of the tangible and intangible assets of a given company, which are very much specific to that company. These tend to relate to the practical concerns of a business as well as their values which they don’t want to compromise. To be a bit more concrete, at Skills Matter of course we want to make money, but not without also ensuring we are both providing the space for and delivering quality events to as many and as diverse an audience as possible and to maintain our position as a premier location for tech professionals.
So if increasing business value is the underlying metric of moving up this hierarchy, is moving up the hierarchy the only way to do so?
The answer is no. By focusing on business value as our metric of success, many different components come into play and overlap one another. We can begin to think about our progress up the AI hierarchy of needs as being a single aspect in a much bigger picture. So what other ingredients are there to a data-driven organisation, and ingredients that are more in line with our original definition? The following come from “Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking” by Foster Provost & Tom Fawcett.
Achieving a Competitive Advantage
Thinking specifically about data, we want to make sure that the data we have (or want to collect) is or will become a valuable asset to the company. To get to this point, we need to make sure we’re making the right kind of decisions when thinking about overall business strategy — thinking in terms of the CRISP-DM model. That is to say, does our data collection makes sense in relation to how we make use of it? Are we collecting data other competitors are not collecting, or if not are we utilising it in a unique way?
Sustaining a Competitive Advantage
Following on from the previous point, it not enough to simply find ourselves with a competitive advantage, but to ensure this persists. To ensure the long term value of our data assets we want to make sure we are continually investing in the talent and technology responsible for this initial lead. This becomes something of a virtuous circle, as in the long term an initial competitive advantage develops into a historical advantage that makes it that much harder for competitors to catch up.
Developing and Maintaining a True Data-Driven Culture
There are many different aspects to this, but generally it means that everyone in a company is empowered to look at relevant, typically live data, and make use of it as part of their day-to-day role.
Importantly, by opening up access to company data we should anticipate being able to look at more relevant questions for the business. When it comes to data science, it is very often that domain knowledge comes before technical knowledge in ensuring we’re asking the right questions, collecting the right data, and drawing reasonable conclusions.
And that is the core connection we can make between pursuing more technically challenging questions and providing access company-wide.
An Economist Intelligence Unit report found that 83% of respondents believe that ‘most to all’ employees should at least be familiar with data and data analysis for occasional use, if not regularly as part of their jobs. This should be the least surprising of the lot as this is typically thought of as the defining characteristics of a data-driven organisation and brings us full circle with our starting definition.
To wrap up, we have seen that the real driving question, the why of data-driven organisations, is arguably about generating greater business value using data as a starting point. The actual development and growth of a data-driven organisation is multifaceted. With this understanding in mind, we can begin to think in turn about how each component of a data-driven organisation contributes to this overall goal, and how they begin to overlap with each other.
For more on this topic, I highly recommend the following:
- Creating a Data-Driven Enterprise with DataOps: Insights from Facebook, Uber, LinkedIn, Twitter, and eBay by Ashish Thusoo & Joydeep Sen Sarma
- Data Driven: Creating a Data Culture by DJ Patil & Hilary Mason
Tom Martin spoke at Infiniteconf (the conference on Big Data & AI) 2018, on Thursday, 5th — Friday, 6th July in London. You can watch his talk for free here: https://skillsmatter.com/skillscasts/11885-lightning-talk-the-why-of-data-driven-organisations
