6 Common Sense Things to Consider When You’re in Charge of a Data Platform.
If you’re currently in charge of a data platform, or aspiring to be, here’s a common sense checklist to help structure your approach.
1. Reducing complexity for users
Ask yourself: “what is the skill profile of someone who wants to get insights from our platform? Can someone create new data and publish it?”
If the answer is “they need to know about AWS, Python, Airflow and SQL”, you’ve probably built something that isn’t going to get a lot of adoption across the business.
As the platform lead, you should be constantly striving to make data easier to use. That might be through integration with data visualisation tools, custom applications, Excel plugins, templated data transformations, more.
While you should be encouraging everyone in your organisation to pick up data skills, it shouldn’t be a prerequisite to use your platform.
If you’re not reducing complexity for your end-user, you might be missing out on a large audience in your organisation.
2. Cost optimisation
Everyone in the IT industry has experienced cloud bill shock at some point. If you haven’t, it’s coming.
Whether it’s an instance you’ve left accidentally running or a gradual creep up in monthly charges, cloud costs can sting if left unchecked.
As a platform lead, you will need to hawkishly scrutinise the bills. Get into the habit of discussing daily costs with the team as often as you can.
All major cloud providers offer cost alerts. Set them up.
If you’re working in an Agile environment, make it a quick agenda item in your stand up.
You should be encouraging everyone in the team to take ownership of keeping costs in check.
3. Data access
I’m a big believer in giving people in the organisation access to data. It has its challenges around sensitive data, data quality, lack of consistency in approach, but the rewards it can deliver are huge.
You don’t have all the answers. There are smart people at your organisation; people who have different thoughts, opinions and experiences and they will be able to use the data in a way that hadn’t even occurred to you or your team.
Do everything you can to minimise the risk of sharing data with people. Having a permissions and governance model that allows you to safely and confidently share data with other people is something you should strive for.
4. Security
This should really be top of the list, but I talk about it so much I needed a break.
It’s a large topic in itself, but some of the very basic questions you should be asking:
- How are you controlling authentication (username/passwords) and authorisation (who can see what) on your platform from end-to-end?
- Have you configured AWS Security Hub or Microsoft Defender for Cloud to help monitor activity in your cloud accounts?
- How do users currently log in to the platform? Can you lock down access to your corporate network only (rather than making it available on the internet)?
5. Data Governance
It sounds obvious, but you and your team need to know what you have in your platform.
Without a clear strategy around governance, you’ll have things like duplicate data, out-of-date information, undocumented logic, access issues, more.
With strong governance, users can understand where data has come from and any logic that has been applied.
More importantly, it means they can trust the data you produce.
As the platform lead, you should ideally have a strong idea of what data is currently in your environment and any potential issues or improvements that need to be made to that data.
6. Adding value
Don’t lose sight of the ultimate goal of a data platform.
You want to enable people in your business to use data to make better, more informed decisions.
You should always be asking: “where are we adding value to the organisation? Are there any areas of the business that can really benefit from our platform?”
Conversations with Finance, Marketing or Supply Chain teams will typically be useful. They usually have more data that they can deal with and are crying out for help.
With data platforms, we’re attempting to use data to solve business questions. To do that, you need to first understand the questions. The only way you can do that is to speak to your users.
DataJoey can help with all of the above.
We’ve spent our careers building cloud data platforms.
We’ve utilised the technologies that work best, and purpose built functionality where we thought the market was currently lacking. It’s all packaged into DataJoey, so that you can cut down on the amount of effort involved in building and maintaining a best-in-class data platform.
If you’re interested in learning more, please get in touch on our website here: https://datajoey.com/contact or send me an email at andrew.wild@datajoey.com
