The Startup
Published in

The Startup

Building a Data Platform to Enable Analytics and AI-Driven Innovation

Build a Data Mesh & Set up MLOps

Key Challenges

  • The size of data that you will employ will increase 30–100% year on year. You are looking at a 5x data growth over the next 3–4 years. Do not build your infrastructure for the data you currently have. Plan for growth.
  • 25% of your data will be streaming data. Avoid the temptation of building a batch data processing platform. You will want to unify batch and stream processing.
  • Data quality reduces the farther away from the originating team the data gets. So, you will have to provide domain experts control over the data. Don’t centralize data in IT.
  • The greatest value in ML/AI will be obtained by combining data that you have across your organization and even data shared by partners. Breaking silos and building a data culture will be key.
  • Much of your data will be unstructured — images, video, audio (chat), and free form text. You will be building data and ML pipelines that derive insights from unstructured data.
  • AI/ML skills will be scarce. You will have to take advantage of packaged AI solutions and systems that democratize machine learning.

The 5-step journey

Step 1: Simplify operations and lower the total cost of ownership

Step 2: Break down silos, democratize analytics, and build a data culture

Step 3: Make decisions in context, faster

Step 4: Leapfrog with end-to-end AI Solutions

Step 5: Empower data and ML teams with scaled AI platforms

Next Steps

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Lak Lakshmanan

Operating Executive at a technology investment firm; articles are personal observations and not investment advice.