3 ways to succeed in implementing data -intensive applications

Data-intensive applications are on the rise and — like homegrown applications — the need to custom build them for every data-intensive problem out there will widdle away with the rise of software that’s pre-built and easily deployable to solve common pain points of the modern enterprise.

Surprisingly, people who set their teams on a path to implementing data-intensive applications don’t always seem to realize that they are building systems from the ground level up.

For executives and directors who are brought in from big tech to digitally transform non-digital native companies, this can be a massive problem.

Architecting a data-intensive application by using Spark, Kafka, Airflow and S3 is akin to building an app using Ruby on Rails, Postgres and React. And don’t kid yourself, Databricks and Confluent products facilitate the building of this infrastructure. They are not end-to-end solutions on their own.

Enterprises today, especially the non-digital native ones, tend to vastly underestimate the costs of building and managing data-intensive applications, both from a compute perspective but more importantly from a people and time perspective.

That’s not to say that none of them have been successful. The success stories are out there; from acceleration of query performance to improvements in time-to-market and optimization of compute for specific types of jobs. If you want to read about it, a product marketer has written about it.

The problem with these success stories is that they often neglect the total cost of ownership of these systems. The people costs, the ongoing maintenance costs and the hard-to-measure opportunity cost.

My advice for senior executives and architects is to:

1. Move your technical talent up the stack and closer to the business, that means focusing efforts on insights generation, data modeling and building predictive models
2. Buy what you don’t absolutely need to build, especially non-value add activities like moving data around from one system to another
3. Architect for your current level of scale and the ever changing technology landscape. That means building in flexibility into your architectures and understanding that your trade-offs change with data scale

Moving your people up the stack will result in a greater number of data driven initiatives getting done and tighter coupling of the work to business value.

Adopting a buy don’t build methodology can accelerate your time to market dramatically and reduce your maintenance costs. Not only that, but you can start to build teams that look more deeply into optimizing your particular architecture. This is the buy then build model that many companies have successfully adopted.

Lastly, starting small and scaling up is critical to a successful architecture. By not skipping the scale up phase you can save yourself the many headaches of having inexperienced teams fall flat on their faces as they try to move straight into hyperscale territory. Start small, build momentum and drive towards the business outcomes that the business needs.