Polylithic Data Architecture

Polythictic Architecture and Lego

I use the word ‘Polylithic’ as an umbrella term for micro-services and modern data architecture solutions. This term being the contrast of ‘Monolithic’ which is the umbrella term for old data architecture solutions. One can think of it like Lego…

If you played with Lego as a kid you knew that some of the cooler bits you could build with were the small, funky and useful ones that could connect many other blocks. They could create more unique arrangements. And the ones we disliked the most were the bigger Lego chunks that served only one purpose, and could not be used for much else. I don’t know if you could relate. For me, it was incredibly frustrating having this one big chunk of plastic exist with so much potential yet with so little versatility. This is much like Data Architectures that are Monolithic and Polylithic in nature.

Data Warehouse vs. Data Lakes

The old school method of creating a data warehouse has now become obsolete. Even if it is done in the cloud, and more commonly done on premises in the old-school way. The architecture of a data warehouse we used to know can not keep up with the rapid changes in a business. Most businesses understand that their data exists to give them information to react to. However in today’s fast paced industries, reacting to intel is already too late. What is needed is accurate predictions and immediate, actionable information from business data.

Since the monolithic and structured data warehouses are fixed in their schema, businesses are not able to rapidly ask new questions, or change existing questions to suit their needs.

In a matter of months, these warehouses are no longer relevant to the evolving business and its requirements. This is where the Polylithic architecture comes into play. Where we have micro-services placed on top of a data lake, or collection of raw unstructured data. These services in the cloud (cloud being a vendor or perhaps even a local cloud solution) can be spun up and shut down without a second-thought.

We no longer find value in a structured space of data, and rather see the qualities of rapid development, quick change, and the total lack of permanent hardware when hovering our solutions over a wide set of raw and unstructured business data.

The idea of permanent structures have been replaced with methodologies and ideas like ‘structure on read’ or ‘none permanent’ and ‘volatile services’. It’s essentially a world where nothing persists in the way that it used to.

We can compare today’s data technologies in a business as behaving much like the newer generations, kids of today. Myself being one of them, being all too aware of my shortening attention span in the realm of mass media. We want faster, quicker, instant, with a healthy dose of successful solutions given to us now. Before we lose our attention and move to the next buzzing question. We want many little things that can run independently of each other. Which can be dropped without costly or time-wasting consequences once they have served their purpose or were discovered to have no value at all.

Moving from the Monolithic, Siloed and Warehouse of the data world; to the most literal of new, scary, different and unusual of micro, cloud, pipeline and Polylithic.

It is a brilliant solution to a slow data donkey trying to gallop with the torqued up racehorse of live data found in every business in the moment.

What is better, a painting or a photo?

Articles such as this should not be written to hate on the old ways of data architecture and solutions. Think of it like, the invention of the camera that superseded the painting of portraits. Portraits had their place and value at the time. They now exist as valuable historical novelties and artifacts. But nobody really wants to sit still in front of an artist for a month to get a representation of their likeness and exorbitant prices. When all we really need is a photo from a camera that costs next to nothing to develop. And if we throw the photo away, it would be easy to re-create or recover. And look where we are now when society decided to adopt the new technology. Selfies are free and instant and can be found on anything with a CPU and a lens.

In this regard, thinking that old Monolithic Architecture is totally redundant is not necessarily correct. There is always a time and place for everything. Even in the world of technology.

Try to be agnostic…

All of this Polylithic Architecture has begun to develop other greater ideas for the business world. Where solutions become agnostic to technological and vendor preferences. We now see no reason to stick to one service, vendor or application simply because they are the presumed best fit for our needs. When needs are constantly evolving, and new unusual problems are coming out of the woodwork. With micro services and the decreasing of hardware and permanent architectural solutions. We can afford to chop and change as we please to deal with everyday issues. If we don’t like service A we can move to service B in a heartbeat. Or, service B has a great tool for one section of the business, but service A is better suited for the rest. Why not use both? Agnostic development permits and perhaps even encourages ideas such as this. It no longer comes across as an absurd and costly resolution to frustrating problems when we want to drop a service once it has served its purpose.

Let me try to explain myself

Imagine a BI visualization tool in a company that is powerful and versatile. The product came out a year ago and everyone is jumping on the bandwagon. All your queries and calculations can be done within this tool, all you need to do is connect to your data. Over the next year or so, the dashboards, visualizations and scripts in the tool expands rapidly. And the business becomes more and more dependent on it. One day, the owners of the visualization product stumble and their product goes bust. No more support… no more updates. The tool is rapidly becoming obsolete and your business has all its calculations, scripts and configurations inside of it. It would take years to get it out and into another, better BI visualization tool. BI aside, scenarios like this are a bit of a nightmare to consider.

The idea of agnostic development in a Polylithic Architecture means all these scripts and calculations are not done in a particular vendor or application. But rather is done as close to the raw business data as possible. The BI Visualization tool does nothing but present the data to business. That way, should the tool fall flat in the near future, it is an effortless process to switch to another one. Modern technologies support ideas such as this.

This example is what I mean when I say that “it is a world where nothing persists in the way that it used to.’’ That BI tool became too much of a dependency, and replacing it costs time and money.

If we were to adopt the agnostic methodologies in Data Engineering and Polylithic Architecture, we would find that coping with rapid changes is feasible to our businesses.