Infinite Data Experience
“I consider that a man’s brain originally is like a little empty attic, and you have to stock it with such furniture as you choose. A fool takes in all the lumber of every sort that he comes across, so that the knowledge which might be useful to him gets crowded out, or at best is jumbled up with a lot of other things so that he has a difficulty in laying his hands upon it. Now the skillful workman is very careful indeed as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these, he has a large assortment and all in the most perfect order. It is a mistake to think that that little room has elastic walls and can distend to any extent. Depend upon it there comes a time when for every addition of knowledge you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.”
― Sir Arthur Conan Doyle, A Study in Scarlet
As a software architect, it’s clear for me, that if I can imagine it then we can build it. Especially nowadays when a palette of technologies is rich as never before. I like databases from elasticsearch as elastic data cache through HDFS databases like Hive or HBASE and I never forget about the fun I have had with Oracle SQL:)
As an architect, I see them all as barrels in all of its’ sloth where I have to make two steps:
- Put data into the barrel
- Get data from the barrel
Oh yes, a great data architect/engineer can make it as fast as possible in all ways the word fast can be applied to databases. Today I want to share with You an idea of database-less architecture. I know it’s not fair. Nowadays technologies are not so far. But if I utilize streaming especially with help Kafka I love as a streaming platform, then with minor help of databases, I can make a simulation of database-less idea. I call it Infinite Data Streaming.
The video below shows the event represented as a small star. Then there are streaming spots represented by cylinders(in the real implementation I work on they are implemented via Kafka streaming or KSQL). An event goes through these streaming spots, it mounts up thanks to transformations, aggregations, enrichment and finally it’s experienced by the user when it’s put back into user world. This process of mounting up looks infinite for me, that’s why I used infinite Canon a 2 per Tonus, from Bach’s “Musical Offering” as a background for this video. In fact, this canon inspired me a lot.
Moreover, there is an infinite loop between two such streaming apps and then real Infinite Data Experience starts.
Thank You for watching:)
The Crème de la Crème will come when especially the modulator uses streaming BI idea as brilliantly described by Mark Palmer in his article How To Query the Future. With the help of Reinforcement learning, it promises a lot of thrilling journeys.
So let’s focus on how to make data to be helpfully agile rather than well conservated inside database barrels because as Mr. Koulemainen in my favorite finish metal movie Heavy Reissu said: ”It’s better to sh*t yourself than to be forever constipated:)”
