A Torrid Relationship with Business Intelligence
When I first met Business Intelligence it was a hot relationship, full of wonderful experiences, great scenery, long walks, and conversations with data. Things were going fast and we just held on and let the good times roll. We got along so well, and then one day it got hot! This is the part of story where you put the kids to bed. Open a bottle of wine. Put on some soft music and open up your inhibitions! BI and I were making crazy passionate dashboards and visuals. It got crazy at first. We broke through and connected with the mainframe and eventually had our own BI stack. We went way further and faster that way! We could go all night long at a fast and furious pace. We even got requests from other people and organizations to perform! These crazy acts and my passion for BI turned into a thriving love affair. Our relationship not only survived the seven year itch it was on fire.
And then it happened. Just like that, BI started to slow down. Requests were getting crazy. The need for speed was more ravenous than we expected. MORE DATA, MORE DATA, MORE PEOPLE! OH NO! The requests were just too much to handle, the data sizes too big! Overnight my relationship with BI got weird and uncomfortable. We just couldn’t keep up. We started to blame ourselves and others. The network is slow, the SAN is not fast enough, and you BI server, you just can’t scale to my needs. Your index structure isn’t right! The blame game started. The friction got out of hand but not in a good way.
So we invited OLAP to join us in our relationship. Woooo, we loved OLAP. It was fast and could do things very fast. CUBES WERE HOT! It was a virtual Twister party of intrigue and pivot tables. Soon the three of us were spicing things up, but OLAP was complex, costly, and was sort of like a Diva. OLAP needs lots of attention and brains to make it work properly. It also required its own place to live. OLAP was also stubborn and needs lots of attention to keep up with the fast pace of our job requests. It demanded its own data, ETL, and this weird language called MDX. It turned out OLAP was just another tool in my love affair with BI. A crutch in my relationship with BI.
Then one day it happened. It was a mind blow. It was the fastest thing I had ever seen. Like a Ferrari it raced into our lives. In-memory came to town and blew our minds! In-memory was sleek, high-speed/low drag. It moved at the speed of thought. However, like a Ferrari this high speed maniac required a lot of maintenance and specialized skills. It required its own copy of data and was another tool to learn. In memory was also very tender and you really have to know what you are doing. You have to size it just right, not only for data size but also workload size. If you don’t then In-memory has a fit and spills to disk or just shuts down entirely. Oh well, another Diva!
Then one dreaded day it happened, one I will never forget! I got a ‘Dear John’ letter from my BI stack. “It’s just too much. All the data copies, all the hassle, all the ETL, all the data movement we couldn’t take it anymore. BUT HADOOP! HADOOP, WE JUST CANT DO IT!” My BI left me stranded, alone, disappointed, and deeply saddened. The scale, the burden, the size, the concurrency, OH MY! BI left me. I was heartbroken and lost.
They say a broken heart takes the same time that a broken bone takes to heal. But I couldn’t wait. Then I found this new thing, it had all the appeal of the BI tool that just left me. Heck, it even claimed to be able to help fix my relationship with my traditional BI partner. I was intrigued! I was alive! I felt the tingle of love again! Being cautious I entered into this relationship lightly. I found it easy to use. It had neat features like Machine Assisted Visuals! It ran on Apache Hadoop and was the only BI platform where you could put your Data and BI tool on the same stack! I couldn’t believe it! So I studied it, learned that I could make elegant and beautiful things! But then it got super interesting. I noticed that these beautiful things were slow too. Oh no, I am back where I was? Fear not this tool said I have it covered. Smart Acceleration, what’s that? Well it is the ability to speed up a dashboard without having to write code, move data, make copies, do expensive ETL, or have to become a NASA rocket scientist to make it work. “WHAT!, THAT IS TOO GOOD TO BE TRUE.” “Nope”, BI said, it’s is very real. “WHAT IS IT?” I asked. “It’s Arcadia Enterprise.” Learn more about it here: ARCADIA ENTERPRISE! BI even told me that it does Kafka Topics and KSQL. So I downloaded a free version of it (ARCADIA INSTANT) and started my love affair with BI again!
BI and I are back at it like a couple of crazy teenagers!