Engineering A Data-Driven Infrastructure
A Seven Stage Checklist To Get You Started
Does your company have the infrastructure it needs to be data-driven? If I had to bet, I am going with — No. It is a sad reality that most organizations fail to deliver in one of more areas. This article provides a cursory checklist of items that are non-negotiable, if you are committed to make great and measurable decisions.
Item #1: Instrumentation — Are you recording it?
Data-driven companies need data. If you don’t log it, you will never understand it, measure it, or improve it. Instrument everything!
Item #2: Collection & Storage — Is it being collected?
Logs are great, databases SHOULD be better. Sorry… some aren’t but I leave that for our articles by Seth Goldberg and Greg Anderson. After you log it, you need to store it. Preferably with as little molestation as possible. If there is too much T in your ETL, you will suffer later.
Item #3: Organize — How are you storing it?
If all your data rest in a single production database, I won my bet. If your adb (analytic database) does not mirror your production db AND reflect your business model, I won again. You may be able to get away with not mirroring, but if your adb is just a lagged prod db… you aren’t ready.
Item #4: Access — Can people use it?
If only that guy in the back corner can access or understand your data, I win again. You see why I am willing to make the bet. Well-organized data should be easy to access. Access means “understand” here, not touch, change, move, or anything else.
Item #5: Audit — Is it recent? Is it reliable? Is it right?
Another article in itself. In most warehouses I audit — 50–80% of the tables have not been updated in more than 6 weeks. That makes it better described as “discarded history” than “useful information”. If you remember the last warehouse outage, if you can’t box the numbers, if you have two or more numbers for everything — pay up.
No seriously — hire a good analytic services group to audit and fix it! A good time for self-promotion:
Item #6: Reporting — Is it reported?
While related to #4, this is a bit larger. Do you have solid KPIs? Are they reported on? Are they Leading or Lagging indicators? Does your reporting tie to the activities of your employees or the levers of your business? Another article of its own — an interesting take here.
Item #7: Action — Did you develop the proper feedback hooks?
One of the biggest killers of analytic insight is the platform’s total inability to leverage feedback. If you’ve gotten to this step, you need your platforms to be able to receive feedback coming back from your decision teams and the adb. This is several articles but:
- provide a hook or two to collect a flag or two each evening
- provide a hook or two to collect a flag once the user is ID’ed
- create hooks to utilize geography, time, and other classic context data
Basically, design your systems in advance to respond three to five pieces of data that will later be fed back via your data-driven processes. This is the essence of data-driven experience. Nothing fancy — simple things like change a link, change an image, change some text based on simple flags.
And these flags have no need to be crazy detailed. A simple single digit numeric is sufficient to do 80–95% of anything you may ever need. But if you don’t add them, you will waste tons of effort, energy, and money. I have witnessed Fortune 500’s in the Technology space do this to the tune of billions in lost revenue.
That is far from all…
…but it is sufficient to get you started. It is sufficient to capture the majority of the major returns and revenue opportunities. It is sufficient for you to win my bet. Do you have all of it? Do you even know? Good luck! And thanks for reading!
