Record your data, the smart way. Here is how we did it.
In the last years we have been told time and time again that data is a key asset of organization. Data has even been called the oil of the digital economy. That makes it all the more compelling that some organizations fail to store the data they produce. While it might not be clear what the use of the data you record is, it is often smart to use a simple service to store it. This way you make sure that you have access to all data whenever the opportunity arise, so you can perform analytics on it.
Recently we have experienced that organizations sometimes gather large amounts of data but they do not record it. Last week we sat down with such a client. They invited us to talk about innovation and data. Especially because of their decision to start an innovation lab. a set of innovation partners to enhance their customer journey. However their proposal lacked a solid data strategy. So here is what we came up with. No matter what you do innovation you are developing, make sure you collect the data it gathers.
We advised them to do so for three reasons. First, one of the key assets an innovation lab produces is data. It would be foolish to not think about a strategy that involves this data. Second, being able to provide data to new partners could catalyze innovation. Finally data can provide answers to questions that have not been taught up before.
While the client recognized the necessity to start recording the data their projects produced, they gave us some serious feedback on our proposal. First they mentioned that they did not have the IT resources (staff & hardware) to manage such a project. Second they said that storing these huge amounts of data would be expensive. Finally they argued that creating a project to handle these amounts of data would consume scarce time and resources.
So we decided to come up with a concept that would address these issues. We looked at what the easiest and most scalable way to store data is. Finally we came up with an idea for a generic API. We offered them a generic API based on lambda functions (FaaS) as API endpoints and a NoSQL database. This way you, or in this instance our client, only pays for the amount of requests our service handles combined with a low service fee for actually storing the data. Storing data with this simple configuration is such a compelling idea that we ended up offering it to several of our customers.
There is one thing that keeps bothering me, and it is platform dependency. AWS seems to be very skilled at luring developers in, without having a real way out. To be truly flexible, a platform independent, solution would be great. So that is something I will look into in the upcoming weeks.