Every organization during its lifetime is continuously acquiring data. The data that is useful to the organization is mostly stored in the database or Excel sheets where the management can do analytics. But there is another kind of data that is captured by organizations but is hardly used. This kind of data is mostly unstructured, such as documents, images, videos, logs, etc. It is difficult to analyse such data, and it forms almost 80% to 90% of the data that an organization has. Such kind of data is called dark data. Gartner defines dark data as the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes.
Why does an organization want to keep such kind of data? The reason is that it feels it can be useful in the future. You must have yourself noticed numerous flaws in storing such kind of data. Let us try to understand few of those flaws:
- Storage Cost: The data lies in the storage without ever being analysed or used. This adds up to the server cost.
- Security: Since the data is stored, which may contain important secret company information such as customer records, it is available for misuse, which might damage the company’s reputation.
- Opportunity Cost: Not able to analyse dark data can lead to opportunity cost, as if there are ways to understand such kind of data it might save cost to the organization or increase revenue drastically.
- Usefulness: Dark data may be only useful if it is analysed as soon as it is collected by the organization. There is no use for such data if it is analysed three years later.
How is Artificial Intelligence (AI) going to help?
AI techniques are slowly maturing in understanding dark data. Current AI systems can be trained to understand images, videos, text, and any kind of unstructured data. An organization that can generate lots of dark data can train AI systems to perform analytics and gather insights. Once analysed organizations will no more require to store dark data, but will need only to store the analytics.
At Cere Labs we have been applying AI to draw insights out of dark data and intend to help organizations make the most of it. We have gained significant momentum in applying advanced AI techniques in understanding handwritten forms and various business documents. We believe that the proper Utilization of dark data can unlock new possibilities for an organization and the rate at which AI adoption is taking place today, dark data analytics is going to be the norm in the next 2–3 years.
~ Written by Siddhesh Wagle (Head Of Research, Cere Labs)