Creating commercial edge from hibernating datasets

Kumar Tanmay
Published in
6 min readDec 13, 2018


Illustrative image of stats from Varchev

Risk professionals, wealth managers and underwriters are in a race against time to stay ahead of the next unfavourable market shift. The good news is that there is an overwhelming amount of data available to mitigate risk today. But the increasing volume of data also presents a challenge. Rapidly converting this structured and duplicative data, adding contextual information and running analytics in real-time is a cumbersome and time-consuming process for risk professionals who are already over-subscribed.

The prevailing belief is that the promise of the predictive power that comes with data is the most powerful edge to emerge in the last decade or two. This data refers to the unique information hibernating in different files within the system (e.g. bank statements) and it is not available from traditional sources of the market data providers (e.g. Credit Rating Agencies). As a result, disruptors born in the digital age can swoop in and attack the market through rapid delivery of digital products and services combined with advanced algorithms and full access to information.

Every industry is going through some form of information-based disruption; this is causing businesses to modernise their practices, leveraging new data, accelerating key processes, and delivering digitally-enabled experiences in the process — YCombinator.

Why are data hibernating in the system matter?

Wealth managers, risk professionals and investors are always looking for a new source of data that gives them an edge over the market, either create a new source of income or new insights on risk mitigation. Sources of data-driven insights have evolved in last one hundred fifty years.

The recent decade of a bull market (exception: technology stocks) is a testament that traditional sources of market data have been lacking. Hedge funds heavily rely on insights known to very few and according to Credit Suisse hedge fund index, 5-year rolling return on hedge funds have dropped from 18% in 1999 to 4% in 2016. During the same duration, the data economy has exploded and here’s why technology stocks disrupted every industry it touched.

Technology stocks: Data as a moat

Over the last ten years, data has been increasingly driving business decisions. Google and Facebook have used personal data to make online ads more targeted for digital denizens. Amazon, eBay and Yelp analyse personal recommendations and preferences to help buyers make better decisions about products and services. Netflix and Amazon use data to recommend content and products. In the next ten years, most if not all decisions will be aided — if not primarily driven — by data. — Naren Gupta, Co-founder, Nexus Venture Partners

Thanks to technology companies such as Apple, Netflix and Amazon who have never let their data sit idle and used it to create seamless experiences and spoiled consumers across the globe. This abundance of data changes the nature of competition. Technology giants have always benefited from network effects of data. By collecting more data, a firm has more scope to improve its offerings, which attracts more users, generating even more data, and so on. e.g. The more data Tesla gathers from its self-driving cars, the better it can make them at driving themselves — part of the reason that less than a decade old-firm is now worth more than GM and sold more units than Mercedes in US last quarter. Vast pools of data can thus act as protective moats.

Top 10 companies by market capitalisation — Source: S&P 500 as on 10 Dec 2018

Clearly, the prevailing belief that predictive signals buried within mountains of data are the next source of an exponential edge are coming true because the underlying success factors of these technology stocks are making every piece of data work round the clock. The future of creating assets in any business will rely heavily on mining information hiding in different files and connected ways of living and doing business. That’s why even traditional sources of market data providers are failing to predict market shifts. Very often it’s not the legacy technology that stops businesses from changing but remaining tied to a legacy business model.

Examples of datasets currently producing an edge in the markets:

  1. Automobiles: GPS and telematics data from vehicles
  2. Agriculture: Satellite images and sensors placed in crops, fields and connected equipments
  3. Customer insights: App usage in telecom devices
  4. Human resources: Job listings on digital job boards
  5. Logistics: Cargo data aggregated through connected devices
  6. Retail: consumer footfall captured on cameras and transaction data
  7. Financial Health: Metrics captured by digital credit and wealth management businesses

The landscape of finding data continues to evolve and some sources of mountains of data piling up inside one’s own pavilion such as sentiment data and satellite data have already become as common as stock price history.

Low hanging datasets

One of the lowest hanging fruits of datasets is hibernating in banks and it can be monetised or used for risk mitigation like never before. Imagine when banks begin to analyse buying behaviour or assess one’s credibility by tracking payment of utility bills or payment to vendors. This can be achieved by feeding behavioural data into their own analytical engines. (There’s a solution to achieve this). Just as Amazon does, financial institutions (FIs) need to harness their considerable datasets and analytics capabilities to be relentlessly customer-centric.

It is not surprising that asset managers and banks are more ravenous about these new data sets than ever. Organisations are spending tens of thousands of dollars for untapped data that are hibernating in their systems. According to Opimas, analysis spending on non-traditional data to find edge will exceed $7 billion by 2020 at an annual growth rate of 21%.

For this, businesses are reinventing the entire business process, including reducing the number of steps required, reducing the requirement for physical documents, developing automated decision making, and dealing with regulatory and fraud issues. Data models are adjusted and rebuilt to enable better decision making, performance tracking, and customer insights.

The biggest opportunity for wealth managers, risk professionals and investors in this decade comes from the signals buried in the data generated by the digital economy. Hibernating data is the deepest, least utilised source in the world today.


Companies that fail to mitigate market risk quickly go extinct. Mitigating risk is extremely difficult and it never gets easier, a reality illustrated by deteriorating technology stocks and hedge fund returns. This is why almost every business and fund house is eager for more and newer data to develop intelligence before any unforeseen risk becomes a reality or an opportunity slips under the nose. In Wall Street parlance — Find your alpha or die!

Finding a product that helps you transform data hibernating in your digital files, e.g. data from scanned documents or PDFs, can make a big difference in capturing the information you need for data-driven insights to significantly improve the quality of your financial assets. Such products aggregate hibernating financial data from such files by transforming into quantified and actionable intelligence for its customers.

Without automated and complete aggregation from multiple data feeds to their data hub, organisations are wasting critical time on tedious and manual processes based on homegrown integrations.

Inkredo provides essential analysis and correlation that FIs need to translate raw and hibernating data into true intelligence. We help you reduce the noise of false positives in minutes, and what’s left is true information in the form of pre-built protocols, reports and dashboards that you can immediately apply and manage within your current infrastructure.