Streaming Unstructured Data — Analytics in Real Time

The Big Data Explosion

There is much current writing on the phenomena of big data with the explosion in data volumes and new and varied types of sources. Data analysis is vital to businesses today. It informs everything — cost analysis, inventory control, financial reporting, lead generation, customer profiles, sales acceleration and more. As economic and societal activities have become increasingly digitized in peoples’ lives, vast amounts of data are created every moment. As technologies have become more powerful and advanced, enterprises have been able to access new sources and types of data.

Not so long ago, businesses mostly dealt with data science and analytics applied to structured data, a data model of specific fields or tables of variables like relational databases and spreadsheets, which can be easily queried. An example is a customer relationship management (CRM) database. Organizations use traditional business intelligence and data warehousing tools for decision making from structured data sources, which provide analytics from a historical perspective. Sometimes it would take days or longer to produce reports.

In today’s digital environment, businesses have access to vast amounts of unstructured and semi-structured data already in digital form. Unstructured data does not have defined fields or tables and includes such things as pictures, graphics, reports or PDF files. Semi-structured data has some structured fields (such as the sender and recipient in an email) as well as unstructured data (the body text of the email). Another example would be a picture (unstructured) with a keyword tag associating a name, location or date with it (structured).

Increased computing power and speed have dramatically reduced data processing cycle time, increasing the ability to query data and get a rapid response. The ability to store large amounts of data has also dramatically increased. Technologies are now available that can handle streaming data such as event streams, sensor data or machine data.

Real-Time Analytics

These advancements have given rise to “Real-Time Analytics,” sometimes called “Operations Intelligence,” a field of analytics that provides visibility into business processes, events and operations as they are taking place — insights into new unstructured and semi-structured data in real time.

The term analytics in general refers to the process of sorting through and analyzing a set of data to collect valuable information. It is the tool used to understand data by defining data patterns that are meaningful to an organization.

Real-time analytics consists of dynamic analysis and reporting based on data as it enters into a system; that is, analytics that can be accessed as the data comes into a system. While the term implies instant access and use of analytical data, it is generally considered to be less than a minute or fraction of a minute — a level of computer responsiveness that a user senses as immediate, or that enables a computer to keep up with some external process such as presenting visualizations of ongoing activity. Real-time analytics allows dynamic analysis and visualization of streaming unstructured and semi-structured data. In addition, technologies can now correlate and analyze data collected from multiple sources in various latencies.


Real-time analytics provides visibility into business activities as they happen. Organizations are able to monitor and analyze business processes, operations and events in real time to see business opportunities, performance issues or organizational threats; and then make more informed decisions and take meaningful actions immediately.

Business activities have become faster and faster often requiring quick decisions and responses. Real-time analytics provides insights from new real-time information and reduces the time from when an event happens somewhere and someone is able to respond or react to it. As the speed of business activities accelerates and processes become more time sensitive, the ability to make decisions and actions on an immediate basis with current information from multiple sources creates a competitive advantage.

Broad Applications

Real-time analytics have many and varied applications across an organization’s activities to provide better visibility for quicker response.

  • Managers can remotely view order information as soon as an order is made, updated, processed or shipped, tracking the orders in real time.
  • Financial institutions can monitor credit card activity while a card is being swiped to detect unusual buying patterns and prevent fraud.
  • E-commerce companies can see continually updated customer behavior on the company website, such as when viewing a page, navigating to different pages, using or abandoning a shopping cart or any other activity; and immediately respond to user behavior with an offer such as an incentive to a hesitant customer to accelerate the sale.
  • Organizations can monitor social media conversations, sentiments or patterns relating to the company or its brand to quickly react to trends, or to direct an appropriate response to positive or negative commentary.
As the economy becomes more thoroughly digitized, analytics is the tool to understand that data and in turn, to best put that data to productive use.

Real-time analytics combines and analyzes data as events happen so organizations can visualize events immediately, make fast decisions based on current information and take the right actions at the right times.