Enhancing the Human Analytical Process through Complex Event Processing
At Cogility, we’ve been working with complex event processing (CEP) as a foundational part of our data analytics work since our founding. It’s a process that not only expands the analytic capability of human analysts but mirrors the human capacity to see patterns. But what do we mean by CEP?
CEP can be defined, at its most basic level, as event processing that analyzes and combines data from multiple sources to infer events or patterns, which outcomes in turn can suggest more complicated circumstances. Webster’s gives us multiple definitions of “cognitive.”
- Related to or involving conscious intellectual activity (thinking, reasoning, remembering
- Based on empirical factual knowledge
The concept of CEP can be best demonstrated through a simple and real-time example that you could run into every day. For example, what do these observations mean when these events happen within an hour?
- Bells ringing
- Man wears tuxedo
- Woman wears white dress
- Rice throwing
By processing and analyzing these events that have all taken place within an hour, we can make a theory that a higher level event has taken place: a wedding. This example may seem basic, but it is the outcome of a sophisticated analytical process. And CEP is a process that can be applied to the world of data analytics.
We’ve created our standalone Cogynt analytic as a method for human analysts to extend and enhance their own analytic capability. It takes cognitive event processing to the next level of detecting patterns from observed events. Patterns enable more sophisticated conclusions. Using our wedding example, we can see that if the ringing bells, tuxedo, white dress, and rice throwing happen Saturday mornings at a given location, in all likelihood that given location has a high probability of being a church…but if those events happen seven days a week, and the location is in Nevada, then that pattern indicates the location may be a Las Vegas wedding chapel.
Applying this level of pattern analysis to data analytics at an enterprise level means empowering analysts to view patterns but it also requires real-time results for fast confirmation of theories based on the most recent events. Our development of Cogynt has arisen out of the need to solve this challenge.
We’ve built Cogynt to solve this challenge in two ways:
- By building our Cogynt product on the Apache Flink streaming platform and Apache Kafka we are leveraging those powerful platforms to enable instant feedback by performing analysis on events from streaming data from multiple sources in real time.
- The other way we’ve enabled instant feedback is through a user-friendly visual tool for analysts without programming knowledge. Cogynt puts power directly into the hands of data analysts and SMEs without coding or programming knowledge by building a front-end declarative authoring tool for creating models and templates
- Without waiting for scripts or coding from programmers, analysts save time through immediate hands-on model building and validation.
We’ve built our Cogynt product to enable this level of analysis through hierarchical cognitive event processing (HCEP). Computational hierarchy is an arrangement of groups and templates that form a hierarchy, where templates at lower levels publish events to higher level groups and templates, allowing an increase in the signal to noise ratio as templates are populated up the hierarchy.
The data statefulness enabled through the Flink platform means the Cogynt analytic persists information about the state of all pending groups and templates. This quality allows incremental evaluation of CEP patterns and response to new events in the shortest time, while continuing to track potential patterns over a very long period of time.
Imagine this capability applied to complex challenges in industry and security, determining threats and risks. In the weeks to come, we’ll dive into use cases showing how Cogynt’s enhancement of the analytic process can be applied to real-world data challenges. We’re also looking forward to giving our presentation at Flink Forward, April 1–2 2019 in San Francisco.