The context

I recently faced the design and development of a complex software solution. Its goal was the monitoring of several sources of operational data to detect patterns and scenarios of interest (PSI), and the analytical exploration of the involved activity.

The problem at hand

This type of problem falls into the category of operational intelligence solutions. For these cases, event stream processing has proved to be an effective approach.

High level view of the logical decomposition

It makes little sense to provide a detailed functional description of the approach to this problem in a post. Instead, I provide a 10000fts functional view to illustrate some of the major logical components. The next figure illustrates such a view.

Logical components description

Ingestion

A mediation layer produces several sources of operational data which are available for ingestion. The Raw Ingestion module handles the continuous extraction of the information from these sources and to push them into the system.

Enrichment

Enriched data streams are the result of combining the data streams of raw records and the entities information. These enriched records are self-contained. They include event record data and attributes of interest from the involved entities.

Stream processing and detection

Data Stream Metadata holds the information about the structure and schema of all the data streams. This allows extending the scope of the system, by adding new data streams of raw records, without affecting the design.

Alarm spotting

The PSI detection process produces a stream of positives where the instances of matched patterns are published. These are later aggregated, by different criteria, to spot alarms. This aggregation activity provides the core of the discovery and investigation processes.

Data flow sketch

The following image provides a global sketch of the data flow in the system. Please, note this is an oversimplified illustration of the overall data flow. I basically illustrate the nature of data at each stage, the transformation activities and the incidental information used in these activities.

Conclusions

In this post, I introduced as reference a design problem of a software for the monitoring of several sources of operational data to detect patterns and scenarios of interest. I also mentioned some NFRs that had a great impact on some of the architectural decisions.

What’s follow

In the next post, I will present a high-level description of the microservice architecture designed to solve this problem. I will argue about the feasibility of such a design when the context is right enough to pursue the expected goals.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Anesto del Toro Almenares

Anesto del Toro Almenares

Passionate software engineer with +12 years' experience crafting software. Lean-Agile evangelist. Emerging blogger. Father of 2 girls, owners of any spare time.