Real-time Systems Landscape using Neo4j

What are the challenges in understanding Systems Landscape?

In any larger organisation, where there are more than thousands of systems (internal / external), it is imperative to understand & co-relate systems and their integrations with different downstream and upstream systems considering environment (development / production) constraints. In addition, following are the challenges faced by organisations to maintain System integrity:

1. Systems Repository, Configuration Management Database / CMDB or other equivalent systems in any organisation are not real-time or not updated.
2. Interface not available for querying & visualizing the system landscape for Project Managers / Business stakeholders for better planning.

What is the impact of not having a Real-time System Landscape view?

Typically, any larger business programs would have impact on one or more systems and would have been impacted due to the above mentioned challenges. If there are number of parallel ongoing programs, then the impact would be significant. Impact on the projects are typically:

1. Project Planning — To identify impacted systems change due to data model / feed change, it would involve repeated data collection exercise. Redundant effort & budget spent on repeated task.
2. Testing phase — Environment constraints reducing the agility of the program, resulting in delayed delivery of projects.
   Example: SystemA has only 1 dev and 1 production instance, which needs to be tested for 3 large programs, which has impact on SystemA.
   Understanding the system dependencies and environment constraints would result in better project planning. It enables to respond to ever changing business needs quickly.
3. Stakeholder Management — Often, it becomes difficult to align the business stakeholders on the complexity of environment / system constraints. A significant effort is spent by the Project Manager, repeating the details to different stakeholders.

How these challenge can be mitigated using Neo4j?

Neo4j can augment the current system landscape details in a graphical visualization model, that can be queried through intuitive tools. The above mentioned challenges were mitigated using Neo4j. On a high level, the solution implementation for Real-time System Landscape is as shown below.

Real-time Systems Landscape Architecture

Critical features of Neo4j that enable this solution are:

Graph database: Capturing, persisting and searching capability on the complex system and integration details is possible through underlying Property Graph model (Entity [Node, Relationship], Path, Token, Property).

Data Model for Real-time System Landscape

Extensible: Neo4j provides an extension capability through custom procedures or functions development. The data extraction, cleansing and applying ML / NLP for extracting system information from structure / unstructured data can be implemented specific to the organisation needs, with limited development effort.

Cypher: An intuitive query interface provided for querying using SQL like interface. The repeatedly used queries can be saved for future purpose through bookmarks.

Drivers availability: The availability of APIs in different programming languages (like Java, C#, JavaScript, Python) for Neo4j, enables rapid development for integrating with the database. Vizjs was used to visualize the Neo4j data, as highlighted in the section below.

A sample real-time system landscape, providing system integration details with querying capabilities can be readily used by IT project team member and business users. The visualization helps identify environment conflicts through different color coding for resolving the Project planning issues.

Real-time Systems Landscape Neo4j Visualization

Conclusion:

Neo4j provides required tool sets for quickly developing, deploying & visualizing graph database for the organisations with limited development resources.

I had developed above real-time systems landscape dashboard using Neo4j and Visjs for a customer in less than a week. The dashboard provides the change in system landscape as you move the project timeline.

This visualization helps in better planning and avoid environment conflicts and resulted in saving significant effort & dollars spent on planning and testing phases of the project.