A new approach to application development using Cypher/GQL (Graphs)
“A mind that is stretched by a new idea or experience can never shrink back to its old dimensions.” — Oliver Wendell Holmes Sr.
Introduction
This article is a revised and enhanced version of a previous piece I wrote on LinkedIn in 2016. It focuses on my experience at a core-tech startup where I co-invested and worked, developing a groundbreaking, fully managed SaaS platform built entirely on a native Graph Database (NoSQL) we developed. I was fortunate to work with an exceptional and talented group of professionals. Sadly, the startup did not endure. However, our collective efforts resulted in the development of an extraordinary technology that surpassed anything with which I had previously worked. Together with the technology, we conceived new ideas, concepts, and methodologies for using Graph Databases to build Applications. These ideas, concepts, and methodologies have long-lasting value and should be shared as they are highly relevant today.
Graph databases have evolved significantly in the past decade, bringing unprecedented value to Data Science and, most lately, enhancing the accuracy and reliability of GenAI models by adding context via RAG.
Two weeks ago (April 12, 2024), the International Organization for Standardization (ISO) published its GQL (Graph Query Language) standard under ISO/IEC 39075:2024, which is a close reflection of Cypher and its open-source implementation, openCypher, developed by Neo4j. This is the first database query language ISO certified since SQL in 1987.
With this significant milestone, I’m confident that GQL will make Graph Databases predominant, as SQL did with Relational Databases.
Please do not confuse GQL with GraphQL; they are very different.
- GQL is a declarative graph query language designed explicitly for querying graphs databases.
- GraphQL is an API query language that allows front-end developers to request precise data from the back-end APIs.
In this article, I will discuss some of the ideas, concepts, and methodologies we conceived years ago and how they can be implemented using Cypher in a manner that they should apply to any Graph database implementing GQL/Cypher/openCypher. I will use Neo4j+Cypher to illustrate these concepts. I will strive to convey the concepts and ideas in simple terms and use illustrations to make them accessible to a broader audience. Ultimately, this article introduces a new, simplified, intuitive software development approach for building applications using Cypher/GQL (Graphs Databases). Although I will be using a step-by-step “how-to” approach and sharing some code, the primary intent is to point a direction.
Lastly, I want to dedicate this article to those who spent years thinking and drafting on a whiteboard, designing, and coding to transform an idea into a mind-blowing platform, without which these ideas and concepts I’m about to share would not exist.
The status quo
Understanding the evolution of established concepts in application development can provide context and a foundation for understanding the new concepts being introduced. By revisiting these established concepts, we can gain insights into the challenges they addressed and how they have been improved or transformed over time.
For a long time, application development was equated with a single approach.
In simple terms:
- We represent and maintain “things/objects/entities” along with their attributes/properties using Tables, Rows, Columns, or other data structures and databases.
I will use the term “objects” to describe “things/objects/entities” and the term “properties” to describe attributes/properties throughout this Article.
- We use programming languages to iterate through arithmetic loops and logic gates (OR, AND, XOR, NOR, NAND, XNOR, and NOT), enabling us to read and write data from databases. Likewise, we incorporate logic and control flow, manipulate these “objects,” and create user interfaces.
This is rooted in the fact that the heart of a modern computer, the microprocessor (also known as the CPU — central processing unit), conducts basic arithmetic operations via its arithmetic-logic unit (ALU). Indeed, from the earliest mechanical computers of the 19th century to today’s electromechanical, analog, and digital computers, the primary function envisioned by their creators was calculation, making arithmetic the ideal (and perhaps the only) mathematical model to employ.
So yes, we’ve been using an arithmetic “Calculator” all this time. 😉
This approach has worked unquestionably well, from the first system built to the latest and most sophisticated systems we use daily. It powered the first scientific systems, then the first “Data Processing” systems, such as accounts receivable, billing, and general ledger. It landed a man on the moon and now enables all our daily applications.
Databases, Programming languages, and Methodologies have evolved significantly, but the underlying approach to developing the applications remains the same.
Adding complexity
One of the aspects that dramatically changed along the evolution was the number of user interactions with the systems.
The first systems most likely had only one user, with minimal User Interface (UI) and a few interactions; they were monolithic and fully stateful (the system maintains information about the state or status of the interactions during a session, and they will be available in future sessions). One would just input parameters (if any), hit a key, and wait for the results.
As systems evolved and started to support business, they became multi-users with a UI that a trained “user” could manipulate. The system would still be monolithic and stateful but now run on a terminal and support many user interactions simultaneously. These systems had to deal with data concurrency and consistency, adding a level of complexity.
Subsequent technological advancements, such as Client-Server, Internet/Intranet, and Mobile Apps, meant that everyone became a “user” constantly interacting with systems through sophisticated UIs. We connect to these systems via various means of connectivity, both reliable and unreliable, in a fully stateless manner, where the system does not retain information about the state or status of interactions during a session, and they are not available in subsequent sessions, further complicating the landscape.
The evolution of information systems from the early monolithic, single-user, few interactions, and stateful programs to today’s complex architectures has introduced numerous capabilities, programming frameworks, system architectures, advanced user interfaces (UIs), and communication protocols. This transition, from the single-user, stateful interactions of the first systems to the current stateless systems accommodating thousands of concurrent users, has fundamentally transformed the software development landscape. Consequently, software developers now require a prominent level of specialization, as managing these intricate systems has become a distinct discipline within the realm of Computer Science.
Despite all the technological advancements and efforts to make these technologies more straightforward and reduce the cost, time, and effort of building new software solutions, it is still complex and costly to transform an idea that addresses a problem from another domain (non-computer science) into an effective computer-science solution.
The problem
There remains to be a significant gap between the problem and solution domains. Both parties need a unified end-to-end framework and understanding that can be utilized to address the issue and devise a solution collaboratively.
It’s not surprising that 66% of technology projects are unsuccessful. Moreover, numerous potential solutions remain indefinitely on hold in backlogs due to resource constraints. The production of software developers is not keeping pace with demand. The International Data Corporation (IDC) predicts a worldwide deficit of four million developers by 2025. It’s reported that the global talent shortage may lead to around $8.5 trillion in unrealized annual revenues by 2030. (source)
A new approach
To remove the complexity of the existing application development approaches, we must make them more intuitive and natural by establishing a common language and understanding of what we are trying to solve and how we will solve it. Instead of adding complexity by creating layers of abstraction and different representations of reality, we will imitate how nature works and use similar mechanisms. After all, nature works perfectly well. As our purpose is to build information systems, we will start by changing how we store information. I’m sorry to tell you that nature does not store information in the form of tables, rows, and records. We will use a database that stores information like nature does (e.g., your DNA). We will take a step-by-step approach and apply some interesting concepts to do so.
In essence,
1. We will adopt a Domain-Driven Design to design the Domain Model and elaborate on how our solution will address it. By doing so, we will stay at a level that allows stakeholders with different domain knowledge to contribute actively.
2. To do so, we will use elementary Set Theory concepts/mindset and represent the Domain Model in a Graph. (This is where the magic happens!)
3. We will add a contextual layer to the Model using Semantics to describe the Sets and the Graph relationships, making it intelligible to any Domain Expert.
4. At this stage, we will have a Graph representation of the Domain Model that is 100% loyal to its real-life counterpart. Nevertheless, it might still not reflect how it will “behave” once its users (Actors) start interacting with it.
5. We will use a combination of Interaction Design and Event-driven architecture to map the different types of users and their touchpoints. We will design asynchronous and atomic User interface (UI) blocks for each user touchpoint. These UIs will expose projections of the Graph and interact with the Model, creating “objects” and altering their state. This exercise will allow us to refine the Model even further.
6. We will map the supporting Microservices and Application Programming Interface (APIs) needed to fulfill the Model.
Although the above steps might give the impression that this new approach is complex, given all the principles involved, it is pretty simple and intuitive once you start working with it.
The links I added are just for reference for those who are unfamiliar and have a curious mind (like mine). I intentionally used Wikipedia instead of complex academic references to make it more digestible; after all, this should be a practical guide, not a PhD Thesis. You don’t need to go through all that.
Although each of these principles deserves a chapter in a book, as this is just an Article, I will stay at a high level and use simple language and illustrations to keep this as simple as possible.
We will end up with:
· A Graph Data Model (the Information Architecture) — Graph scripts (Cypher) to generate the Nodes and Relationships with their respective properties and semantics at their initial state. Map all the graph structures representing business objects and their respective scripts (Cypher).
· An Interaction Model (the Interaction Architecture) — UI wireframes and their respective Graph transaction scripts (Cypher) that will interact with the Model, retrieving information, exposing projections of the Model, creating “objects,” and altering their state.
· A bill of materials (BOM) comprised of UIs that must be developed for each user type in accordance with the Interaction Model. A list of supporting microservices that will need to be developed using the traditional method and the potential third-party APIs that we can use.
As you will see later, the resulting graph data model is straightforward to understand and faithfully represents the domain model.
This will allow us to symmetrically transpose the Graph Data Model to a Graph Database (aka NoSQL) and use it as the problem domain experts designed it. This is unlike a Relational Database Management System (aka RDBMS, soon-to-be NoGQL 😉), where a data analyst needs to create an Entity Relationship (ER) model and then normalize, leaving the resulting RDBMS in a shape that the problem domain experts can no longer relate to without ER training.
I will now explain some of these above concepts, and as we progress, we will start to blend them in.
The Domain-driven design approach
Domain-driven design (DDD) is a software design approach that relies on input from domain experts to align the software with the specific domain. The results are systems closely reflecting the business domain, making it easier for both the Problem (Business) and Solution (Developers) Domain experts to understand and maintain.
The domain-driven design promotes using a common language between domain experts, users, and developers. This ensures that everyone involved in the project understands the domain concepts consistently from conception to implementation. It implies a strong collaboration between the Problem (Business) and Solution (Developers) Domain experts.
Using SET
We will use basic Set concepts to structure and describe the Domain Model.
In the vast landscape of existence, from the microcosm of atoms to the macrocosm of the universe and everything in between, organization emerges through the lens of set theory. At its core, set theory defines relationships and structures among elements, creating order from chaos. So, instead of defining the world through the lenses of Arithmetic, which we are familiar with in computer science, we will use Set.
In our everyday lives, set theory permeates the organization of households, businesses, and societies. Your business can be viewed as a Set of departments {Marketing, Finance, Operations, Human Resources, IT, etc.} with their respective sub-departments. Similarly, your house is a set of accommodations {bedrooms, bathrooms, kitchen, living room, dining room, etc.}. Within each accommodation, you have sub-sets of furniture and appliances such as {wardrobe, cabinets, refrigerator, etc.} that can have their subsets such as {drawers, shelves, etc.}. If you did not have these Sets in your home, it would be chaos, with things spread everywhere (like my son’s room).
Society itself can be analyzed through the lens of set theory. Individuals are grouped into categories based on factors such as age, gender, ethnicity, and socioeconomic status. These sets intersect and overlap, shaping social dynamics, hierarchies, and cultural norms.
Through the lenses of set theory, the complexity of our world becomes more manageable, allowing us to analyze, understand, and navigate the intricate web of relationships and structures that define existence.
Your Domain will not escape these rules, and we will analyze them through the lens of a Set. I like to refer to this as a “Set mindset” or just “Mind-Set” as I call it.
The beauty of Sets and how we use them in our daily lives by adding semantics is so innate that if you visit me at my house, where you have never been before, I ask you to bring me a “spoon.” You will get one without any help — just by interacting with the environment and traversing through my house (no arithmetic needed).
As you will see later, we will map the Domain Model using this “Mind-Set” to normalize some “objects,” structure “object” hierarchies, and group them in Collections.
What is a graph database?
A graph database is a NoSQL database designed to store and retrieve information in ways other than the tabular relations used in relational databases. It stores data as a graph consisting of nodes, edges, and properties.
In a graph database:
· Nodes: Nodes represent objects, such as people, places, or things. Each node typically has one or more properties, which are key-value pairs containing information about the entity it represents. Therefore, a Node contains a set of properties that describe the identity of the “object”. As Nodes can represent different “objects,” we associate a Label to Nodes that represent similar “objects”. The Labels end up constituting a Set by itself.
· Edges: Edges represent the relationships between nodes. They define how nodes are connected and can have properties that describe the relationship’s nature or characteristics/weight. More importantly, the name (Type) and direction of these connections provide a fundamental semantic to the relationship. The edges (connections/relationships) between nodes, with their names and directions, are as relevant (or more) as the properties that define each node.
· Properties: Properties are associated with nodes and edges and provide information about them. They can be simple values (e.g., strings, numbers) or complex data structures.
Now that we have some basic concepts of Sets and Graphs, we will combine them to build the Data structures.
Representing sets using graphs provides a robust visual and mathematical framework for understanding relationships and structures. Interestingly, sets can be directly represented as graphs, where each element corresponds to a Node, and the connections between Nodes represent relationships between elements. Moreover, properties of sets such as subsets, unions, and intersections can be represented and understood graphically through operations on graphs.
In summary, representing sets using graphs simplifies the visualization, analysis, and manipulation of sets, offering a versatile framework for understanding their properties and relationships by any domain expert.
Describing Objects in a Graph with a Mind-Set
Like any modeling exercise, we will analyze each Type of “object” that comprises the Domain Model and the properties that describe their unique identity. These “objects” should contain the information needed to perform their role on the Model. However, as we use nodes on a graph instead of records on a table, we will leverage the relationships between nodes and their semantics to enhance each object’s description.
Let’s use the classic Product description. Suppose we use the standard Table/Record approach to describe a Product. In that case, we will use a Set of properties such as {Product Category, SKU, Product Name, Retail price, Cost price, Weight, Size, Product images, and Serial Number} to describe the product.
One of the current limitations of this new approach to developing applications using Graphs is that most of the Graph Databases in the market are built and optimized for Knowledge Graphs. Due to this, the standard Datatypes that most RDBMSs support are not supported in Graph Databases. I’m hopeful that the Graph Database vendors will add support for other Datatypes at some point, or they will miss a great business opportunity to explore workloads other than Knowledge Graphs.
After a proper normalization exercise to reduce redundancy, we will end up with three entities representing the “object.” We will need to create some Primary Keys (PKs) and Foreign Keys (FKs) to guarantee the uniqueness and integrity of each “object,” and, therefore, we will need to add additional Key properties in each record. By doing so, we are adding back some redundancy at an attribute level.
On a Graph, the normalization of the Product will follow similar patterns, but each Node will have only the properties relevant to its role in the Model (not the System). We need not add Key Properties from other Nodes to add integrity. Therefore, no redundancy will be added. The relationships (Edges) that connect the Nodes will maintain the model’s integrity. These relationships, their name, and directions will add semantics to the Model. In some graph implementations, the relationships can also have Properties that add additional context or weight to these relationships. This result is a more intelligible model due to the semantics of the relationships. The resulting Graph representation is more self-explanatory for a lay reader than the traditional ER representation. You literally “Read the Graph” following the relationship directions.
Adopting a proper naming convention for “objects” (Nodes) and relationships (Edges) is a good practice.
I will use “PascalCase” (no spaces and starting each word with a Capital letter) for “objects” (Nodes), and I will use “SCREAMING_SNAKE_CASE” (All Capital letters with an underscore character as word separation) for Relationships (Edges). You can read this good article about this.
For “objects” (Nodes) Labels, we generally use Nouns as they represent Nouns/Proper Nouns, and for Relationships (Edges) Types, we use Verbs or Prepositions. You can also read this good article about relationship naming.
As we add more “objects” to the Model and increase its complexity, the relationships and their semantics will maintain the Model in a very readable shape. This allows the Problem Domain experts to participate actively in the Solution Design process.
The resulting structure in a Graph, once we start adding data to it, will be very different from what we are familiar with Tables and Records. Unlike Index/Table scans, where the Database systems perform a sequential or binary-tree scan (arithmetic sequence/loop) of a Table/Index to locate the record we are targeting, on Graphs Databases, we will use a method called Traversal that relies on an index-free adjacency (when using a native Graph). This means that every Node in the Graph has its adjacent Node’s information (address) for each relationship it contains. No index scan is needed. All you need is a starting node (called an “Anchored node”), and you can quickly traverse the graph following the “Semantic Relationships” that were added to describe the model.
Another important point when adding properties to an “object” is to avoid adding properties that define states if the state has relevance to the Domain Model. In this case, the state is not a property of an “object.” For instance, in our Product example, a Product {SerialNumber: xyz} can be in the “Warehouse,” in the “Store,” or “Sold.” These are not product properties; they are states, and their relationship with the warehouse, store, or customer objects will define which state the product is in.
As you will see later, a state is defined by a relationship between two objects at a given time due to some user or system interaction with the model.
Key Rules:
1. Keep the properties that define each object to the essential ones that truly define the “object” uniqueness (proper noun).
2. Use names (Labels/Types) for objects and relationships that add contextual meanings (Semantic) to the Model. Stick to a naming convention.
3. Remember that properties that define the state of an “object” should not be a property of an object. You want to avoid updating the properties of “object” to reflect the state they are in.
Object Structures
Things start to get interesting as we dissect each “object” of the model and realize that for them to perform their role in the model, they need other supporting “objects” that are connected and grouped in a particular way, creating a unique structure/anatomy. Like molecules and the elements, they are formed. Some “objects” play the role of Elements, and some will play the role of Molecules. These connections between the “objects” within their structures and with other “objects” of the model are crucial and will add life to the Model as we apply semantics.
In an ideal scenario, the “objects” do not change the values of their properties after they are created. They remain as they are, and the only thing that changes is their relationships. Although this seems odd and gives the impression that little can be achieved in such a manner, you must look at how nature can work this out. For instance, Hydrogen (H), Oxygen (O), and Carbon (C) are essential elements in organic chemistry and can combine in various ways to form numerous molecules. The elements (and their properties) do not change. The only thing that changes is the quantity of each element and their relationships (chemical bonds)
Likewise, life as we know it relies on six essential elements (C, H, O, N, P, S). These elements are crucial for building the molecules necessary for biological processes. Although these elements are abundant in the universe, the different chemical bonds (relationships) between them make life possible. It’s all in the relationships!
Like molecules (Fig. 7 - below), “objects” often need structures comprised of other “objects” and/or sub-structures (collections) that group together other “objects” of a similar type. Therefore, every instance of a particular “object structure” in the Model must have the same structure to perform its role.
For instance, in a Product Retail Domain Model, a customer “object” will need other supporting “objects” or “sub-objects” to perform its role in the Model. Besides the essential properties that define the customer, it must have at least one billing and shipping address. One or more payment methods, one shopping cart or wish list, pending orders, past orders, product reviews, etc. Each of these “objects” will have its properties and need to be connected by specific relationship Types to the customer “object” and between them. This set of “objects” with their properties and the relationships between them will define the customer’s anatomy in the Model.
So, every time we create a new instance of a customer “object,” we must create the same minimum structure for it to perform its role in the Model.
(This is a simplified example of a customer resembling the molecule above. 😉)
As you will see in this customer example Graph, some particular Nodes (Wallet, ShoppingCart, AddressBook) do not represent an “object” with their respective properties but only have a Label and serve to aggregate “objects” of similar Types. This is not a normalization anomaly, as it might seem to some Data Scientists. It is intended. These are collections, and we will discuss them in the next section.
Key Rules:
1) When we create an “object” of a particular Type/Label, we must understand and create the entire structure that supports its role in the Model.
Collections
Collections are an essential type of “object” as they organize the Model in the same way we understand the Domain (and the world around us). They are the most straightforward implementation of Set in a Graph and play an essential role in the Model’s construction. In the graph, they are nodes, like any other node, representing an “object.” The difference is in how we use them.
As their name suggests, they aim to aggregate “objects” of the same (or similar) Type. Just like a drawer in your closet or kitchen cabinet, they “generally” hold objects that have similar uses. Functionally, they are like a Table in a traditional database. Another analogy would be the Ganglions in your nervous system, which are collections of neuron cell bodies that act as synaptic relay stations between neurons.
Collections add order to the chaos.
A good day-to-day example would be a Keychain. Keychains are collections of keys, and you probably have more than one keychain, as you don’t want to carry all your keys in one big keychain. Despite holding similar objects (a collection of keys), you want to separate them by some logic that makes sense to your mental Model.
Collections can aggregate “objects” at a Domain scope (like Global Variables in a program) or at an Object scope where they are part of an “Object Structure,” as I demonstrated before.
Establishing a proper naming convention for “collections” and the relationships (Edges) with the “objects” they “Contain” is a good practice.
A Collection is an “object” (Node) like any other “object” in the Model/Graph, with the difference that instead of representing a single “object” with its respective properties, it will represent a Set of “objects.” As such, I will use a modified “PascalCase” (no spaces and starting each word with a Capital letter) by terminating the name with a Capital letter to differentiate an “object” from a “collection.” (i.e., PascalCasE)
Regarding the Relationships (Edges) of the “collection” with the “objects” it contains, I will use “SCREAMING_SNAKE_CASE” (All Capital letters with an underscore character as word separation).
To maintain consistency, we will pick one relationship Type (name), and use it consistently on the entire Model for all objects that are elements of any given Collection, denoting that the Collection “contains” the object that the directed relationship points to.
In this article, I will always use a single verb, “HAS” alone (without any underscore), to connect all the “objects” that are elements of any given Collection. Therefore, we must not use “HAS” alone in any other relationship in the Model for any other relationship that is not a Collection item.
In this way, when a Node’s Label starts and finishes with a capital letter, you will know that it is a Collection, and any “object” it connects to with the relationship “HAS” is an element of that particular Collection.
The cardinality of a Collection is the number of “objects” connected with a “HAS” relationship to it.
Therefore, if we apply this naming convention to the Object Structure we used previously, it should look like this:
Just as a curiosity, this is the Cypher/GQL command used to create the entire Customer structure, including an empty Shopping Cart, an Address Book with two addresses, and a Wallet with two payment methods:
CREATE (c:Customer {FirstName: "John", LastName: "Connor" , DOB: "02051985", ID: 83773729322 ,Phone: "(303)2345683", Email: "JohnC@email.com" , Photo: "https://photos.com/img/JohnC.jpg"})-[:HAS_CART]->(:ShoppingCarT {Name: "ShoppingCarT"}),
(c)-[:HAS_WALLET]->(w:HAS_WALLET {Name: "WalleT"})-[:HAS]->(:PaymentMethod {Type: "CreditCard", Issuer: "Amex", NameOnCard: "John Connor", CardNumber: 3742454554001263, ExpDate: "05/26", CVV: 9065}),
(w)-[:HAS]->(:PaymentMethod {Type: "DebitCard", Issuer: "Visa", NameOnCard: "John Connor", CardNumber: 4701322211111234, ExpDate: "12/26", CVV: 837}),
(c)-[:HAS_ADDRESS]->(ab:HAS_ADDRESSBooK {Name:"AddressBooK"})-[:HAS]->(:BillingAddress {Name: "Home" , StreetNum: 950, StreetName: "S Elizabeth St", Complement: "Suite #5", City: "Denver", State: "CO", ZIP: 80209}),
(ab)-[:HAS]->(:ShippingAddress {Name: "Work", StreetNum: 600, StreetName: "17th St", Complement: "Suite #5899", City: "Denver", State: "CO", ZIP: 80202});It creates 8 nodes and 7 relationships, sets 36 properties and 8 labels, and takes around 60ms to execute in Neo4j Aura.
Object Structure Collections
One of the main goals of having Collections associated with an “object” is to optimize the number of relationships a particular “object” has. Although the relationships between “objects” are as meaningful as the properties that define them, having too many properties and relationships on one “object” can add some overhead to its role in the Model. Like a span of control in your organizational structure.
To access the Collection of an “object,” we will always use the “object” as the anchor Node and traverse to the Collection via a known connection; then, we will always use the connection “HAS” to fetch the “objects” of that Collection.
Collections are nodes like any other node in the graph; therefore, they can also have properties that can be used to add an extra level of semantics. I don’t recommend giving them unique IDs for them to be accessed directly, bypassing their parent (anchor) “object,” or you can create some inconsistencies in the Model.
Let’s take the Product Retail example we used earlier. Each Store object (as part of its structure) might have a series of Collections connected to it (e.g., Employees, Inventory, SoldProductS, ReturnedProductS, etc.), as you would not like all the Products (in the Inventory, Sold or Returned), Employees, etc. connected to the Store “object” directly.
The SoldProductS collection, following our naming convention, would look like this:
As we saw before, each Customer object will also have a series of Collections connected to it (e.g., ShoppingCarT, WalleT, AddressBooK, PurchasedProductS, etc.). Suppose a particular Customer purchases a Product from a Store. In that case, the same Product “object” (not a copy of it) will be connected to the Store’s SoldProductS collection with a “HAS” relationship and connected to the Customer’s PurchasePorductS collection with a “HAS” relationship.
Likewise, if that Product has an Active Warranty and that information is stored in the Model, that Product “object” will be connected to the Warranty collection with a “HAS” relationship.
The memory footprint of a Node in a Graph
I’m sorry to get into the nitty-gritty here, but it is vital to bring this up so you can consider it further.
Just like the atomic mass of the elements in a periodic table, comprised of the combined mass of protons and neutrons, the Node’s total memory footprint is the combined memory footprint used to hold the Node Label, the Properties that define the “object” it represents and the information about its Relationships to efficiently traverse without an Index (index-free adjacency) on a native Graph.
Depending on the Graph implementation and optimization (Graph vendor), the Node, besides its label and properties and the address of the Nodes it connects to, might also hold the information of the Nodes that connect to it. Again, depending on the implementation, this information can be stored within the memory footprint of each Node. Therefore, if a Node has many relationships (From it and To it), its memory footprint will grow proportionally to the number of these relationships.
There are positive and negative sides to this:
For instance, imagine if your “Address Book” could contain not only all the information of the people you know but also all the information of those who know you. Yes, that would be a massive violation of privacy if you don’t have their consent, but presuming you have it, how valuable would that information be? (that is the positive side)
- This is extremely valuable for Knowledge Graphs, which leverage these relationships to extract knowledge and provide incredible insights into the data.
Now think about the size of your Address Book (memory footprint); if you are a “Celebrity,” everyone knows. And how much effort (compute cycles) it would take you to find the phone of someone you know but forgot the name of and only know that it starts with the letter “S” (this is the negative side)
- This could be an issue for Application/Transactional Graphs. As the Nodes will likely be stored in cache memory while the Application interacts with them, the smaller the memory footprint of a Node, the more Nodes can be within the cache memory, and the more efficient the system will be. You want to keep your Nodes as lean as possible, containing only the information needed to perform their function on the application.
As you saw in the examples above (Fig. 13), the direction of the relationships is crucial. The relationship of the “objects” Store and Customer to their respective Collections and from the Collections to the Products they contain are directed relationship (one way). The Product does not need the information about what Collection it belongs to. Likewise, the Collections do not need information about the “object” they belong to. This information is irrelevant to the Products and the Collections.
In this way, depending on the Graph implementation (Graph vendor) and how optimized it is for this type of usage (Application Graph), if a Product belongs to an infinite number of Collections via a directed relationship (Collection –[:HAS]-> Product), and the Product Node does not store the information of the Nodes connected to it, its memory footprint should remain the same.
As you can see, Collections play an essential role in keeping the memory footprint of Nodes reasonable in size for both the ones that connect to them and the ones that they connect to (at the cost of an extra traversal/hop). For instance, if you look at our object structure (Fig. 8 and 10) from a Graph standpoint, we could perfectly connect the Customer to all the Nodes in its Collections using different relationship names, but this would increase the memory footprint of the Customer node.
Domain Collections
Unlike the “Object Structure Collections” we just saw, where every “object” of a particular type will have one or more similar Collections as part of their structure, Domain Collections are unique in the entire Domain Model. They can represent a class of “objects” and can help organize the Model, or they can group “objects” that have common characteristics or are in a particular state within the Model.
As Domain Collections are unique in the Model, they can be used as anchors and accessed as starting points for Traversals. Therefore, they must have at least one property that characterizes their uniqueness. They are generally Supersets or Subsets in the Model.
In the above example (Fig. 14), if you want to access the Premium Customers, you can use the PremiumCustomerS collection as an anchor and traverse the Customers connected with a “HAS” relationship.
State and Sets
As mentioned earlier, the connections/relationships between “objects” (Nodes), with their names and directions, are as relevant (or more relevant) as the properties that define each “object.”
One of the characteristics that a relationship between two objects specifies is the State of a particular “object”.
We often use the properties of an “object” to denote its states when, in fact, the state an “object” is in is not part of the “object’s” identity. It is a condition or situation the “object” is in at a particular time as measured by the “observer” (Einstein’s special relativity). In our case, the “observer” is the Model. Within a Model, if a state is a pertinent transient condition impelled by it, it should be represented by a relationship within the Model and not by a property of an “object”. If the state is non-transient or non-pertinent to the Model but just informational, then it could be a property of an “object”.
In the illustration above, if both John and Mary are pertinent “objects” and the marital state is also relevant for the Model, a relationship between them will define their marital state. In this way, we don’t need to include and update an object’s properties to symbolize its state. This is quite convenient for concurrency control.
Using collections to represent a state
We can use Domain Collections to denote a state by connecting and disconnecting “objects” to them. If an “object” is or is not an element of a collection, a state is implied.
If we apply this principle to our Product example and have two collections illustrating the Products that are in stock (InventorY) and the ones we have sold (SoldProductS), then as soon as a Product is sold to a customer, a “HAS” relationship from InvertorY will be deleted. A new “HAS” relationship will be created from SoldProcutS.
Building the Graph Data Model (aka Information Architecture)
We will now use the concepts we described earlier to build the Graph Data Model.
First, we will spend valuable time with the domain experts, interaction designers, and a graph (Cypher/GQL) expert in front of a whiteboard drafting the Model. We will map all the “objects” with their respective properties. We will identify the “object structures” with their respective collections and see if the structure fulfills their role in the Model. You will likely get it wrong on the first path. No worries. As you discuss and refine the Model, things will take shape. Identify the domain collections that will contain the initial “objects” and if you can, try to identify the domain collections that represent states that the “objects” can be.
With the help of the Cypher/GQL expert, we can start building the first Graph draft of the Model so that everyone can familiarize themselves with it. By the end of the first session, you should be able to have a first (incomplete) draft of the Model in the Graph database.
Things will start to sink in as you visualize them in a Graph explorer (i.e., Neo4j Browser).
When you see the first “object” structure as a Graph and start clicking on each Collection and its objects, things will start to make sense.
You will then realize how straightforward one Graph (Cypher/GQL) command can create a whole object structure in a few milliseconds.
// CREATE a Customer with ShoppingCarT (empty), WalleT (2 cards) and AddressBooK (2 addresses) collections
CREATE (c:Customer {FirstName: "John", LastName: "Connor" , DOB: "02051985", ID: 83773729322 ,Phone: "(303)2345683", Email: "JohnC@email.com" , Photo: "https://photos.com/img/JohnC.jpg"})-[:HAS_CART]->(:ShoppingCarT {Name: "ShoppingCarT"}),
(c)-[:HAS_WALLET]->(w:HAS_WALLET {Name: "WalleT"})-[:HAS]->(:PaymentMethod {Type: "CreditCard", Issuer: "MasterCard", NameOnCard: "John Connor", CardNumber: 2222420000001113, ExpDate: "08/26", CVV: 321}),
(w)-[:HAS]->(:PaymentMethod {Type: "DebitCard", Issuer: "Visa", NameOnCard: "John Connor", CardNumber: 4701322211111234, ExpDate: "12/26", CVV: 837}),
(c)-[:HAS_ADDRESS]->(ab:HAS_ADDRESSBooK {Name:"AddressBooK"})-[:HAS]->(:BillingAddress {Name: "Home" , StreetNum: 950, StreetName: "S Elizabeth St", Complement: "Suite #5", City: "Denver", State: "CO", ZIP: 80209}),
(ab)-[:HAS]->(:ShippingAddress {Name: "Work", StreetNum: 600, StreetName: "17th St", Complement: "Suite #5899", City: "Denver", State: "CO", ZIP: 80202});Depending on the size and complexity of your Model, after a few sessions, you will have a decent representation of your Model on the Graph in a way that everyone in the room will naturally understand and “read”, regardless of their domain experience.
Feel free to bring in other stakeholders to provide feedback.
You will already have an eye candy to show off. 😉
This approach for building applications using Graphs prompts you to build a Model that reflects the world as it is. When in doubt, while building the Model, a good practice is to “zoom out and look around” to find similar patterns in nature (from the macrocosm to quantum mechanics and everything in between), see how these patterns behave in different scenarios, then zoom in and apply these patterns you found to the Model.
I intentionally used different analogies and theories from other domains to reflect this as I wrote this Article.
And by doing so, I’m building relationships/connections in your brain’s graph. 😉
Interaction Model (aka Interaction Architecture)
Once we’ve established a clear understanding of the data model (Information Architecture), our next step is to delve into analyzing the interaction model. This involves examining how the relationships between various “objects” will dynamically change as different users interact with them in various states.
Similar to any methodology, we begin by identifying all the users engaging with the model. This helps us determine which objects within the model each user’s role will interact with based on their unique user journeys and the sequence of events these interactions will generate.
At this stage, we transition into wireframing the user interfaces (UIs), crafting user experiences, mapping the events, and identifying the “object” (anchor nodes) and properties that each interaction/event will need to fetch information or to make changes to the Graph.
Ideally, each UI should be atomic, stateless and operate as a function, receiving parameters from the calling UI and passing parameters to the following UI they call. These parameters will be used to parse the Cypher/GQL commands the UI executes against the Graph.
A word on Identity & Access Management
In most cases, the User of the Application will likely have a corresponding “object” in the Graph, and this “object” will likely be the anchor Node for most of his/her Cypher/GQL queries, so consider linking the user “object” to the Identity & Access Management system you will use to authenticate and provide access to the Application.
By the way, graphs are highly efficient in building access management solutions.
We proceed by sketching out the UIs that users will encounter, outlining the user events they’ll initiate, and specifying the corresponding Graph commands (such as Cypher or GQL queries/commands) they’ll execute within the Graph environment.
While designing the interaction model, it’s common to iterate and refine the “objects” and their structures from the data modeling phase.
To build the UI components, you will pick a frontend framework that best suits your application needs. Given that there is documentation and hooks available to use React with Neo4j, I suggest using React to build your first WebApp using this Graph development approach. This will allow you to build user interactions and perform Cypher/GQL graph commands from your frontend stack.
I’m confident that as Cypher/GQL starts to be widely adopted, low-code UI platforms like Retool, Bubble, Appsmith, JetAdmin, and others will add support for Cypher/GQL. So here is my request to all these low-code UI platform companies to consider Cypher/GQL in your integration roadmaps (GraphQL is not enough).
With Graph Data Modelling and a good low-code UI platform using Cypher/GQL, you can quickly transform your ideas into solutions with minimal code in no time, providing unprecedented productivity.
We will then examine how each User interaction affects the Model’s state.
You will be using the Graph as a state machine.
If your model is well-designed, you will notice that after the “objects” are created, you rarely update their properties and that most of your transactions consist of creating (connecting) or changing (disconnecting + connecting) the semantic relationships between them.
Coincidentally, this is how your brain works, wired like this. Around 86 billion neurons, wired similarly to a graph, creating connections between them based on your understanding of the world and trying hard to make sense of it all. And surprisingly, they work perfectly well (ok, almost perfectly 😉)
For this model to work correctly, it is crucial that a “Disconnect object” followed by a “Connect object” occurs as an ACID (Atomic, Consistent, Isolated, Durable) transaction. Otherwise, the integrity of the Graph will be compromised. Therefore, using an ACID-compliant Graph database is a must.
You will basically use these Cypher/GQL Graph operations to run your entire application:
· Read objects (MATCH, RETURN),
· Create objects (CREATE or MERGE),
· Update object (SET),
· Disconnect objects (DELETE),
· Connect objects (CREATE)
No, I did not forget the “Delete object”.
There are reasons for this, given that the principles behind this approach are building a Model that is 100% loyal to its real-life counterpart and represents the world as it is. As such, we cannot unsee what we saw, unsay what we said, unhear what we heard, and consequently, we should not uncreate what we created.
As we create objects, other objects can then establish relationships with them, and they can become an intrinsic part of another microsystem or macro system. Deleting objects can potentially have huge implications and create inconsistencies within the Model or other Models that share objects.
Even though Cypher/GQL has a DETACH DELETE clause to delete nodes and any relationships connected to them, only do so if you are 100% sure there will be no implications.
Your application will consist of isolated UIs that trigger events upon load, user input, or exit. They will likely execute commands against the Graph for each event to perform the above Graph operations.
You will notice that little code is left to write after defining all UIs and their respective Graph operations, which will perform all the alterations to the Model. The majority of the application logic will reside in the graph.
However, Graph operations and low-code UI frameworks are not enough, as you will need some procedures and perform some calculations. For this, you must identify what microservices or APIs your application will need to call. These Microservices will be developed using the traditional approach you have used and consumed by the UI. They can query the graph together with any other data source with no restrictions. The number of supporting microservices will depend on the Domain and the application complexity you are building.
Give or take, and besides the UI components, you will likely need to code ~20% of your application, as Graph commands (connect/disconnect) will handle most of the application logic.
Depending on the Application and its Domain, you might need to code more, and in some cases, you will conclude that this Graph Approach is unsuitable for what you are trying to build. For instance:
· Ledger applications that require constant/sequential scans and updates of “object” properties (Graphs are unsuitable; use tables instead).
· Applications with low levels/frequency of user interactions (batch processing) will not benefit much from this approach.
· Automation systems are unlikely to benefit from this approach, but exceptions could exist.
On the other hand, applications that require constant user interactions, where the user is presented with information and needs to assess and act, such as booking systems, commerce, workflows, procurement, tender management, Line-of-business (LOB) applications, etc., which constitute the great majority of Applications we use on our day-to-day. These are all perfect candidates for this approach.
This concludes the conceptual portion of the Article. In the following section, we will build a model and a mockup application that applies the concepts we just learned to a real scenario so that you can see how it all comes together. We will design the Model and the respective Cypher/GQL code to build and manipulate the Graph.
If you are not interested in seeing some code, you can skip to the end of the article, where I make some final considerations.
Practical example/demo (show & tell time)
Now that we have covered some fundamental concepts of this new approach to developing applications using Graphs. Let’s see how they work.
To illustrate this approach, I’ll pick a scenario that should be familiar to many readers of this article:
I will use a very simplified ride-hailing booking system, which will help illustrate many of the concepts we covered before. I intend to build a WebApp mockup, and I will focus more on the database side, exploring Graph implementations rather than UI/UX.
I will highlight as BOM (bill of material) the components of our solution that must be developed using a traditional method.
The Graph Data Model/Information Architecture
We start by designing the Domain Model and listing the different “objects” that will make up our simplified Ride-Hailing Model.
We will have:
1. Passengers
a. Payment Methods (so Passengers can pay for the ride)
b. Favorite Address (Address Nicknames)
c. Addresses (the proper addresses; Origin, Destination, etc)
2. Cars
a. Location (the car location)
3. Drivers
a. Credit Method (so we can pay the Driver)
4. Bookings
a. Rating (in case a Passenger wants to rate a ride)
So, we have 10 different types of “objects” in our simplified ride-hailing booking Model. The Passenger and the Car/Driver will be the main “protagonists” of the “plot” (Model), where the Passenger will create a Booking for a Car/Driver to pick up and serve. The other objects are supporting objects but fundamental parts of the “plot.”
Let’s look at each object now.
The Passenger
As we want to keep things in order within our Model, before creating Passengers, let’s apply some Set concepts and create a Collection where we will place all our Passengers.
// Create Passengers Collection
CREATE (ac:PassengerS {Name: "PassengerS"}) ;We could also create other Subsets like Premium Passengers, Loyalty Members, etc., but we will keep this simple and just create one collection for the Passengers.
Now, let’s create our first Passenger.
The Passenger object will have the following properties:
Passenger:
{Name: string,
Phone: integer,
Email: string,
Photo: string}
The “Passenger” is also a “proper noun”; therefore, each can only exist once in our Model.
In our example, we will use the Phone as a unique identifier for each Passenger and, therefore, create a UNIQUE constraint for all Passenger objects.
// Passengers are Proper Noun, so we set a Unique Constraint
CREATE CONSTRAINT unique_passenger IF NOT EXISTS
FOR (n:Passenger)
REQUIRE n.Phone IS UNIQUE;For the Photo, as the current Graphs do not support this data type, we will persist the image in another data repository and place the URL for the image (and perhaps a thumbnail, too) on the “object” property.
Let’s start by looking at the anatomy of the “Passenger” object.
I used anatomy because, as explained previously, some objects need a particular structure to perform their role in the Model.
To perform its role in the Model, the Passenger object will need a minimum structure consisting of a Wallet collection to store different Payment Methods, an Address Book collection to store Addresses, and a History collection to store past Bookings. So, we must create other supporting objects/collections for each Passenger we create.
We will connect the Passenger “object” with its WalleT collection with a relationship called [HAS_WALLET], its AddressBooK collection with a relationship called [HAS_ADDRESS], and its HistorY collection with a relationship called [HAS_HISTORY]
The Passenger will need at least one Payment method.
The PaymentMethod object will have the following properties:
PaymentMethod:
{Type: string,
Issuer: string,
NameOnCard: string,
CardNumber: integer,
ExpDate: string,
CVV: integer}
We will also create a direct relationship between the Passenger “object” and the preferred payment method with a relationship called [HAS_PREFERED_PAYMENT] so the Application can default to one Payment Method.
For the Address Book (and their Addresses), to avoid having different passengers enter duplicate addresses in the Model, we will consolidate all the distinct Addresses in one Domain Collection (AddresseS) and then connect a nickname (FavAddress) that the Passenger chooses for that Address on their Address Book.
//Create the AddresseS Domain collection
CREATE (:AddresseS {Name: "AddresseS"}) ;We will use a geocoding system like GeoHash or OpenStreetMap to maintain unique address entries for each Latitude-Longitude pair. For simplicity, we will pick GeoHash for our demo.
That is, for any given address the Passenger enters, we will generate the equivalent GeoHash, store it on an Address “object” (Node), and connect it to an AddresseS collection.
The FavAddress “object” will only have a Name that the Passenger will input. This object will then be connected with a relationship called [IS_ADDRESS] to the Address “object.”
FavAddress:
{Name: String}
The Address “object” will have the GeoHash, and we will add the Address itself.
Alternatively, we could only store the GeoHash on the object and build a Microservice to generate the Address (as street names can change), but for Demo purposes, we’ll keep it simple:
Address:
{GeoHash: string,
StreetNum: integer,
StreetName: string,
Complement: string,
City: string,
State: string,
ZIP Code: integer,
}
As the Address “object” should be unique within the Model, we will create a Unique Constraint for it:
// Address (GeoHash) is a Proper Noun, so we set a Unique Constraint on the GeoHash
CREATE CONSTRAINT unique_GeoHash IF NOT EXISTS
FOR (a:Address)
REQUIRE a.GeoHash IS UNIQUE; The resulting Graph representation of the Address and the AddresseS collection will look like this:
With this, our Passenger structure will end up looking like this:
To demonstrate how efficient Cypher/GQL is, the creation of this entire Passenger structure, with two Payment Methods, one preferred Payment Method, and two Addresses, could be created with only one Cypher/GQL command (below). As an ACID transaction, it will create 10 nodes, 13 relationships, and 33 properties, and it will perform in just around 300 milliseconds on Neo4j Aura.
// Creation of the entire Passenger structure, with two Payment Methods, one preferred Payment Method, and two Addresses (create 10 nodes, 13 relationships, and 33 properties)
MATCH (ps:PassengerS), (ad:AddresseS)
CREATE (ps)-[:HAS]->(p:Passenger {Name: "Peter", Phone: "98456683", Email: "Peter@email.com" , Photo: "https://photos.com/img/Peter.jpg"})-[:HAS_HISTORY]->(:HistorY {Name: "HistorY"}),
(p)-[:HAS_WALLET]->(w:WalleT {Name: "WalleT"})-[:HAS]->(pm1:PaymentMethod {Type: "CreditCard", Issuer: "MasterCard", NameOnCard: "Peter Pan", CardNumber: 2222420000001113, ExpDate: "08/26", CVV: 321}),
(w)-[:HAS]->(pm2:PaymentMethod {Type: "DebitCard", Issuer: "Visa", NameOnCard: "Peter Pan", CardNumber: 4701322211111234, ExpDate: "12/26", CVV: 837}),
(p)-[:HAS_PREFERED_PAYMENT]->(pm1),
(p)-[:HAS_ADDRESS]->(ab:AddressBooK {Name:"AddressBooK"})-[:HAS]->(:FavAddress {Name: "Home"})-[:IS_ADDRESS]->(:Address {GeoHash: "9xj6hs9cxw65", StreetNum: 2315, StreetName: "Krameria St", City: "Denver", State: "CO", ZIP: 80207})<-[:HAS]-(ad),
(ab)-[:HAS]->(:FavAddress {Name: "Work"})-[:IS_ADDRESS]->(:Address {GeoHash: "9xj6kq1xv0c2", StreetNum: 5990, StreetName: "Dahlia St", City: "Commerce City", State: "CO", ZIP: 80022})<-[:HAS]-(ad);The Car & Driver
Before we create a Car and Driver, we must first create the domain collections that will represent the fleet of cars and the many states in which they can be within the model.
If we think of it using a Set approach, we have a set called Fleet, which will include all the Cars that belong to our Ride-Hailing system. Within that set, there will be other subsets that represent different states in which a car could be.
For instance, as a car is registered into the System, this Car will be Pending until some due diligence is done on the Car and Driver to check if it meets the requirements to be part of the Fleet. Once it passes, the car can be Available to take on bookings. Some cars can be Rejected, and we might like to keep a record of these cars. Some cars in the fleet might be Off Duty (shit) or undergoing some Maintenance. And others, hopefully, many of them, will be Busy transporting Passengers. Lastly, some might just be Retired for some reason, and we must keep track for auditing purposes.
As for the Driver, we will keep it simple and have them all in one Set called DriverS.
Here is the Cypher/GQL command to create these Domin Collections:
// Create Car Collections
CREATE (f:FleeT {Name: "FleeT"}),
(a:AvailablE {Name: "AvailablE"})<-[:HAS]-(f),
(b:BusY {Name: "BusY"})<-[:HAS]-(f),
(m:MaintenancE {Name: "MaintenancE"})<-[:HAS]-(f),
(o:OffDutY {Name: "OffDutY"})<-[:HAS]-(f),
(r:RetireD {Name: "RetireD"})<-[:HAS]-(f),
(p:PendinG {Name: "PendinG"})<-[:HAS]-(f),
(j:RejecteD {Name: "RejecteD"})<-[:HAS]-(f);
// Create Drivers Collections
CREATE (d:DriverS {Name: "DriverS"});
Now that we have the Domain Collections, we will create the Car and the Driver.
The Car, which we will use as the primary “object” of our Model for booking, will have the following Properties:
Car:
{Plate: string,
Make: string,
Model: string,
Year: Integer,
Capacity: Integer,
Color: string}
As we would like to know the Car’s location periodically, we will connect a Loc (short for Location) “object” to the Car with a GeoHash and a timestamp. This “object” must be updated periodically via a Microservice on the Client side. (In a realistic scenario, we would likely use a streaming service like Kafka to persist/store the telemetry/location of each Car. We can then use the Loc “object” as a proxy containing a pointer of the respective car stream for the app to retrieve. But for our Demo mockup, we will store it on the Graph so we can “play” with it.)
The Loc “object” will then be quite simple and will have the following structure.
Loc:
{GeoHash: String,
Time: DateTime}
We will create a directed relationship from the Car to the Loc “object” called [IS_AT]
Let’s now look at the anatomy of the Driver “object”.
The Driver will have the following Properties:
Driver:
{Name: string,
ID: string,
License: string,
Phone: Integer,
Email: string}
As both the Driver and the Car have to be unique within the Model, we will create a Unique Constraint for both:
// Car and Driver are Proper Nouns so we set a Unique Constrain - We will use the Car licence Plate as the ID for Car and and the Drivers ID as the ID in this Demo
CREATE CONSTRAINT unique_car IF NOT EXISTS
FOR (n:Car)
REQUIRE n.Plate IS UNIQUE;
CREATE CONSTRAINT unique_driver IF NOT EXISTS
FOR (n:Driver)
REQUIRE n.ID IS UNIQUE;Like the Passenger, the Driver will need a minimum structure consisting of a Wallet collection to store at least one Credit Method for us to pay for each ride and a History collection to store past Bookings. So, we must create other supporting objects/collections for each Driver we create.
We will connect the Driver “object” with its WalleT collection with a relationship called [HAS_WALLET], and its HistorY collection with a relationship called [HAS_HISTORY]
As a Car and Driver will always go together in our Model, we should ideally create both “objects” together and connect the Driver to the Car with a relationship called [HAS_CAR] and the Car to the Driver with a relationship called [HAS_DRIVER]. These relationships should remain the same after being established as long as the Set {Car, Driver} continues to provide service to the Ride-Hailing Business.
Once the Drive/Car is created, we must place the Car into the PendinG collection. This will register the Driver/Car into the system, but the Driver/Car will still need to pass due diligence to start serving bookings.
Finally, our Drive/Car structure will end up looking like this:
The Driver/Car due diligence (back office)
The Driver/Car will initially be “Pending” and waiting for due diligence. We must have a due diligence process and build a UI for the Driver/Car to be either Rejected or Approved.
If the Driver/Car is Rejected, we will place the Car in a Rejected collection.
If the Driver/Car is Approved, we will place the car in the Available collection, indicating that the car is “available” to accept Bookings.
Or
The Booking
Now that we have created the two protagonists of our Model, the Passenger and the Car/Driver “objects” with their respective domain collections and structures, we can analyze the Booking.
Before we create a Booking “object”, we must first create the domain collections that will gather all the bookings in the many states in which they can be in the Model. We will start with a Superset called BookingS, within this superset we will have the following subsets:
WaitinG — for the bookings that are waiting for a Car/Driver to be assigned.
ActivE — for the bookings that have a Car/Druver assigned.
PasT — for the bookings that have concluded the ride.
CanceleD — For bookings that were Canceled.
And the code:
// Create the Booking Domain Collections
CREATE (b:BookingS {Name: "BookingS"}),
(:ActivE {Name: "ActivE"})<-[:HAS]-(b),
(:PasT {Name: "PasT"})<-[:HAS]-(b),
(:WaitinG {Name: "WaitinG"})<-[:HAS]-(b),
(:CanceleD {Name: "CanceleD"})<-[:HAS]-(b);
// A Booking is a Proper Noun, so we set a Unique Constraint - The system will need a Microservice to generate unique BookingID's; we will create manual ones for this Demo mockup
CREATE CONSTRAINT unique_booking IF NOT EXISTS
FOR (n:Booking)
REQUIRE n.BookingID IS UNIQUE;The Booking “object” will have the following properties:
Booking ID — This could be a sequential number or a hash (built from some properties of the Passenger + Timestamp, etc). So, we will need a code or microservice to generate the Booking ID.
Time Stamp — so we can know when it was created
Fare — That will be dynamically calculated based on the distance between the Origin and the Destination, the estimated travel time, time of the day, weather conditions, supply (cardinality of AvailablE) vs. demand (cardinality of WaitinG), etc. An algorithm must be defined.
For the Origin and Destination of the Booking, instead of using Properties and duplicating data, we will connect the Booking “object” with the respective addresses using the relationships [HAS-ORIGIN] and [HAS_DESTINATION]
Similarly, for the Payment Method, we will connect the Booking with one of the Payment Methods on the Passenger’s WalleT that he/she selects. We will default to the passenger’s preferred payment method.
So, our Booking “object” will look like this:
Booking:
{ID: string,
Date: DateTime,
Fare: float}
And the Booking anatomy will look like this:
After defining how the Passenger, the Driver with its Car, and the Booking with their supporting objects and their respective structures in order to operate within the Model at every different state, your model should look something like this:
You don’t need to be a Rocket Scientist, a Data Scientist, or a Computer Scientist to understand this. You don’t even need to be a scientist. No matter what your domain of expertise is, you will be able to understand and have insightful and productive conversations about this model, suggest changes, and find glitches and areas of improvement. You will use the exact mechanisms you did to find a spoon in my house.
The best part is that it will look the same when you finally agree on the Model amongst all the stakeholders and implement it on the Graph Database. It’s a symmetric transposition from your drawing board to the Database. Normalizing or converting it to a different database structure is unnecessary. It will be the same process every time you need to revise or enhance the model.
When transposed to the Graph Database, it will look and work like this:
Now, let’s see what happens when we start changing the relationship between the “objects” to reflect the different states in which the Model can be. Let’s see what the booking lifecycle would look like in each state.
We start at T0 (T Zero) when a Passenger signs in to make a booking.
This is the state of the Model at T0:
T1 — When the Passenger creates a booking, a new booking “object” will be created following the predefined booking structure (Fig. 30). It will have connections to the Origin and Destination addresses, a Fare property, a connection to the Payment method, and a Timestamp property. It will be connected to and from the Passenger who created the Booking. Last but not least, the newly created booking will be placed in the Waiting (state) collection ({booking} ∈ Waiting).
T2 — When an Available Car ({Car} ∈ Available) is assigned to the Booking, the Car will then be connected to and from the Booking. As the Car is no longer available, it will disconnect from the Available collection ({Car} ∌ Available) and be connected to the Busy collection ({Car} ∈ Busy). Likewise, the booking will no longer be in the Waiting state. It will be disconnected from the Waiting collection ({booking} ∌ Waiting), and at the same time, it will be connected to the Active collection ({booking} ∈ Active).
T3 — When the Car drops the Passenger at the final destination, the Car becomes Available again; therefore, we disconnect the Car from the Busy collection ({Car} ∌ Busy) and connect it back to the Available collection ({Car} ∈ Available). Likewise, the booking will no longer be in the Active state. It will be disconnected from the Active collection ({booking} ∌ Active), and at the same time, it will be connected to the Past collection ({booking} ∈ Past) so that we can have a record of the booking. Similarly, as both the Passenger and the Driver would like to recall their past activity, we will connect the Booking to their respective History collection ({booking} ∈ History). We will disconnect the relationship between the Passenger and the Car to the Booking. However, we will keep the connection from the Booking to both the Passenger and Car, as this information is pertinent to the Booking.
T4 — Lastly, if the Passenger wishes, he/she can rate the ride and add a Rating to the Booking. To do so, the Passenger creates a Rating “object” and connects it to a Past bookings.
To put everything into perspective, below, you will see an animation with all the states of the model as the different users interact with it, and a series of events unfold during a Booking timeline:
The Interaction Model/Interaction Architecture
Now that we have a defined Data Model, we will identify the user interactions with their respective events, which will create the “objects” and “object structures” and change the relationships between them.
As mentioned earlier, each UI should be atomic, stateless and operate as a function, receiving parameters from the calling UI and passing parameters to the following UI they call. These parameters will be used as anchor nodes to parse the Cypher/GQL commands the UI executes against the Graph.
We start by wireframing the UIs that each User will use to interact with the Model
As this is a mockup, I will not use a proper Identity Management system to pass the Passenger or Driver’s unique identifier (anchor Node). Still, instead, I will use the Passenger{Phone} and the Drivers{ID} as their anchor Node.
As mentioned, I will focus on the Graph Cypher/GQL commands rather than the application’s UI/UX aspects.
The Passenger (Passenger App)
Passenger Particulars & Structure
BOM: We will need a UI to collect the Passenger object information and create the Passenger object and its minimum structure in the model, connecting it to the PassengerS domain collection by executing the Cypher/GQL command below:
MATCH (ps:PassengerS)
CREATE (ps)-[:HAS]->(p:Passenger {Name: "<Passenger Name>", Phone: "<phone>", Email: "<email address>" , Photo: "<URL provided by the microservice>"})-[:HAS_HISTORY]->(:HistorY {Name: "HistorY"}),
(p)-[:HAS_WALLET]->(w:WalleT {Name: "WalleT"}),
(p)-[:HAS_ADDRESS]->(ab:AddressBooK {Name:"AddressBooK"});BOM: we will need a microservice to store the Photo provided by the Passenger on a BLOB repository and return the URL of the photo.
Passenger Payment Methods (Anchor nodes: Passenger Phone)
BOM: We will need a UI for the Passenger to input the PaymentMethod information, create the PaymentMethod object, and connect it to the Passenger’s WalleT collection. This UI will also request the Passenger to select the preferred PaymentMethod if more than one was inputted and connect it to the Passenger “object”
MATCH (p:Passenger { Phone: "<phone number>"})
CREATE (p)-[:HAS_WALLET]->(w:WalleT)-[:HAS]->(pm1:PaymentMethod {Type: "<type>", Issuer: "<issuer>", NameOnCard: "<name on card>", CardNumber: <card number>, ExpDate: "<exp. Date>", CVV: <CVV>}),
(p)-[:HAS_PREFERED_PAYMENT]->(pm1);Passenger Address Book (Anchor nodes: Passenger Phone)
BOM: we will need a UI and Microservice to check and convert an address inputted by the user into a GeoHash. The UI will then execute the below Cypher/GQL command to check if an Address exists in our AddresseS collection, and if not, create a new Address “object” and connect it to the Passenger’s AddressBook entry (FavAddress). If the Address “object” already exists in the AddresseS collection, it will just connect the Passenger’s AddressBook entry (FavAddress) to it.
These are the Cypher/GQL commands:
// We check if the GeoHash already exists in the AddresseS collection
MATCH (a:Address {GeoHash: "<GeoHash>"}) RETURN count(a);
// IF the GeoHAsh already exists (count(a) = 1), we connect it to the Passengers Address book entry using the Name provided by the Passenger
MATCH (a:Address {GeoHash: "<GeoHash>"}), (p:Passenger {Phone: "<phone number>"} )-[:HAS_ADDRESS]->(ab:AddressBooK {Name:"AddressBooK"})
MERGE (ab)-[:HAS]->(:FavAddress {Name: "<Nickname>"})-[:IS_ADDRESS]->(a);
// ELSE if the GeoHash does not exist (count(a) = 0), we create a new GeoHash Node, connect it to the AddresseS collection and to the Passengers Address book entry using the Name provided by the Passenger
MATCH (p:Passenger {Phone: "<phone number>"} )-[:HAS_ADDRESS]->(ab:AddressBooK {Name:"AddressBooK"}), (ad:AddresseS)
MERGE (ab)-[:HAS]->(:FavAddress {Name: "<Nickname>"})-[:IS_ADDRESS]->(:Address {GeoHash: "<GeoHash>", StreetNum: <number>, StreetName: "<street name>", City: "<city name>", State: "<state>", ZIP: <zip code>})<-[:HAS]-(ad);
Passenger Booking (Anchor nodes: Passenger Phone)
BOM: We will need a UI for the Passenger to make a Booking. This UI will ask for the Origin and Destination addresses and the payment method (we will default to the preferred payment method, and the user can change it if he/she desires) The user can select any saved addresses (FavAddress) or input new ones. If new addresses are input, we must convert the Addresses into GeoHash and check if the GeoHash is already in our AddresseS collection. If so, we will connect the Booking to the proper Address using the respective relationship (HAS_ORIGIN or HAS_DESTINATION]. If it does not exist, we create an Address “object” on the AddresseS collection and will connect the new Address to the Booking with the proper relationship.
We will need a Microservice to calculate the Fare using the Origin and Destination provided, applying the Fare algorithm to return the ride Fare. Upon acceptance (event), we create the Booking and connect it to the Passenger with the [HAS_ACTIVE_BOOKING] relationship, the Addresses as described earlier, and the Payment Method with the [HAS_PAYMENT]. We will place the Booking in the WaitinG collection for it to be served. We do this by executing the below Cypher/GQL command:
// We Create the Booking and connect it to both Origin and Destination Addresses and to a Payment Method. We then connect the Booking to the WaitinG collection - create 1 Node, create 6 relationships in one Transaction
OPTIONAL MATCH (p:Passenger {Phone: "<phone number>"})-[:HAS_PREFERED_PAYMENT]-(pm), (w:WaitinG {Name: "WaitinG"}), (oa:Address {GeoHash: "<GeoHash>"})<-[:HAS]-(:AddresseS)-[:HAS]->(da:Address {GeoHash: "<GeoHash>"})
CREATE (b:Booking {BookingID: "<BookingID>", Date: localdatetime.transaction(), Fare: <fare> })-[:HAS_PASSENGER]->(p),
(b)-[:HAS_PAYMENT]->(pm),
(b)-[:HAS_ORIGIN]->(oa),
(b)-[:HAS_DESTINATION]->(da),
(p)-[:HAS_ACTIVE_BOOKING]->(b),
(w)-[:HAS]->(b);Passenger Booking Rating (Anchor nodes: Passenger Phone + Booking ID selected)
BOM: We will need a UI for the Passenger to rate the ride and execute the Cypher/GQL command:
// Passenger Rates the ride:
MATCH (p:Passenger {Phone: "<phone number>"})-[:HAS_HISTORY]->()-[:HAS]-(b:Booking {BookingID:"<BookingID>"})
CREATE (b)-[:HAS_RATING]->(r:Rating {Date: localdatetime.transaction(), Stars: <Rating>, Comments: "<Passenger comments>"});The Drive (Driver App)
Driver/Car Particulars & Structure
BOM: We will need a UI to collect the Driver’s information and create the Driver object and its minimum structure in the model (WalleT and HistorY collections), connecting the Driver to the Driver’s domain collection (DriverS). The UI must request at least one credit method, create the CreditMethod object, and connect it to the Driver’s Wallet collection. This UI will also request the Driver to select the preferred CreditMethod if more than one was inputted and connect it to the Driver “object”. This UI will also need to gather the Car information and create the Car “object” with its initial Location object. We will need a Microservice to gather the GPS Latitude and Longitude of the Driver, create a GeoHash, and update the Car’s location “object.” Lastly, connect the Pending domain collection to the new Car “object”.
// Driver/Car Particulars & Structure
// Create a new Driver and structure
OPTIONAL MATCH (dc:DriverS {Name: "DriverS"})
CREATE (d:Driver {Name: "<Driver Name>", ID: "<Driver ID>", License: "<Drivers License>", Phone: "<Driver Phone Number>", Email: "<driver email>" , Photo: "<URL of Driver's photo>" }),
(dc)-[:HAS]->(d),
(d)-[:HAS_HISTORY]->(:HistorY {Name: "HistorY"}),
(d)-[:HAS_WALLET]->(w:WalleT {Name: "WalleT"});
// Create a Car and assigne (connect) it to the Driver - set current location (Anchor nodes: Driver ID)
MATCH (d:Driver {ID:"<Driver's ID>"}), (p:PendinG {Name: "PendinG"})
CREATE (d)-[:HAS_CAR]->(c:Car { Plate: "<Car licence plate number> ", Make: "<Car make>", Model:"<Car model>", Year: <Car Year>, Capacity: <Car passenger capacity>, Color: "<Car color>"})-[:HAS_DRIVER]->(d),
(c)-[:IS_AT]->(:Loc {GeoHash: "<GeoHash>" , Time: localdatetime.transaction()}),
(p)-[:HAS]->(c);
// Create a new Credit Method and set it as Prefered (Anchor nodes: Driver ID)
MATCH (d:Driver {ID:"<Driver's ID>"})-[:HAS_WALLET]->(w:WalleT {Name: "WalleT"})
CREATE (w)-[:HAS]->(cm:CreditMethod {Type: "Bank Tranfer", AccName: "<Account Name>" , AccNumber: <account number>, SWIFT_BIC: "<SWIFT Code>"}),
(d)-[:HAS_PREFERED_CREDIT]->(cm);This is what the Cypher/GQL command that the UI will execute to build the above Drive/Car structure and place the Car in the Pending collection for each new Driver/Car that signed up in the System if we were to do it in one unique command.
// All of the abouve in one command: Create one Driver structure with Wallet, History, and its respective Car with the initial Location (Loc) and connect the Driver to the Car and vice-versa. Place the Car in the Peding Collection
OPTIONAL MATCH (p:PendinG {Name: "PendinG"}), (dc:DriverS {Name: "DriverS"})
CREATE (d:Driver {Name: "<Driver Name>", ID: "<Driver ID>", License: "<Drivers License>", Phone: "<Driver Phone Number>", Email: "<driver email>" , Photo: "<URL of Driver's photo>" })-[:HAS_CAR]->(c:Car { Plate: "<Car licence plate number> ", Make: "<Car make>", Model:"<Car model>", Year: <Car Year>, Capacity: <Car passenger capacity>, Color: "<Car color>"})-[:HAS_DRIVER]->(d),
(dc)-[:HAS]->(d),
(d)-[:HAS_HISTORY]->(:HistorY {Name: "HistorY"}),
(d)-[:HAS_WALLET]->(w:WalleT {Name: "WalleT"})-[:HAS]->(cm1:CreditMethod {Type: "Bank Tranfer", AccName: "<Account Name>" , AccNumber: <account number>, SWIFT_BIC: "<SWIFT Code>"}),
(d)-[:HAS_PREFERED_CREDIT]->(cm1),
(c)-[:IS_AT]->(:Loc {GeoHash: "<GeoHash>" , Time: localdatetime.transaction()}),
(p)-[:HAS]->(c);Update Car Location (Anchor nodes: Driver ID)
BOM: To update the Car Location from time to time we will need the Driver Client App to create a GeoHash of its location (Latitude, Longitude) and update the Loc “object” of the Car.
// Cypher command to Updating a Car current location from time to time - The system will need a Microservice to generate the GeoHash of the current car location based on GPS Latitude and Longitude
MATCH (d:Driver {ID:"<Driver’s ID>"})-[HAS_CAR]-()-[IS_AT]->(l:Loc)
SET l.GeoHash = "<GeoHash>", l.Time = datetime();Find and select Bookings that are Waiting.
BOM: Create the UI, which will be the main screen for existing Drivers containing the waiting list of Bookings, ordered by waiting time and/or proximity to the current location (Using GeoHash).
// Find Bookings that are Waiting
MATCH (w:WaitinG)-[:HAS]->(b:Booking)-[:HAS_PASSENGER]->(p:Passenger), (oa)<-[:HAS_ORIGIN]-(b)-[:HAS_DESTINATION]->(da)
RETURN b.BookingID, p.Name, oa.StreetNum, oa.StreetName, da.StreetNum, da.StreetName, apoc.temporal.format(duration.between(b.Date,datetime()),'HH:mm:ss.SSS') as TimeWaiting , b.Fare
ORDER BY b.Date;As we are using GeoHash in our Demo, we can use it to find the Cars that are near a Booking request (Origin), or the other way around, the Booking requests that have an Origin near a Car, and adjust the precision of the GeoHash algorithm to specify the radius of search.
//For Push Bookings (System selects a Car that is near to Booking's origin) - Find all Availabe Cars in the Fleet that are near (~15Km Radius) to Geohash "9xj3f3yvo43o" (1298 S Raritan St, Denver, CO 80223, USA)
MATCH (:AvailablE)-[:HAS]-(c:Car)-[:IS_AT]-(l:Loc) WHERE left(l.GeoHash, 4) = "9xj3" RETURN c ;
//For Push Bookings (System selects a Car that is near to Booking's origin) - Find all Availabe Cars in the Fleet that are closer (~2.7Km Radius) to Geohash "9xj3f3yvo43o" (1298 S Raritan St, Denver, CO 80223, USA)
MATCH (:AvailablE)-[:HAS]-(c:Car)-[:IS_AT]-(l:Loc) WHERE left(l.GeoHash, 5) = "9xj3f" RETURN c ;
//For Pull Bookings (Driver selects a Booking that has an Origin near to them) - Find all Waiting Bookings near (~15Km Radius) GeoHash "9xj6"
MATCH (:WaitinG)-[:HAS]-(b)-[:HAS_ORIGIN]-(a) WHERE left(a.GeoHash, 4) = "9xj6" RETURN b;Driver Selects a Booking (Anchor nodes: Driver ID + BookingID selected)
BOM: Upon selecting a Booking, execute the Cypher/GQL command to commit to a booking.
// Driver selects a Waiting booking: Assigne Car to Booking, Remove Car from Available, Connect Car as Busy, Connect Booking as Active, and Remove Booking from Waiting - create 4 relationships, disconnect 2 relationships in one Transaction
OPTIONAL MATCH (b:Booking {BookingID:"<Booking ID selected>"})<-[r1:HAS]-(w:WaitinG)<-[:HAS]-()-[:HAS]->(a:ActivE),
(d:Driver {ID:"<Driver’s ID>"})-[:HAS_CAR]->(c)<-[r2:HAS]-(:AvailablE)<-[:HAS]-()-[:HAS]->(u:BusY)
DELETE r1, r2
CREATE (a)-[:HAS]->(b),
(c)-[r:HAS_ACTIVE_BOOKING]->(b),
(b)-[t:HAS_CAR]->(c),
(u)-[:HAS]->(c);Driver Drops the Passenger (Anchor node: Driver ID)
BOM: Upon completion of the ride, the Driver must indicate that the Passenger has been dropped at the Destination and execute the following Cycher/GQL command:
// Passenger is dropped at destination: Delete active_booking from Passenger and Car, Mark Booking as Past, Mark Car as Available - connect 4 Edges, disconnect 4 Edges in one Transaction
OPTIONAL MATCH (dh:HistorY)<-[:HAS_HISTORY]-(d:Driver {ID:"<Drivre’s ID>"})-[:HAS_CAR]->(c)-[r1:HAS_ACTIVE_BOOKING]->(b)<-[r3:HAS]-(:ActivE)<-[:HAS]-(:BookingS)-[:HAS]->(t:PasT),
(a:AvailablE)<-[:HAS]-(:FleeT)-[:HAS]->(u:BusY)-[r2:HAS]->(c),(ph:HistorY)<-[:HAS_HISTORY]-(p:Passenger)-[r4:HAS_ACTIVE_BOOKING]->(b)
CREATE (t)-[:HAS {Date: datetime()}]->(b),
(a)-[:HAS]->(c),
(dh)-[:HAS {Date: datetime()}]->(b),
(b)<-[:HAS {Date: datetime()}]-(ph)
DELETE r1, r2, r3, r4;
Mark a Car as OffDuty or Maintenance (Anchor node: Driver ID)
BOM: Create a UI for the Driver to indicate when they are Off duty or when the car is on Maintenace — Non availability for booking.
// Mark Car as OffDuty
MATCH (d:Driver {ID:"<Driver’s ID>"})-[:HAS_CAR]->(c)<-[r:HAS]-(a:AvailablE)-[]-(:FleeT)-[]-(o:OffDutY)
DELETE r
CREATE (c)<-[:HAS {Date: datetime()}]-(o);
// Mark Car from Available to Maintenance
MATCH (d:Driver {ID:"<Driver’s ID>"})-[:HAS_CAR]->(c)<-[r:HAS]-(a:AvailablE)-[]-(:FleeT)-[]-(m:MaintenancE)
DELETE r
CREATE (c)<-[:HAS {Date: datetime()}]-(m);Car/Driver due diligence (Back Office App)
BOM: We will need a UI for the due diligence manager to Approve or Reject the Cars in the PendiG collection and execute the respective Cypher/GQL command to connect the Car to the proper Collection.
// After validating a Car/Driver, we move them to the AvailablE collection so they can start receiving Booking requests and serving Passengers
MATCH (c:Car { Plate: "<Car licence plate>"})<-[r1:HAS]-(p:PendinG {Name:"PendinG"}), (a:AvailablE {Name: "AvailablE"})
CREATE (c)<-[r2:HAS]-(a)
DELETE r1;
// If a Car does not pass de Due Diligence we disconnect it from the PendinG collection and connect it to the RejecteD collection for the record/auditing
MATCH (c:Car {Plate: "<Car licence plate>"})<-[r1:HAS]-(p:PendinG {Name:"PendinG"}), (j:RejecteD {Name: "RejecteD"})
CREATE (c)<-[r2:HAS]-(j)
DELETE r1;Removing a Car from the Fleet
BOM: We will need a UI for the back office to mark a car as retired and execute the respective Cypher/GQL command to connect the Car to the proper Collection.
// Mark Car as Retired
MATCH (c:Car {Plate: "<Car licence plate>"})<-[r:HAS]-(a:AvailablE)-[]-(:FleeT)-[]-(e:RetireD)
DELETE r
CREATE (c)<-[:HAS {Date: datetime()}]-(e); This concludes our Interaction Model/Interaction Architecture section. As you can see, little code is left to write besides the UIs (HTML, CSS, and JS) and the few Microservices that need to be developed. The Graph will handle most of the application logic and state handling.
Get the code and try it out.
The Cypher code for this Demo is on GitHub. You can try it with Neo4j Aura. If you don’t have an Aura account yet, get one HERE. If you are new to Cypher, you can get started at Neo4j’s GraphAcademy.
Conclusion
Graph Databases can do way more than Knowledge Graphs, RAG, Fraud Detection, Recommendation Engines, and other specific workloads. They are transformational and can potentially be the foundation for our daily information systems. They help bridge the gap between the problem and solution domain, establishing a collaboration framework where both sides can work together like never before. They have space to evolve and support application development workloads, and I pointed out some limitations.
With the recent publication of an ISO Global Standard Query Language for Graph Databases (GQL), the path is open for them to transform the industry.
Based on my experience 10 years ago, I shared one approach to using Graph Databases for developing applications, and I’m sure many more will emerge. I spent about 4 to 5 years working with and perfecting the approach I shared. And most recently, I adapted it to Cypher/GQL. It completely changed my understanding of Application Development (and I have over 40 years of experience in this space).
So, I had to share it.
It might take you time to really comprehend it, but when you do, that quote at the beginning of the article will make a lot of sense.
Graph Databases are the future!
Disclaimer: This article was written with GenNI (Natural Intelligence) as GenAI still needs to figure out whether all this I wrote is possible. I had to rush before it did, so excuse me for any typos. 😉