Image for post
Image for post

Building a low-code platform with Neo4j (2/4) — Data

Stefan Dreverman
Jun 5 · 6 min read

This is part 2/4 in a series on how to start creating a low-code platform. In this part, the design of reading/writing data is explained.

It might be the most crucial aspect of the platform, since data structures and instances of these structures are all defined and created with this code.

The most important functionalities in the data-layer are:

  • CR(U*)D any type (the data structure)
  • CRUD instances of any type (data that conforms to a structure)

The requirement of ‘creating the data structure once’ (see Part 1) dictates that the data structure is only stored in the database because this is where we keep our data, which includes data about data structures.

Another requirement is that you’d want to utilize the language and structure of the database as much as possible. The reason to use the language and structure of the database is simple: Databases have optimized a lot for data storage and retrieval. They also provide valuable additional functionality to manipulate and work with data.

Although this requirement sounds obvious, I’ve worked with existing platforms that adopt(ed) their own data structure on top of the DB structure. Not using the database structure is sub-optimal: database functionalities cannot be used and the read/write speed of the database can never be met. This can lead to serious performance issues.

*the U (Update) is between brackets here. Since changing a type can have big consequences for existing instances of that type, I’ve not implemented this in the PoC.

Since the platform is using Neo4j, the structures created with the platform must be natively readable by Neo4j. The following meta-meta model is one of the possibilities:

Image for post
Image for post
The meta-meta data structure

Those of you who know Neo4j would say that this is the data structure of Neo4j. The answer is: “Yes, in adapted form. And purposefully so”. In doing so, this model can also be expressed in terms of native Neo4j/Cypher, so the power of Neo4j can be used to its full extent.

While Neo4j does store the structure of data it holds, Neo4j does not dictate the data structures that are going to be written to it. This is good nor bad, it’s just the way Neo4j works. Neo4j expects you to define that structure yourself. For this platform, we do want to know and dictate the structure and we’ll use the database itself to store it. :-)

Domain and MetaLookupAttr are different/new from the standard Neo4j model:

Domain — I’ve used labels to name the domain a metaType is in to define a collection of strongly connected metaTypes within one domain. This also allows two metaTypes with the same name to exist in the same database, as long as they belong to two different domains.

MetaLookupAttr — Where an attribute can be filled by the user itself, there are attributes that are looked up in reference data. It allows users to define lists (for example a MetaType called Shape, which contains Square, Circle and Triangle), force the user to choose one of these values AND store the value into the attribute field (instead of creating a MetaRelation).

There are other types (of attributes or relation) that can be added. Like a QueryableAttr, which can be used to sum the price of all OrderItems related to an Order. These are behavioral in nature so for this PoC I’ve only included one example of a behavioral addition.

Data and its structures need to be loaded for use with Functional modules and need to be served to a Screen for input purposes. Reading the data (structures) is as easy as an Neo4j Cypher query. This is used a lot, it’s put in a service that handles all the generic read/write functions:

Neo4j Service (Data Service module) — This service contains all functions to read/write data structures and instances. It relies on the meta-meta model given above.

This is the function that retrieves all attributes for a metaType:

public function get_typeattr($neocl,$metaType,$domain)
{
$query = 'MATCH (n:'.$domain.')-[:HASATTR]->(a:MetaAttr) WHERE n.name = {metaType} RETURN a.name AS name,a.attrtype AS type,a.defval as defval, a.description as desc ORDER BY a.name';
return $neocl->run($q, ['metaType' => $metaType]);
}

And this is a function to delete an instance:

public function delete_instance($neocl, $metaType, $instanceID)
{
$query = 'Match (n:'.$metaType.') WHERE n.in_id = {instanceID} DETACH DELETE n';
$result = $neocl->run($query, ['instanceID' => $instanceID]);
}

Note 1: $neocl is the neo client connection I’m using to execute the queries and retrieve its results. It’s instantiated elsewhere.

Note 2: Neo4j Cypher enthusiasts that see the string concatenation for the parameter: Unfortunately, labels cannot be passed as parameters, which I need because of the abstraction going on.

There are several of these functions that help read/write the data needed for use in the functional modules. The larger ones deal with creating or merging data structures and instances. They all basically translate data from the database to serve as interface-readable data or vice versa. It’s best to explore those from the PoC itself, since there’s a lot of mapping and transformation going on.

Once a basic interface is created, it can be used in Functional modules. The first of which are dedicated to enable the end-user to create, read, update and delete data (structures) in form of an:

Edit Controller (Functional module) — Contains default Screens to write/read structures and instances. This controller contains a generic twig-form in which the fields are rendered by the controller. The latter can be done by:

  1. Fetch a structure (with the Neo4j Service)
  2. Iterate structure → foreach($attributes as $attribute){…
  3. Generate form elements → $form->add(…)
  4. Present form → $form->handleRequest($request)
  5. Process results (with the Neo4j Service)

This allows data structures to only exists in the database. The code is generic so it can handle all data structures that conform to the meta meta model.

The following form is used to define the data structure:

Image for post
Image for post
Screen to add a new type

It allows users to enter 0..n: attributes, lookup attributes and relations. In the PoC, the Name and Description attributes are mandatory, because I’ve found it useful for fast prototyping and creating generic screens.

Once the data structure is created, a form can be generated to create instances with this structure. In the following example an instance of the type Frame is being edited:

Image for post
Image for post
Editing an instance of metaType Frame

Note that when editing an instance, both the instance and the data structure of the type need to be retrieved: The form is generated with the data structure and the fields are filled with the instance data.

In part 3, the meta model for Screens (not the input ones) in the PoC is explained.

And have a look: I’ve published a PoC for you to discover its inner workings and then make the decisions to build your own. It can be found on GitHub — InfoNotion. (Want to contribute or build the real deal? Please send a note!)

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store