CortexDB: From context management to an application
The agile development of applications requires an also agile architecture. The CortexDB and its tools provide developers with the flexibility needed for agile implementation. This partly requires a rethinking in development.
When using classic project methods, the contents are already identifiable in advance, the costs can be calculated and risks can be identified. At the beginning of the project, a definable final state must be assumed, which can be described and planned in detail. These methods are therefore considered a monolithic approach.
A time buffer is added for unexpected events or new findings during the project phase. Thus, there is a fuzziness in the planning, which increases during the ongoing project and has to be compensated later by an increased use of resources or capital. This often has negative effects on costs, time requirements and the quality of the result.
In contrast, agile development follows a product vision, which is maintained in a collection of all previously known components (so-called backlog). This is subject to constant adaptation and is based on the expectation that after each development cycle a potentially usable result is available. The success is thus based on close teamwork, fast feedback, high reactivity and continuous process improvement. The members of the team therefore carry a high level of responsibility for the success of the project.
Database driven applications in a dynamic environment
Applications for a wide range of specialist areas must meet requirements that are subject to business objectives. An application is therefore developed to achieve business objectives. However, these goals are constantly changing. This means that adaptations to the application are necessary and software systems must be made available more quickly, more frequently and in better quality.
Within companies, database specialists and administrators are often assigned to operations and not to development. In addition, a distinction is also often made between database administrators and developers. Administrators perform general administrative tasks and have little knowledge of the applications; database developers work closely with application developers and have a deeper knowledge of the applications. Users, developers, database developers, and database administrators also have different views of the development and deployment process. In addition, databases are considered a sensitive component of an application. Especially in relational databases with a grown structure and a large number of dependencies, a deep understanding is necessary in order to be able to implement upcoming changes as risk-free as possible. This circumstance has led to the fact that in many companies databases have now become a bottleneck in the otherwise agile development process.
As a solution for this circumstance, there is no need to extend the databases from a certain complexity or for further requirements. From this point on new databases are created (a new data silo). This creates a silo landscape that distinguishes between operational (application) databases and analytical databases. Interfaces are then to be integrated between the individual silos in order to transfer, synchronize, aggregate or archive data. A zoo of different, self-contained systems is created.
The agile development process for database requirements
Based on applications for different business areas, such as CRM, ERP, master data management, etc., changes to the database are often objects and functions within a business process. For example, fast result finding (persons, articles, …), changes (new content), connection between objects (customer A bought product B), and many other requirements.
In a relational database model, the objects of the department are divided into different tables. Simple changes from the perspective of the user department can therefore result in complicated processes from the perspective of the database developers.
On the one hand, object-oriented mapping is mandatory for simplification (object-relational mapping), on the other hand, there is no generally accepted model for object-oriented data modeling within relational databases. Due to this complexity, the usage of agile project methods for (relational) database changes is limited. It requires the assignment of database administrators and developers to the software development and furthermore longer processing cycles (“sprints”) for the changes. This circumstance can be countered with a schematic database.
The CortexPlatform as a solution strategy
The CortexPlatform includes the database system CortexDB and tools based on it for data integration, interfaces for developers and other administrative tools and applications. The objects of the business processes and their context are in the foreground and the CortexDB manages data in the sense of an “Information Context Management System” (ICMS — see link list below). The technical mapping of the different requirements of individual departments is therefore performed exactly as they have been defined.
The usage as a schema-less database with the integrated functions of the CortexDB is possible via the corresponding API just like with other products. Here the content of the database is accessed directly to create, read, update or delete objects. Every change to an individual object is transaction-safe. Figure 2 shows in the middle part the database level with the API for the CortexDB. This enables direct communication with the database without the need for additional tools; only the basic functions of the CortexDB are available. This includes, for example, a complete index of all contents, which can be used universally without administration. This simplification for developers is supplemented by additional functions when accessed via the API functions of the CortexUniplex.
The CortexPlatform in the database environment
In general, database applications work on the basis of relational databases. These are table-oriented solutions that specify a table schema and depend on configurable indexes. Such an index is configured and activated by an administrator or developer. This is usually a copy of the contents of one (or more columns) as a sorted list, so that access can be made very quickly under algorithm control.
Searches within the indexed fields can therefore only be performed in the combination as represented by the index. Searching using other fields and their combinations results in additional indexes. As a consequence, each additional index requires more memory and has to be adjusted.
Schemaless databases are generally assigned to the NoSQL area. Records are not subject to a superordinate schema and are considered as so-called “containers”. This allows the necessary flexibility when storing objects. However, as with SQL databases, an index is required for selection. This index must be configured by administrators and developers and demands the same consequences as in a relational environment.
The Concept of the CortexPlatform
The challenges of table-oriented databases for use with SQL and schema-free databases (NoSQL) show that there are requirements for users on the one hand, but also for developers on the other hand, to store data records in such a way that they can be selected quickly, but also to store them in the sense of objects. This results in a combination of schema-less storage with a universal index. This is offered by the CortexPlatform combined with the CortexDB at its core.
The CortexDB is to be assigned to the schema-less databases. Records correspond to containers without a predefined structure. This also applies to individual field contents whose type is not required for the CortexDB. Only when using an application it is necessary to interpret and display the contents accordingly.
In addition, the CortexDB indexes all contents of all fields of all containers in an universal index. Manual adjustments are no longer necessary. It is irrelevant which field type actually exists. Texts, dates and numerical values, as well as graphs (links) to other objects are subject to automatic indexing and can therefore be selected in any combination. The CortexUniplex application offers functions based on this and its own API.
The CortexUniplex as an entry into development based on the CortexDB
The CortexUniplex uses the containers within the CortexDB for the integrated data, but also for its own structure. So developers can access all contents and use data with the configured settings. The result is a database environment with agile customization options.
The configuration of the CortexUniplex means that the UniplexAPI can access and use the configuration. Preconfigured selections and output lists (also pivot and formatting of the contents), rights and roles and other settings are available to a developer. Correspondingly, the configuration of automatisms and their execution takes place within the CortexDB with the CortexUniplex. The results are also available via the API; the development effort is reduced to the interactive application and the definition of processes in advance.
The UniplexAPI is accessed, which in turn uses the configurations of the CortexUniplex to access the data via the API of the CortexDB. The functions of the database are thus easily available.
The internal structure of the CortexDB
The CortexUniplex and the various APIs allow access to the data within the CortexDB.
- for transaction-dependent automatisms at the time a container is changed,
- for subsequent automatisms after changing containers (queue),
- for the time-controlled execution (to the minute) of code by the database server.
Using stored code
For simple changes based on the transactions of the containers, the first form of automatisms (so-called “reporters”) is suitable. The CortexDB generates results based on certain rules or its own code and writes them to the previously changed container or to a second container linked to it. Typical use cases include summing up values, counting specific occurrences, or checking field contents and setting a status.
The subsequent automatism (so-called “agent”) is used to process changes across data records and can also access other resources of the operating system (for example, to write a file, connect to another database, and so on). Here, the initiator of the execution is again a transaction.
The time-controlled execution of code (the so-called “scheduler” and most interesting of the three) can be used free of transactions and runs exclusively within the database. Any content (container, values or code) is accessible. Variables can be stored in code, in fields of the code container or in other containers. The configuration and modification of the stored code is done within the containers. These can be edited - in addition to API usage - with the help of CortexUniplex.
Usage of the CortexUniplex
Within CortexUniplex, the output and processing of the containers is done in a configurable representation (the so-called record type). The schematic storage is thus presented to a user in an understandable way and can be used for various applications.
For the representation and access, corresponding rights and roles can be defined. A user therefore only gets access to the information for which access rights exist; this UniplexAPI also observes these access rights.
In addition to this, the configuration of selection options (so-called portals) can be carried out. This enables the repeated use of selections for different use cases. For example, the selection of data records with a certain status, certain contents in different fields or also according to processing statuses. The mapping of work processes is thus possible and the configurations of these selections are also available via the UniplexAPI.
List representations are mandatory for output within CortexUniplex. These allow the presentation of single or many containers with selected contents. Please note that the line display of a container can contain a large number of sub-lines (so-called sub-lists), for example, to display and use information from other data sets via graphs (links). This makes it possible, for example, to display a hierarchical list of parts of a master data set or a networked structure. As with the functions mentioned above, these lists are also available via the UniplexAPI.
Each output via the UniplexAPI returns a so-called JSON object. The handling is therefore possible in any programming language.
This JSON object contains the configuration of the displayed contents (field configuration), as well as the contents themselves. Further processing in another application or user interface is thus easily possible.
The lists as output of many data records (e.g. as a selection result), as well as the search configurations of the portals, the data record configurations, rights and roles and also pivot lists for aggregated information are available in this way. The logic of the application itself, the data model for the objects and automatically running processes are thus available via the CortexPlatform in a configurable way. These functions are used via an individual application.
The configuration of the CortexUniplex and the use of the UniplexAPI for the exchange of JSON objects allows the further processing of the objects. The technical format JSON can then be converted into a layout by a developer and the CortexUniplex functions and the mapped business processes can be used in a separate application.
From the point of view of software development, the separation is clearly visible here in the sense of the paradigm “model - view - controller”. Since the Model (database) and Controller (logic) areas are configured on the server side of the CortexPlatform, only the View (the application) needs to be developed.
For the implementation of automatisms and other code, the database is also suitable as a place for storage and execution. Without any further data flow, the information within the CortexDB can be selected, read and modified to write new findings into existing or new data record containers.
Summary and perspective view
The CortexPlatform offers with the included applications and the CortexDB the possibility for agile use of your own data base. The CortexDB API and especially the UniplexAPI support this.
With the UniplexAPI the use of all configurations is possible. The implementation of individual applications based on the UniplexAPI is therefore quickly possible.
The structure of the CortexDB allows the use and selection of all information by the integrated index. This eliminates any adaptation to index structures. With the data sets (containers) and with the complete data model, you can work just as flexibly and agilely as in the rest of the project.
Furthermore, the CortexDB requires only low hardware requirements (low footprint). The operation in small environments (e.g. single board computers or microcontroller boards like Raspberry Pi etc.) is therefore possible.
Dr. Barry Devlin
“CortexDB Reinvents the Database” - http://bit.ly/CtxDB-9sight-1901
“Making Data Agile for Digital Business” - http://bit.ly/CtxDB-9sight-1902
“Managing Data on Behalf of Different Actors” - http://bit.ly/CtxDB-9sight-1903
“Distilling Deeper Truths from Dirty Data” - http://bit.ly/CtxDB-9sight-1904
“CortexDB Drives Agile Digital Transformation” - http://bit.ly/CtxDB-9sight_1905