CQRS — Part 2: How it works?
Written by Christian Paesante
In the previous article we introduced the problems having a data model well suited for commands but not for queries and we introduced CQRS with its technical challenges and peculiarities from a theoretical perspective.
In this article we are going to deep down on its possible implementation and further optimization introducing a 2-Tier CQRS architecture. We are going to explore how it further simplifies the projection of events by still achieving good scalability.
Suppose you have propagate user information like its name or email with its balance which have to be displayed into the homepage of the app.
Suppose also the Anti-Money Laundering team needs to see information about customers wire-transfers used to buy BTC. Between the various information, it may be useful for them to see also user information that are the same projected for the user balance.
Most of the time you’ll find yourself building the same portion of a projection over and over again. From a software perspective, that means it would be extremely useful to encapsulate that logic in a sub-projection that can be reused from other main projections.
The problem is that, as we explained in the previous part, in order consumption of events can only be granted on events belonging to single entities.
For example, suppose you are nesting user information in a wire-transfer document under the payer field.
{
"wiretransfer_id": 1,
"payer": {
"user_id": 3,
"first_name": "Satoshi",
"last_name": "Nakamoto"
}
}But you can’t process the event bringing the user info (e.g. user_created) when the wiretransfer_created event has not been received yet.
That means there must be an order control determined not only by each single domain entity version in the same aggregate, but also by domain rules between different entities.
Introducing the hold-unlock logic
In order to grant the processing of events in a given order the system requires a way to backlog some events (hold) and resume them for processing when an arbitrary condition is met (unlock).
In order to achieve this, Conio uses a SQL table with the following schema:
Column | Type
-----------+--------
event_id | uuid
unlock_ids | text[]
event | jsonbWhen an event can’t be processed it is stored in the held events table with an unlock_id formatted with the entity type, the entity id and optionally the entity version of the future event that can unlock the processing of the current event. Namely the used format is domain_entity_name/<domain_id>[/<entity-version>]. Where [/<entity-version>] stands for an optional entity version.
When an event is successfully processed, it emits an unlock id with the (e.g. payer/1234567890). That unlock id is used to fetch all the events locked by this event and unlock their processing.
2-Tier CQRS
So far, we discussed how process events by granting to be processed in a meaningful order even when combining different entities in a single projected aggregate.
We also stated that most of the time you need to combine the logic of some sub-projections in order to create more complex ones.
Unfortunately, without a proper transactional environment provided by a RDBMS is really hard to combining event hold logics of different sub-projections in an atomic way.
Moreover very few DB offer the flexibility that a query on a RDBMS offers for combining different records from different sub-projections into a final view customized for your client’s APIs.
For this reason it is extremely helpful and useful that events are processed and projected on a PostgreSQL database. At the same time we don’t want to trade-off on the speed and load capabilities of a NoSQL document-oriented db. Another requirement is that for some data, it is required by our mobile app the possibility to receive push updates (as an example the balance update) in a GraphQL-like way.
Since we were able to project our data and query it in a very flexible way on SQL, we could further project aggregates in by joining all the data necessary and pushing it using a queue on ElasticSearch or Appsync (a GraphQL server fully managed by AWS).
On Appsync in particular, resolvers were dramatically simple due to the fact the final document was pushed complete with all its sub-fields, so no need to implement custom population. Indeed, we used DynamoDB as Appsync datasource, a blazing fast Key-Value store, since we didn’t needed particular query capabilities. Data was already there, ready to be fetched by its id.
If for some reasons we will need to query data with a different way, we can always implement a new SQL projection and spin up the most appropriate stack for the client for querying it, and push tailor-made records/documents/values to it.
Important notes
It may seem that we just created a fancy way to centralize data scattered across different micro-services, in order to be able to query it using SQL joins. Or another way for creating a materialized view for reporting purposes.
This is not the case. Projections are federated one with the other and each one consume only events they have the purpose to project. Moreover the data model is most of the times dramatically different from the one used for commands. Some tables are denormalized, some tables are dropped, new ones are created, new domain entities may originate depending on the context data is queried and hence projected.
More over we didn’t created a reporting database, because all the queries are operational real-time queries, used directly by our customers with their applications which exploits already aggregated data instead of making multiple calls to different services.
By creating a 2-Tier SQL-NoSQL CQRS system, we managed to create a super flexible query model without technology limits that process over 2M+ events per day.
We also saw how embracing CQRS allowed Conio to use a fully managed GraphQL solution provided by AWS AppSync with extremely very simple implementation of GraphQL resolvers and subscription notifications.
Thanks to Alberto Fanton
