Conio Engineering
Published in

Conio Engineering

CQRS — Part 1: What is it?

Written by Christian Paesante

Command-Query Responsibility Segregation is an architectural pattern that allows you to decouple your write model (used for Commands) from your read model (used for Queries).

The benefit of introducing CQRS in your architecture is twofold:

  • it improves maintainability, since it lets you shape your data to best suit your queries
  • it improves performances, since it enables an independent scaling of read and write sides of your system.

The problem

CQRS has become part of Conio’s Core Architecture. Between the various use cases one of the most interesting ones is the user balance within the app.

Conio Wallet Home Page
Conio Wallet Home Page

The user balance is the first and the most valuable information that must be shown to the user. For this reason, on one hand it should be as up-to-date as possible, but on the other hand it has to be cheap to be computed, like for example fetching a simple row on a db containing a user_id — balance info.

In a very simple system, user just buy and sell Bitcoins. When they buy, they receive an incoming blockchain transaction from Conio’s Bitcoin deposit wallet. When they sell, they send an outgoing blockchain transaction to Conio’s Bitcoin deposit wallet.

But since our wallet is a fully functional wallet, users can also send and receive bitcoins with other Conio’s users or addresses not known from Conio, existing on the bitcoin network.

Given that, we need to monitor all transactions occurring on the bitcoin blockchain in real-time in order to detect any transaction involving any of the addresses belonging to our Conio customers. The service responsible for this is called “Crawler”.

The Crawler task is to scan each new blockchain’s block by looking for any transaction whose inputs/outputs match with any of Conio customers addresses and store those transactions in a table; all of this while dealing with blockchain transient forks.

Therefore, since the Crawler has to manage a pretty heavy workload, it is not feasible for it to also transactionally update customer balances. The only possibility left was to compute the users balances by aggregating customers incoming and outgoing transactions. That works fine with 10 users, but not with 10k users in real-time.

Another option could be to asynchronously compute the user balance. But since we already have to do some asynchronous computation, why not providing all the users information already denormalized and fitted for the required query?

Another example where our data model wasn’t fit for the query requirements was the user activities. The user activities are a list of all user transactions marked by type of operation like Bid, Ask, Buy, Sell, Withdrawal, Receive.

An activity is not a domain entity tracked in some micro-service, but instead it is an entity that exists only in the query side that includes data from several other sources. More precisely, in order to build an activity, we need to fetch a list of records from different micro-services and join them in memory in the Backend for Frontend layer. That has two drawbacks:

  • Too much domain information in the BFF layer
  • Too much costly for users with a modest amount of movements

Introducing CQRS

With CQRS we splitted our system in two parts:

  • an online system, consisting of a storm of micro-services mainly used for commands and very simple queries
  • a projections system, used for complex aggregation queries like the user balance or queries that requires lots of joined and nested data

Changes in the online system are asynchronously propagated to the projections system with a queue (a Kafka topic). The main challenge is to guarantee that:

  • Changes are published atomically with the write operations performed in the online system
  • Changes are projected/consumed in the right order

Atomic change publication

Probably, the most powerful pattern for this is Event Sourcing, by using a proper Event Store, where every change over a domain entity is stored as an event. But this requires you to start the development of your system/micro-service adopting this pattern from the beginning.

Photo by Taylor Peake on Unsplash

In Conio, at the time we realized we needed CQRS, we were already serving some customers and it would have required to re-develop most of the systems and making a data migration from a RDBMS database (PostgreSQL) to an Event Store. This would have required an immediate upfront event design for making a proper mapping to the new representation. A part from being really time consuming, it’s really error prone (remember, events are immutable!).
We decided to adopt a hybrid solution instead.

Whenever a change on a record occurs, an event describing what happened is stored within the same SQL transaction on another table designed for publishing events.

An asynchronous publisher will eventually fetch all the events backlogged in the table and publish on whatever queue you want to use, in our case a Kafka Topic with infinite retention.

In our case Kafka is our actual Event Store where published messages (events) published are immutable and are stored forever. Whenever in the future we may need to replay all the events, we will just reset the offset of the consumer group attached to it.

In this way the online system was able to continue to operate with minimum incremental modifications.

In order change consumer

When events are published on Kafka, they can be published out of order and eventually be delivered out of order. Although this can be somehow mitigated, using a multi-threaded asynchronous publisher as described, you don’t have that strict guarantee on the producer side.

Photo by Brett Jordan on Unsplash

To deal with it, the consumer must be aware of which events have to be consumed in-order and which not. A global in-order consumption of events is not possible due to concurrent producers from different micro-services. Hence only a “local” ordered consumption can be granted.

Generally is just sufficient to guarantee that events belonging to the same domain entity are consumed in-order. This means that by keeping a versioning of a domain entity, it is possible to project events using an optimistic locking by propagating the new entity version in the event.

A more complex case is where you need to process events coming from different domain entities that will constitute an aggregate in the projections system. In that case specific domain conditions on the aggregate state and the incoming event have to be applied.

In this article we explored how CQRS may help in certain situations to overcome technical difficulties in providing the most optimal query data representation. We also explored on a theoretical level which precautions have to be taken in order for the system to work properly.

In the next article we are going to explore how Conio implemented CQRS using a 2-tier CQRS architecture and how that allowed for maximum flexibility on technology and speed of development.

--

--

--

Recommended from Medium

Decrease Elasticsearch memory consumption on MacOS

Road TO Dev — BOOM! Let’s visualize our enemy exploding!

Common Data Wrangling Operations in Python/Pandas (Based on My Machine Learning Projects)

Understanding IL code , JIT, CLR

Kubernetes 120: Networking Basics

ASP.NET Core Repository,GenericRepository,Unit of Work pattern

Sin7Y Tech Review (12): A Guide to AppliedZKP zkEVM Circuit Code

Get Worldwide Airport Codes With An API

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Conio Team

Conio Team

La voce di Conio sulle più importati news del mondo cripto.

More from Medium

How the Tinyman Exploit can Strengthen Algorand

All Been Crypto — Week 10 Dec 2021

Build Mobile-Focused Celo dApps With Ankr Protocol!

Decentralized Finance 101 (Clemson X Encode)