Cloud Native Pancakes

Hi all. I haven’t blogged for a while because my hobby project was a bit too repetitive. Now that the bulky work has been done, I have a new interesting subject coming up, which I want to introduce. The situation is, that I have been writing a “translator” to transform the data of a music composition tool for children from a relational structure into an event-based structure.

In this blog post I just want to briefly explain the difference and then I want to point out what excites me so much about these subjects, and finally I want to show you the events that I designed for my own system. I hope you may have some interesting books or articles for me to read as an advice.

how data can be stored

Let’s start with the world how we see it: things have relations to each other. Things “contain” things or “consist” of smaller things. The whole world is a list of things which have lists of things. Let’s take fictitious data as an example, as it could exist in a running program. This program apparently processes data about cooks who bake pancakes for guests.

How you want to store the data depends on many considerations. A typical example of a relational way of storing the data:

Note that the pancake is (apart from a physical pancake) a triangular relationship between a cook, a recipe and a guest. Also note that in the above relational storage, all data is stored only once. The name of a guest, cook or recipe is only one time in the data. The relations between the data are represented by numbers. The example for this is the TABLE PANCAKES.

Now, let’s see how it looks were we to store the data in the form of “events”.

some highlights

To put it simply, with relational data, the data is always in one place, in the database. It’s a simple model to understand. You can start out when you know just a few basic rules. Frameworks will guide you to store the data and retrieve it in your program. The cost to start is low. The design is simple. Along the way this type of database will typically become ugly, costly, a performance problem, something that requires a lot of other things to go down when it itself goes down (for maintenance for example) and totally and very hard to get rid of because of all the relations, all the coupling between the data, and coupling with the frameworks. There are real limits to the scaling out that can be done with this sort of data. The database becomes slow. It’s not so easy to change course.

With non-relational data, you will have the same data in many places. It is hard to think about the relations between the data and how to keep the data compatible with each other when you have them in multiple places. It requires some classical information design skills. But when you have a good basic plan, your possibilities are really unlimited. You can have collections of data all around the world and replay them to transform them as your software evolves. When you read some books about this type of data, you can start with something that will continue to be challenging and it can grow in quantity. But its problems and complexity will not necessarily grow exponentially with the size of it all. It can be feasible and nice and cheap to scale and scale up forever and keep performing well and keep everyone happy. The only limits are your brain. A long time from now it will still be cheap and work very fast.

five things about event storage that I find exciting

• real time streams: with events it becomes possible to store all the things that happen in an application in real time, at the moment when the events really happen.
• collaboration: to share this stream with multiple users at the same time, so that multiple people can work on the same document or music piece or drawing.
• do/undo functionality: I would like events to be suitable for undoing them, to be able to rewind in time and go forward again when desired.
• capturing apart from the result, more interestingly: how people get to a result, and see how computer aided education can benefit from this recognition.
• node equivalence / peer to peer, so that there is in terms of protocol and behavior no real difference between what you would traditionally regard a client and a central server in the data center.

the events that I designed

They are not yet finished but it’s a start, and I made sure to cover everything that is in the current system. Here is the source and here are the tests.