Event Sourcing in PHP

An example naïve implementation using Redis for presistence

I’ve been fascinated with CQRS/Event Sourcing for a while now. I love the entire approach, from modeling your domain after events and aggregates, to creating complex state representations using projections of those events. I’m still figuring out some stuff though, like eventual consistency. I’m just too afraid of that to use it in production yet. It’s not that I don’t trust the pattern: I just don’t trust my little experience with it.

But as an exercise to learn the whole of the pattern, is that I decided to build a full CQRS/EventSourcing library in PHP by myself. (I don’t know if this library will ever be production-ready, but who knows!) Of course, I’ve looked at Prooph components and even implementations in other languages, like C# and tried to take the best of each one of them.

Choosing Redis. A why.

One thing where I put a lot of thought into was choosing an adequate persistence layer for my Event Stream. I never was really that comfortable with SQL, because it is too meh: does not suck at anything (reads/writes), but does not excel at anything either. That’s when I took a quick look to Redis. Until that point, Redis was just an efficient cache system for me. But then I discovered the data structures it supports and also the myriad of different operations it supports on those data structures. Documentation does an excellent job in explaining the cost of each one of them. Also, it has transactional support, so I could persist events and it’s data atomically.

A big part of Event Sourcing is to be able to publish those events to workers that either, perform one time actions with them (listeners/processors) or generate complex forms of state for reporting or views (projectors). Redis has built-in pub/sub capabilities as well, so seemed a good fit for this use case too.

Lastly, clustering and sharding solutions for Redis are extremely common, so I can rest assured my storage is going to scale horizontally if it needs to.

An Example Implementation

This is not the final Api that I will use, but the Event Store interface looks like this:

There’re some things I don’t like, like the metadata being a key value pair: it’s too leaky, so I’m thinking that streams should have their metadata explicitly persisted and provide a method to query it. Prooph does this and I’m currently working on it.

The Redis implementation stores streams in Redis Lists. The list provides the id of the event. The key is composed with this structure: [app-prefix]:stream:[stream-name] . The only stream created by default when someone persists an event is the all stream. This is a list containing all the event ids in order of creation. But also, every aggregate contains it’s own stream, as well as every event type and every aggregate type. Then, the event data is stored in Redis hashes, that are the modulo 655360 of the crc32 hash. This is for perfoming reading operations fast, as well as for spreading the data among clusters evenly.

Notice the RedisEventIterator class. It only contains an array with the id of the events. Each event is fetched on iteration only using a PHP Generator and serialized only then.

The results are satisfactory. Reading operations using the crc32 pad are extremely fast (O(1)). When iterating, it’s a matter of nano seconds to fetch the event and deserialize it. The parameters start and size allows us to query the stream in the order we want. We could make this RedisEventIterator paginatable from the client side and more memory efficient if we include the lrange call inside of it.

This is how it looks in Redis:

It’s trivial to reconstitute an aggregate now or replay the whole stream of events. It’s also really easy to dump the stream in a REST call.

There’s still a whole lot of room for improvement, like storing stream metadata and deferring pagination for the client code in the EventIterator. Also, like Prooph, I would like this implementation to have metadata enricher for events. When all this is done and ready, I think this would be a pretty solid and powerful event store.

Feedback

What are your experiences with Event Sourcing? What is your favorite persistence engine to store your Event Stream? How have you dealt with its gotchas? I would really like to know!