Kafka Stream Processing — Handling Deletions

With stream-table duality, deletions are usually represented by events with a Null value. For an example, think of the domain of a company. It’ll have entities representing, Departments, Employees, Projects etc. An Employee is always assigned to a Department and thus the Department entity id would be a foreign key to the Employee entity. Deletion of an Employee would be noted by an event with the employee_id as the key and a Null value, published to an Employee entity topic (employee_entity). This is all good and well for Kafka stream applications that works only on a single entity type (Figure-1).

Figure-1 — Basic Kafka Stream application components for managing Employee events

But handling deletions is not that straight forward when the stream application relies on more than one related entities.
For an example think of a stream application that executes some business logic based on Employee and Department entities (Event of Department entity would be published to the topic, department_entity).
For that it’ll consume both Employee and Department entity topics, executes the logic and publishes the result to a different external output topic.
Now the key point in designing such a stream application is to make sure that all related events are sent to the same partition. Otherwise we’ll have to rely on interactive queries which would introduce unnecessary latencies.

Following is a basic topology for such an application assuming employee-entity topic to have 20 partitions and department_entity topic to have 10 partitions.

Figure-2 — Basic topology of a stream application that consumes multiple entity types that are related. The re-partitioning stage makes sure that related entity events are put into the same partition index.

As it’s shown, we first publish employee_entity topic stream to an internal topic re-partitioned by the department_id. The department_entity topic is already partitioned by it’s id. Then we can have any number of stream processing nodes consuming the internal re-partitioned employee_entity and the department_entity topics.

Because of the re-partitioning, we can assure that related Employee and Department entity events are put into the same partition index of their respective topics.

Now back to deletion events. Since the value of a deletion event is Null, how come the re-partitioning by department_id performed ?
For that I’ve tried two approaches. Both options use composite keys. But the first option uses composite keys in external topics where as the second approach uses composite keys in private topics specific to the stream application.

Using Composite Keys In External Topics

Instead of using the employee_id as the key, we can use a composite key <employee_id, department_id>. With that, any re-partitioning of Employee events by the department_id can be accommodated. But the option should be used with caution.
Firstly, there could be many services consuming Employee events and some may not be aware of the Department. This is where DDD (Domain Driven Design) comes in handy. In our example, core domain services would be aware of Department and Employee and thus won’t be of an issue. Nevertheless there could be services outside of the core domain which may only be aware of some aspect of the Employee. In situations like that we’ll have to do a context mapping by remapping the event key from <employee_id, department_id> to <employee_id>.
Second concern is that sometimes the composite key may contain much more than just two keys. If that’s the case, it’s better to go with the next option.

Using Composite Keys In Private Topics

This is more complex. But the major plus point over the first approach is that this won’t pollute keys of entity events with foreign keys. Also with the first approach, we always have to consciously check where context mapping is needed, just for event keys.
Following diagram shows the basic topology of using composite keys in private topics.

Figure-3 — Topology of a stream app using composite keys in internal topics to help with re-partitioning.

Let’s breakdown the processes.

1. employee_entity topic contains events related to Employee entity. Event key is employee_id.
2. department_entity topic contains events related to Department entity. Event key is department_id.
3. The first stream processor keeps track of all Employee related events. It’s got a Kafka state store which stores Employee event values against the employee_id.
   - If the event is a create/update then the state store is updated.
   - Then creates a new composite key with the employee_id taken from the original event and the department_id taken from the stored Employee entity.
   - If the event indicates a deletion, then the record is retrieved and removed from the state store. Then creates a composite key with the employee_id taken from the original event and the department_id taken from the stored Employee entity.
   - Once after processing create/update/delete events, a new event is emitted with the new composite key.
4. The re-partitioning phase re-partitions events based on the department_id and publishes to another internal topic with the same number of partitions as department_entity topic.
5. Once that’s done the real business logic of the stream application can be commenced which may involve both Employee and Department events.

Once again the major advantage of this approach is that we can keep the entity topics clean and thus no need to worry about context boundaries when it comes to event keys.

Having said that, we may opt out of using Null values to indicate entity deletions altogether. Instead can use the complete entity payload along with an event type to indicate the deletion. But this violates the stream-table duality.
So which approach to take is up to the development team, but for me, I always like to utilize what Kafka provides us to the fullest and thus would like to minimize interactive queries and take advantage of the partitioning. Always keep in mind that the partitioning key doesn’t have to be the event key.