Event sourcing using gRPC streams as Kafka replacement

Piers Wermbter
Haiilo
Published in
5 min readNov 19, 2020

Back in May, we started our long-awaited journey towards microservices that are self-contained and not dependent on synchronous calls to other services. To operate independently, the services will need a local representation of the domain data from other services.

Overriding message keys

One example is the i18n service from the first part of this series. It needs to know about all tenants and their URL to provide customer overridden message keys (as you can see above) mixed in with our default values to the frontend.

The concept

One concept to deal with this is EventSourcing. A producing service publishes state change events. These events reflect the made changes to the underlying domain data and can be subscribed to by consuming services that need a local representation of the data to be able to act on their own. Most implementations of the EventSourcing concept heavily rely on a central component. Apache Kafka is a popular choice for this.

While this is a very desirable target architecture for us, it is currently not implementable due to the operational model of our product here at COYO. Next to our SaaS offerings we also offer an on-premises solution that is installable on the customer's own metal or virtual machines. Our customers in that model range from ten to over ten-thousand users. Still, we would need a reliable Kafka component that persists the events without gaps or outages otherwise, services are not in sync. That being said, adding Kafka to the operational model would be a nightmare for most of our customers as it would raise the operational costs by a lot.

With Kafka ruled out we made our minds up about a decentralized solution, not relying on that one central component, and which better fits our restrictions. At the same time, it removes the need for one of our product teams to own and maintain the additional central component. One decentralized approach is to work with Atom feeds where every service publishes a feed of changes that others can subscribe to and replicate the data. Those feeds are backed by a persistence layer in the service itself. Each service needs to reimplement the refresh and event persistence logic itself and in an ideal world provide the feed in an aligned architecture-wide format. While we liked the overall approach writing Atom feeds felt kind of old-school. So we had a look at gRPC.

The gRPC “EventSourcing” contract

gRPC is a modern open source high performance RPC framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication.¹

gRPC is based on ProtoBuf and is build on contracts that each server or client can implement and rely on. It is based on HTTP/2 and therefore also supports server-side streaming. A perfect solution to build a subscription endpoint that delivers change events to all interested services. Our gRPC service definition looks like this:

[...]message EventSubscribeRequest {
repeated EventTypeSubscribeRequest types = 1;
bool includePastEvents = 2;
}
message EventTypeSubscribeRequest {
string typeName = 1;
int64 startingId = 2;
}
message EventResponse {
int64 id = 1;
string eventType = 2;
string entityType = 3;
string entityId = 4;
google.protobuf.Timestamp time = 5;
string tenantId = 6;
}
[...]service EventService {
rpc subscribeEvents(EventSubscribeRequest) returns (stream EventResponse) {};
rpc getEvent(StateRequest) returns (StateResponse) {};
}

A producing service will store change events to a certain entity type in its database with an incremental id. These events are then sent out to all-consuming services in the order they are raised. If such a connection is broken up, the consumer can reconnect with its last received event id and continue consumption from there on.

After receiving an event, it is up to the consuming service if it is interested in the current state of the domain data or not. We at COYO do not follow the principle of event sourcing thoroughly though, as we just emit the event without containing any change information, to allow for speedy event distribution. In case of interest, the consumer can get the current state of the entity through the second endpoint and will receive a JSON representation of the domain data which can be locally stored.

message StateRequest {
int64 id = 1;
}

message ErrorResponse {
string type = 1;
string message = 2;
}

message StateResponse {
google.protobuf.Struct content = 1;
string tenantId = 2;
ErrorResponse error = 3;
}

Out of these contracts, we generate libraries that can be used in NodeJS and Java to write client and server implementations of the eventing interfaces.

Pros and cons

We were able to build a version of the EventSourcing concept that does not rely on Kafka or any other central component, which is a huge benefit and necessity as mentioned earlier. Also having an “API contract” through the proto definitions made life easier. But the approach also has its downsides.

  • Consumers need to strictly follow the order of the events and can only consume one event at a time when they fetch the states manually in order. (Something we needed to accept as we wanted to keep the events as slim as possible due to the still existing monolithic main application.)
  • There is no historic state of the entities, just a stream of status changes in form of event names, and one can only fetch the current state of the entity.
  • We experienced some technical difficulties scaling the implementation to millions of events.
  • Every consumer and server need to reimplement the logic of consuming and providing events. (We are currently working on a spring library and a client to cope with this.)

Especially the second to the last point of the “contra” list is worth sharing in terms of pitfalls that can happen while putting gRPC streaming under the load of six million events both on the client (NodeJS) and server (Java) side. So stay tuned for the next part of this series.

[1] gRPC Authors. (2020). gRPC https://grpc.io/

--

--

Piers Wermbter
Haiilo
Writer for

Passionate about WebTechnologies, Cooking, Handball from Hamburg, Germany. Chapter Lead Backend Development @COYO Coach @NeueFische