Introducing Eventuous Connector
What is Eventuous Connector and how to use it
Eventuous connectors allow subscribing to events from EventStoreDB (source) and produce derived events to another system (sink). Some sinks also support reduce-like operations that work as projections.
The Connector is a combination of a real-time subscription (currently only to
$all stream) and a sink. A sink can be configured to produce (producer mode) or to project (projector mode) events. Some sinks support both modes and some only support one.
The benefit of using a ready-made connector over building your own as the Eventuous Connector has quite a few production-grade features built-in:
- Observability that includes logs, metrics and traces
- Checkpoint management
- Fault tolerance such as retries and health-checks
Currently, the following sinks are supported:
- Elasticsearch: producer, projector
- MongoDB: projector
- MS SQL Server or Azure SQL: projector
Below, you can find a description of the different modes that are supported by the connectors.
In producer mode, the connector subscribes to the source and produces events to the sink. How events will look like on the other side depends on the sink implementation.
The producer mode is most useful for the purpose of re-publishing domain events to a broker or to be used as an archive or backup. For example, the Elasticsearch sink can be used in combination with the archive event store to keep archived events in a cheaper persistence tier, whilst keeping the EventStoreDB database size contained.
When used in combination with a message bus sink, the connector can be used to publish integration events for other services to consume. However, right now the connector lacks the transformation capabilities, so we don’t recommend using it in this way when the connector is deployed as-is. However, you can still build a custom connector for that purpose.
The projector mode allows implementing projections that are built using other stacks than .NET. Basically, the projector mode implements the MapReduce pattern. Using a projector you can reduce events to a single piece of state using some unique key.
Because the reduce function requires custom implementation, Eventuous Connector needs to call some custom code that will get events and send the reduce function back. That custom code can be completely stateless and, therefore, can execute in parallel, although the Connector will maintain ordered event processing.
The only mode that the connector is able to execute such custom code at this moment is by using an external gRPC service with bidirectional streaming. The connector will send events to the gRPC service and the gRPC service will send the reduce function back. Each projector sink implementation requires using its own reduce functions set. With this model, the external gRPC service implements a gRPC server, and the connector will connect to it as a client.
A custom gRPC service can be built in any language or stack that supports gRPC. You’d normally deploy it as a sidecar for the Connector pod in Kubernetes or a serverless workload.
For example, a MongoDB projector will expect to get operations line
UpdateOne back from the gRPC service, and SQL Server projector uses a single operation that returns an arbitrary SQL statement based on event data.
The Connector's role here is to maintain the subscription to EventStoreDB, send events to the gRPC service, receive the reduce function back, execute the response, and maintain the checkpoint. By convention, each sink uses its database for checkpointing.
As a result, it’s possible to build a stack-agnostic stateless projector and use the Connector to do the heavy lifting. Each sink also provides observability instrumentation for the database client library it uses.
I will provide examples for different connectors as follow-up posts.
Originally published at https://eventuous.dev.