Finagle Compatible Service Discovery in Golang

What we all know and love about Strava is running on a mostly Rails code base. However, as we’ve grown we’ve slowly been moving to a more service-oriented architecture. Almost all new features are being implemented as their own service and a few existing features, like the feed, have also been migrated. Most of these services are written in Scala and use Finagle/Thrift for their RPC communication layer. A growing number of services are starting to use Go and it’s becoming necessary to cross communicate. A big hurdle in this process has been service discovery, i.e. how clients determine the IP and port for a service that can exist on multiple hosts.

We store the hosts and ports of all services in Zookeeper, a distributed configuration and synchronization service which provides a safe place to put information and an easy way to be notified of changes.

Service discovery is a standard Zookeeper recipe. The service is assigned a Zookeeper zNode, e.g. /discovery/prod/routemaster. Servers register themselves as ephemeral, sequential child nodes containing data about their host and port. Ephemeral so they automatically disappear if the server dies, and sequential so the name of each child node is unique and there are no conflicts. Clients fetch the children of the /discovery/prod/routemaster zNode and watching for changes. Zookeeper handles the details of consistency and synchronization in a pretty elegant way.

We’ve encapsulated this functionality into a golang package called go.serversets. Here’s how it works:

Start by defining the server set:

zookeepers := []string{"zk01.internal", "zk02.internal", "zk03.internal"} serverSet := serversets.New( serversets.Production, "routemaster", zookeepers)

Servers register themselves as an endpoint:

endpoint, err := serverSet.RegisterEndpoint(localIP, servicePort, nil)

Clients simply watch the list of endpoints:

watch, err = serverSet.Watch() endpoints := watch.Endpoints() for { <-watch.Event() endpoints = watch.Endpoints() // endpoint list changed }

The watch.Event() channel will be triggered whenever the endpoint list changes and watch.Endpoints() will contain the updated list of available endpoints. For more details on this see the package readme.

The list of endpoints is only semi-useful as you need a way to load balance over them. So there a couple sub-packages that wrap the endpoint list for common use cases:

• mcset provides consistent hashing over a set of memcache hosts.
• httpset round-robins standard HTTP requests to the set of hosts.

Finagle Compatibility
The Zookeeper zNode data is designed to be compatible with Finagle ServerSets. It is just a matter or matching the namespaces in the ServerSet declarations. This library registers endpoints to zNodes similar to /discovery/prod/routemaster/member_0000000318. For more details see the package readme.

Open Source
The go.serversets package is available on Github and bug reports and enhancements are greatly appreciated. While you’re taking a look, checkout our other libraries such as go.statsd for logging statsd events and go.strava which wraps the Strava V3 API.

Originally published at labs.strava.com by Paul Mach.