Service Discovery, Registrator, & My IP Expectations & Reality

I came across a use case when playing around with SDN that stymied me a bit. When I have a container deployment where some containers belonging to managed services are on a bridge network and other containers for applications are on an overlay network, how do I tell my service discovery tool which one to resolve?

The big problem: How do you control which IP address and port are the “actual” one for a service?

There are many different service discovery tools (or tools with service registration built-in) out there (Nomad, Netflix OSS suite, etc.). There is a great post from NGINX that explains some of the concepts. There are two components to service discovery, discovery and registration. For discovery, there are two patterns:

1. Client-side discovery uses a client that registers the service instances to a central service registry.
2. Server-side discovery uses a load balancer to direct queries for a service from other services, using a service registry.

For registration, there are two patterns:

1. Self-registration
2. Third-party registration

So really, my big problem above wasn’t necessarily about discovery.

My big problem was addressing registration.

If my services don’t register with the right IP address, then other services can’t find it. To play around with a solution, I chose a service registration tool to examine. Registrator is a service discovery tool that grabs the IP addresses of containers and puts them into a backend store. I attempted to reverse-engineer my expected IP addresses, depending on the configuration. It turns out that this is not something that I could easily intuit, and I lost track of what it was really supposed to be. On top of it, if I am going to solve my big problem, I need to tell Registrator which network to register. I made a few edits leveraging some existing Registrator constructs and created a “SERVICE_NETWORK” option that you can pass to the container in similar fashion to “SERVICE_NAME” constructs. The behavior is outlined below.

Does it work? Yes, I can declare the resolution of one port be on one network and the resolution of another to a different network . The implication is that I can have two interfaces in the container for different functions, such as one with overlay networking and another with bridge networking. If one interface has a problem, I can failover to the other one, thanks to my handy-dandy service discovery component. However, I realized a crucial gap.

I have to specify the container network. I would rather not do that…

What if the network I want doesn’t exist or the container isn’t attached to it? Well, I’ve effectively turned my container into a service discovery orphan. Is there a better way to do this? Perhaps. I want some form of intent that is non-specific to containers. There are a lot of considerations for this:

• Is my service globally or locally accessible?
• Does my service has several management functions? (This requires multiple ports. Something like Zookeeper.)
• Are these management functions globally or locally accessible?

So really, what if I could tag my Zookeeper deployment as something like:

• Peer Port: SERVICE_2888_TYPE=management, SERVICE_2888_SCOPE=local
• Leader Port: SERVICE_3888_TYPE=management, SERVICE_3888_SCOPE=local
• Client Port: SERVICE_2181_TYPE=client, SERVICE_2181_SCOPE=global

The idea is that any network (whether it be routed or NAT) that I deem as “globally accessible” should be the one that is in use for my SERVICE_X_SCOPE=global. Any service that is a management service may have different rulesets on who or how it can be discovered. I can effectively swap out the networks on these interfaces without concern, which makes me far less uneasy.

Overall, I think I have to ponder more about this space. The workaround above sort of gets me what I want but I’d love a better way of abstracting it. Let me know if you have recommendations!

References

1. My Registrator Fork (rebased on master, not v7!)
2. My Service_Network Docker Image