by Scott Morris
Here at Opsline, we’ve been busy deploying containerized client applications onto ECS. However, some of these applications need a service discovery layer so they can find each other and communicate. AWS has now announced its new Route53 Auto Naming API for ECS service discovery, but previous to this announcement, we needed to look for a service discovery solution and settled on Consul for our ECS application stacks.
Consul in Client Mode
Running Consul in “client” mode on ECS is fairly straightforward. The recommended architecture is to run a single Consul client container per ECS instance, on the Docker host network, ensuring all application containers on the ECS instance can reach Consul via http://[ECS instance ip]:8500. Achieving this one-Consul-client-per-instance architecture is simply done by launching Consul client ECS tasks via a CloudFormation Init command embedded in the ECS cluster nodes’ user data.
Consul in Server Mode
A quick Google search might suggest you should run your Consul “server” containers on a separate stack of EC2 instances in an Autoscaling Group. Amazon’s own suggested reference architecture does just this by launching the Consul Server container with docker run in each instance’s CloudFormation Init list of commands to run at instance launch time (see https://github.com/awslabs/service-discovery-ecs-consul).
However, we wanted to run Consul Server on ECS itself. At first glance, it might seem logical to run your Consul Server tasks on the same ECS cluster as your Consul Client tasks. After several attempts to do so, it became clear it’s much easier to run your Consul Server tasks on a separate ECS cluster.
So, we decided to reuse our ECS cluster CloudFormation template and launch another ECS cluster dedicated solely to running Consul Server tasks as an ECS Service. In this configuration, we can lock down the number of ECS cluster instances and the number of desired Consul Server running tasks in the Consul Server ECS Service definition.
Compared to running Consul Server on a cluster of regular EC2 instances, the cost is the same (ECS only costs what the underlying EC2 instances cost), and you get the service management features of an ECS Service along with the benefit of reusing the same ECS Service template pattern. The Consul Clients running on the ECS application cluster join the Consul Server cluster via the recommended--retry-join-ec2 option, which we configure with the stack name of the Consul Server ECS CloudFormation stack. The end result is a scalable ECS application cluster always running one Consul task in “client” mode and a non-scaling ECS cluster dedicated and sized for only running a Consul ECS Service in “server” mode.
We’re hiring! Want to come and help us solve some of the complex problems in the DevOps space? Drop us a line at work@opsline.com with your resume and we’ll be in touch.
