Using AWS ECS Service Discovery for communication between Microservices in .NET & C#

AWS ECS Services allows us to build reliable and fault tolerant Microservices. In this story we will see how to create an AWS ECS Service (an Aggregator Service), which internally talks to two other services using AWS ECS Service Discovery. And, we will also create all the ECS Services programmatically using the ECS SDK.

First some backdrop on AWS ECS Services.

AWS ECS Tasks allow us to run dockerized applications within AWS Cloud. However for building reliable applications & Microservices, we need to be able to monitor our tasks health and load, and ensure minimum and enough instances of the task are running based on application requirements. AWS ECS Service does exactly this. Read more about AWS ECS Services and how they fit into overall ECS architecture from below link…

What is Amazon Elastic Container Service?

AWS ECS also comes with Service Discovery, which lets tasks communicate with each other using simple service names, irrespective of where they are running and how many of them are running. This is essential for Microservices architecture where Microservices have to talk to each other. One can use ELB/ALB for communication between the services, but it quickly becomes prohibitively expensive. Here is a good article on Service Discovery and how it helps in building reliable and cost effective inter task/service communication within your AWS infrastructure.

Amazon ECS Service Discovery | Amazon Web Services

It is recommended to read my other related stories on pushing a .NET Web API Docker image to AWS ECR Repository programmatically, creating a ECS Task definition out of the AWS ECR repository and running the ECS Task programmatically, for more detailed information on a few steps that we will cover in this story. Here are the related story links…

Pushing a local docker image to Amazon ECR Repository using .NET 5, C# and AWS ECR SDK

Create and Run AWS ECS Task programmatically using .NET 5, C# & AWS ECS SDK

Now let us get back to our story. We will simulate a small book store scenario, where we will have a Dashboard microservice that fetches a user’s book purchases and book recommendations in a single service call. Purchases are managed by a Purchases microservice, while recommendations are managed by a Recommendations microservice. Dashboard service essentially acts as an aggregator of the information from the purchases and recommendation services.

Note: In this sample, I’ll not focus on dependency injection and other aspects of a good microservice design. This sample is about using ECS Service Discovery to communicate between microservices. In the process we will cover how to efficiently call into multiple microservices using .NET TPL (Task.WhenAll) and async/await.

Here are the steps that we will follow to achieve the inter-microservices communication using ECS Service Discovery.

1. Define a AWS Cloud Map Namespace, which allows for discovery of services within a VPC.
2. Create a Purchases microservice using .NET ASP.NET Core Web API framework. Expose a simple HTTP GET endpoint in the Purchases microservice to return the list of book purchase objects.
3. Create a AWS ECS Service for the Purchases microservice. Attach this service definition to the Cloud Map Namespace created in Step-1.
4. Create a Recommendations microservice using .NET ASP.NET Core Web API framework. Expose a simple HTTP GET endpoint in the Recommendations microservice to return the list of book recommendation objects.
5. Create a AWS ECS Service for the Recommendations microservice. Attach this service definition to the Cloud Map Namespace created in Step-1.
6. Create a Dashboard microservice using .NET ASP.NET Core Web API framework. Expose a simple HTTP GET endpoint in the Dashboard microservice to return a Dashboard object that has the list of purchases and list of recommendations fetched from the Purchases and Recommendations microservices.
7. Clean up the AWS resources.

The working sample code (Visual Studio 2019 Solution) is available on the GitHub. Below is the link.

https://github.com/apasumarthi1999/AWSECSServiceDotNet

You should have an AWS account with ECS cluster created with a VPC and appropriate security groups and subnets that can be accessed through Internet (to test our Web API). You also need an IAM user with programmatic access through Api Key and Secret Key with permissions to manage AWS ECS services and AWS CloudWatch Logs of your AWS account.

You have to put your AWS related Api & Secret Keys and other settings in the appsettings.json file in the console application project, based on your AWS account configuration.

Step-1: Define a AWS Cloud Map Namespace, which allows for discovery of services within a VPC.

Creating an AWS Cloud Map Namespace is easy using AWS Console. Just go to the Cloud Map service and create a namespace, which can be used for “API calls and DNS queries in VPCs”. Below is a sample screenshot.

Once the namespace is created, note the namespace id and put it in the “ServiceDiscoveryNamespaceId” setting in the appsettings.json file of the .NET console project in the solution.

Steps-2 & 3

Create a Purchases microservice using .NET ASP.NET Core Web API framework. Expose a simple HTTP GET endpoint in the Purchases microservice to return the list of book purchase objects.

Create a AWS ECS Service for the Purchases microservice. Attach this service definition to the Cloud Map Namespace created in Step-1.

Check out the PurchasesApi project in the sample solution for the Purchases microservice and the endpoint it exposes. Run the Docker file in the project, which creates a docker image of the Purchases microservice. This docker image is then used to create the AWS ECS Service for the Purchases microservice.

Steps-4 & 5

Create a Recommendations microservice using .NET ASP.NET Core Web API framework. Expose a simple HTTP GET endpoint in the Recommendations microservice to return the list of book recommendation objects.

Create a AWS ECS Service for the Recommendations microservice. Attach this service definition to the Cloud Map Namespace created in Step-1.

Check out the RecommendationsApi project in the sample solution for the Recommendations microservice and the endpoint it exposes. Run the Docker file in the project, which creates a docker image of the Recommendations microservice. This docker image is then used to create the AWS ECS Service for the Recommendations microservice.

Below is the code snippet from this sample’s .NET console project that creates the Purchases and Recommendations microservices and associate them with the Cloud Map namespace for service discovery.

Step-6

Create a Dashboard microservice using .NET ASP.NET Core Web API framework. Expose a simple HTTP GET endpoint in the Dashboard microservice to return a Dashboard object that has the list of purchases and list of recommendations fetched from the Purchases and Recommendations microservices.

The same method as shown in the above code snippet is used to create the Dashboard service, but with the ability to get the public IP of the service task, so that we can test the dashboard endpoint.

Below code snippet from the .NET console project of the sample solution demonstrates the usage of the above method for creating different services discussed in this story.

Finally we call the HTTP GET endpoint of the Dashboard service to fetch the purchases and recommendations data. Below is the code snippet from the .NET console project that does this.

The Dashboard service endpoint will then call into the Purchases and Recommendations microservices endpoints to gather the required information. Below is the implementation of the Dashboard service endpoint. As you can observe in the below code snippet, the dashboard service’s GetDashboardmethod calls into the Purchases and Recommendations microservices using the service name (purchases-svc, recommendations-svc) that we assigned to these services during ECS service creation along with the Cloud Map namespace name (ecom) that we created. This way we get a nice DNS like service endpoint access, without actually owning any domain names and using relatively costlier ELB/ALB solutions.

There is a small technique that is used in the above code snippet to call into the microservices. We used Task.WhenAll to await on the microservices tasks, thus allowing the two microservices to run in parallel and return results simultaneously. This improves the performance of the Dashboard service, while still getting the scalability benefits of using async/await.

Step-7

Clean up the AWS resources.

This step has to be done manually after execution of this sample. So please carefully look at the AWS resources created in your account and clean them up accordingly. Below are the resources that would be created as part of running this sample.

1. Cloud Map Namespace
2. Three ECS Services (DashboardApi, PurchasesApi, RecommendationsApi)
3. Three ECS Task Definitions
4. Three ECS Tasks (running)
5. Service Discovery entries that are created while creating the ECS Service definitions
6. Other resources like VPC, Subnets, Security Groups that you would have manually created for testing this sample

Voila, we were able to create and test out inter-microservices communication using AWS ECS Services and Service Discovery.

Happy AWS coding!