As part of implementing services on AWS ECS for one of our customers, we recently had to build a highly available and has high performance shared storage system. Why build one when you already have Amazon EFS? Well, EFS is still not available in Mumbai region— thats why. :)
That leaves us with the option of doing things ourselves. We generally start looking at traditional NFS based methods. But unfortunately, NFS won’t cut it for the following reasons
- Lack of fail over capabilities
- Issues of data integrity (unless using the TCP protocol instead of UDP)
- Lack of ability to resize disk partitions readily
- Except for special implementations, such as under NetApp, not always the greatest performance.
This post explains how we can use GlusterFS to deploy a highly available and high performance shared external storage system for services that run on ECS clusters. Read on.
- GlusterFS is a distributed file system that can scale up to several petabytes and can handle thousands of clients. It clusters together storage building blocks over RDMA or TCP/IP, and aggregates disk and memory resources in order to manage data in a single global namespace.
- GlusterFS is a user space file system and hence uses FUSE (file system in user space) to hook itself with the virtual file system (VFS) layer. Gluster uses already tried and tested disk file systems like ext3, ext4, xfs, etc, as the underlying file system to store the data.
- For more information about Glusterfs please refer this link.
So How Do We Deploy?
The below high level arch diagram represents how the whole setup works.
Create an empty ECS cluster where we will create an EC2 instance explicitly and install ECS agent and register it with this empty ECS cluster.
- Goto AWS ECS cluster from the console ==> cerate Cluster
- Select EC2 Linux + Networking
- Give Cluster name, Check Create Empty cluster checkbox and create cluster.
Note: We can also create non-empty ECS cluster where ECS will create ECS node using Amazon linux ECS optimised AMI. In our case, we will be testing this setup using Ubuntu AMI.
- Launch Ubuntu EC2 instance as ECS node and install docker.
- Launch EC2 instance using Ubuntu AMI.
- Goto Launch instance.
- Select Ubuntu 16.04 from community AMIs (
- Launch the instance with required type, security group and key pair.
Install docker since its required to run ECS agent on node. Use following steps to install docker.
For more details about docker installation refer this.
Installing ECS agent
- Download tar using wget for agent image. Refer this page for ECS agent docker images as per your region and other information.
- Create ECS config file and add below content.
Create Gluster Servers
Launch Two ubuntu EC2 instances which will act as a Glusterfs servers.
Installing Glusterfs client on ECS node
Putting application code on Glusterfs volume
- Download a Dockerfile, index.js and packge.json file for node application from this link.
- Create a docker image using following Command
$docker build -t newimage.
- Create a AWS ECS Repository and push the image into repo. Refer this link AWS ECS Repository
- Put all application related file in
/mnt/glusterfs/app-1location on ECS host.
- Create a AWS ECS definition mount host
/mnt/glusterfs/app-1location to /app container location. Open port 80 for communication.
In above image of ECS job definition Container configuration /app is the path in container which will be mapped to Host path.
In below image image we can see that Host path which is being mapped for glusterfs volume is
/mnt/glusterfs/app-1 which is shared across all Glusterfs servers.
- Create ALB and ECS service
- Create Internet facing Application load balancer to attach ECS service.
- Create ECS service create Target group listening to 80 port forwarding traffic to it.
- Stack testing.
- Test Glusterfs failover by rebooting primary server and try to access the mount point on client. It should give connection within 2 seconds since we have set the ping value.
- Also test the Node application from Web for same scenario.
I hope you found this useful. Feel free to reach out, if you get stuck following the above steps. Happy containerization! :)