Docker Data persistence using AWS EFS service.

Everybody in this world knows the importance of Database for any application.Your database should be available and accessible every-time .

Because containers are likely to be short lived and are even more conceptually ephemeral than virtual machines, you should never store data or logs in a container. Instead, store data and logs by leveraging Docker’s volume mounts to create either a data volume or a data volume container that can be used and shared by other containers.

In April 2015 AWS announced a new cloud service: Elastic File System — a Shared File Storage for Amazon EC2.

What is Elastic File System ?…..

Simple, scalable, and reliable file storage for the AWS Cloud

Amazon Elastic File System (Amazon EFS) provides simple, scalable file storage for use with Amazon EC2 instances in the AWS Cloud. Amazon EFS is easy to use and offers a simple interface that allows you to create and configure file systems quickly and easily. With Amazon EFS, storage capacity is elastic, growing and shrinking automatically as you add and remove files, so your applications have the storage they need, when they need it.

Multiple Amazon EC2 instances can access an Amazon EFS file system at the same time, allowing Amazon EFS to provide a common data source for workloads and applications running on more than one Amazon EC2 instance.

It’s designed for high availability and durability, and provides performance for a broad spectrum of workloads and applications, including Big Data and analytics, media processing workflows, content management, web serving, and home directories.

Elastic File System Features

  • Elastic File System will be simple and scalable.
  • Designed to use with AWS EC2 Instances.
  • Sharable across multiple ec2 instances.
  • Storage capacity (and cost) is automatically scaled up or down as you add or remove files.
  • Like most AWS services, you pay only for what you use.
  • Elastic File System files are stored across multiple Availability Zones within a region.
  • Amazon VPC security groups and network access control lists allow you to control network access to your EFS resources.
  • The cost of storage is based on the average monthly storage space used, at a rate of $0.30/GB-month (about twice the charge for a standard EBS volume).
  • Can be used seamlessly with database instances as storage — throughput and IOPS are scaled accordingly.

Setting up and accessing the Elastic File System from EC2 instances:

  1. Create a file system.
  2. Create a mount target.
  3. Enable an NFS client on your instances.
  4. Using standard mount commands, the remote file system will appear on your ec2 instances as a local partition.

Let’s give it a try and run a Owncloud container with MYSQL as backend on EFS…

Prerequisites :

1) Required 2 EC2 instance with docker installed and both should be in same availability zone.

2) Required packages nfs-utils for (RHEL/CENTOS) & nfs-common (ubuntu) should be installed on instance.

2) EFS cluster and ec2-insatances should be in same VPC .

3) Create mount point and Mount efs on your instance. (#command mount -t nfs4 -o vers=4.1 /data ) open port 2049 bi-direction-ally in security group.

4) Restart docker service to make sure changes are reflected properly.

5) Run docker container with volume mounting

Here in this POC we are using Owncloud and MYSQL images from Docker Hub. Owncloud is opensource tool and used to store files same as google-drive and MYSQL database is used in back-end for the Application.

Execute below command on Instance 1 :

docker run -d -p 3306:3306 — name mysqldb -v /data/mysql-database:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=mailjol@123 mysql

In above command we have mounted database from container to Host machine on /data that is EFS partition.

docker run -d -p 80:80 — name owncloud — link mysqldb:mysql -v /data/owncloud:/var/www/html/data owncloud:8.1

Now we have mounted Owncloud database to Host machine on EFS partition where your uploaded files are been stored .

Hit your server public IP in browser and completed the initial setup for login (Note : Use MYSQL database for connecting Ownloud app). Once you are logged in upload some file in your drive.

Now its time to crash the application and check data availability stop docker service to crash the application abruptly.

Assuming you have mounted same EFS on 2nd Instance now run both (MYSQL/Owncloud) container and login with your username and password .You should be able login see the files that you have uploaded earlier. BiinggOOOO data is available .

Kudos!!!!!!! you have successfully tested data persistence using AWS EFS service.