Or when LATEST is not the latest. A Terraform story

Background

In April 2020, AWS announced:

This is big news for users of ECS Fargate as persistent storage so far was very limited:

  • 10 GB of Docker layer storage
  • An additional 4 GB for volume mounts

while ECS tasks can disappear (conceptually) anytime. This makes it ECS-Fargate not a great fit for low performance database and anything else that requires durable storage for longer periods of time while ensuring easy accessibility.

With EFS now being available, bigger storage should be available for reasonable price: EFS charges on GB-month basis (unlike the blocke storage in EBS).

Also access to volumes in in ECS is (like Docker) limited to other ECS tasks only, whereas EFS can be accessed by EC2, too.

Usage

With all this being said, let’s use a simple example that showcases the usage of EFS in ECS Fargate with a InfluxDB task to achieve this:

We use Terraform as Infrastructure-as-Code tool:

Terraform Example

First we need an EFS file system (think about it like a NFS share). The Name tag is used to put a proper name on the resource in the AWS console:

locals {
# Common tags to be assigned to all resources for this exercise
common_tags = {
Project = "Medium"
Env = terraform.workspace
}
}
resource "aws_efs_file_system" "influxdb" {
tags = merge(local.common_tags, {
Name = "Example-${terraform.workspace}-influxdb"
})
}

The EFS must also be made available on the same subnet that will be used for the ECS task:

resource "aws_efs_mount_target" "influxdb" {
count = length(aws_subnet.public)
file_system_id = aws_efs_file_system.influxdb.id
subnet_id = aws_subnet._[count.index].id
security_groups = [aws_security_group.efs_influxdb_access.id]
}

and allow access to it only for the one security group:

resource "aws_security_group" "efs_influxdb_access" {
name = "Example-${terraform.workspace}-EFS-influxdb-access"
description = "Allow access to the Influxdb EFS"
vpc_id = aws_vpc._.id
ingress {
from_port = 2049
to_port = 2049
protocol = "tcp"
security_groups = [ aws_security_group.influxdb_access.id ]
}
tags = local.common_tags
}

So finally the ECS service can be defined:

resource "aws_ecs_service" "influxdb" {
name = "influxdb"
cluster = aws_ecs_cluster._.id
task_definition = aws_ecs_task_definition.influxdb.arn
desired_count = 1
launch_type = "FARGATE"
platform_version = "1.4.0"
network_configuration {
security_groups = [aws_security_group.influxdb_access.id]
assign_public_ip = true
subnets = aws_subnet._.*.id
}
tags = local.common_tags
}

with the following task definition.

The secret sauce

The important part here is to realize that the platform_version must be set to 1.4.0. While The Terraform documentation states

- this is results to 1.3.0— which does not support the efs_volume_configuration section.

resource "aws_ecs_task_definition" "influxdb" {
family = "Example-${terraform.workspace}-influxdb"
container_definitions = data.template_file.influxdb.rendered
task_role_arn = aws_iam_role.influxdb.arn
execution_role_arn = aws_iam_role.influxdb_execution.arn
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
cpu = var.influxdb_cpu
memory = var.influxdb_memory
volume {
name = "influxdb-storage"
efs_volume_configuration {
file_system_id = aws_efs_file_system.influxdb.id
}
}
tags = local.common_tags
}
data "template_file" "influxdb" {
template = file("task-definitions/influxdb.tpl")
vars = {
cpu = var.influxdb_cpu
memory = var.influxdb_memory
region = var.aws_region
log_group = aws_cloudwatch_log_group.influxdb.name
}
}

with the most basic InfluxDB task definition:

[{
"name": "influxdb",
"image": "influxdb:1.6",
"essential": true,
"cpu": ${cpu},
"memory": ${memory},
"portMappings": [{
"containerPort": 8086,
"hostPort": 8086
}],
"mountPoints": [{
"containerPath": "/var/lib/influxdb",
"sourceVolume": "influxdb-storage"
}],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "${log_group}",
"awslogs-region": "${region}",
"awslogs-stream-prefix": "ecs"
}
}
}]

Please see the documentation for further environment variables.

All this leaves us with a proper ECS task running InfluxDB with persistent storage on EFS:

Appendix

With regards to the performance of EFS volumes for ECS, please see the story InfluxDB Performance: EFS vs. ECS Storage

The full code can be found in this GitHub fargate-influxdb-efs module.

CloudSpout.io serves to channel the power of the Cloud for today’s business & tomorrow’s growth.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store