Running a InfluxDB Time-Series database on a AWS ECS cluster

Every self-respecting business is looking at adequately measuring and monitoring KPI’s (Key Performance Indicators) or other critical parameters.

Often, such data presents itself as a time series: a series of data points indexed (or listed or graphed) in time order. Say server CPU usage or wind-speed at a certain location.

Time-series data can be stored in a wide variety of data-stores, including relational and non-relational databases, each with there respective trade-offs. In the last decade a new category of databases has been developed that are optimized for such datasets: TSDB’s or time-series databases.

Typically a time-series database has high ingest rates (but rarely overwrites a record) and a number of functions (as aggregation) that perform much better than any other traditional databases when it comes to large sets of time-series data.

Picking a TSDB

For a personal IoT project I evaluated a number of TSDB’s and settled on InfluxDB. Mainly because it was easy to start with and worked out of the box.

One of the challenges seemed to be that there isn’t a non-commercial versions out that there that has good redundancy and back-up build-in. The only solution to have some sort of high-availability seemed to run two identical instances of InfluxDB, each in another Availability Zone and having our client application write to both at the same time. This seems a bit awkward at first, but ended up working fine.

Running InfluxDB on AWS ECS

Persistant Storage

The easiest solution seemed to be to attach a AWS Elastic Filesystem (EFS) to both EC2 container instances and mount a data storage volume to the containers. As we are running relative small datasets I haven’t been to worried about performance issues.

The Cloudformation template below can be used as starting point to bring up a EFS that is accessible from two subnets, each in a separate AZ. A little hard to read, but notice how the EFS DNS name and mount command are exported for use in the template that will bring up both database containers.

myPrivateEfsSg:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !ImportValue myVpcID
GroupName: EfsPrivateSecurityGroup
GroupDescription: Security group for EFS mount
SecurityGroupIngress:
— IpProtocol: tcp
FromPort: 2049
ToPort: 2049
CidrIp: !ImportValue myPrivateSubnet1CidrBlock
— IpProtocol: tcp
FromPort: 2049
ToPort: 2049
CidrIp: !ImportValue myPrivateSubnet2CidrBlock
SecurityGroupEgress:
— IpProtocol: “-1”
CidrIp: “0.0.0.0/0”
ElasticFileSystemRetain:
Type: AWS::EFS::FileSystem
Condition: Retain
DeletionPolicy: Retain
Properties:
Encrypted: !FindInMap [ EncrpytionBoolean, !Ref EncryptionState, Boolean ]
KmsKeyId:
!If [ UseAWS-ManagedCMK, !Ref 'AWS::NoValue', !Ref Cmk ]
FileSystemTags:
- Key: Name
Value: !Ref 'AWS::StackName'
PerformanceMode: !Ref PerformanceMode
ElasticFileSystemDelete:
Type: AWS::EFS::FileSystem
Condition: Delete
DeletionPolicy: Delete
Properties:
Encrypted: !FindInMap [ EncrpytionBoolean, !Ref EncryptionState, Boolean ]
KmsKeyId:
!If [ UseAWS-ManagedCMK, !Ref 'AWS::NoValue', !Ref Cmk ]
FileSystemTags:
- Key: Name
Value: !Ref 'AWS::StackName'
PerformanceMode: !Ref PerformanceMode
ElasticFileSystemMountTarget0Retain:
Condition: Retain
DeletionPolicy: Retain
Type: AWS::EFS::MountTarget
Properties:
FileSystemId: !Ref ElasticFileSystemRetain
SecurityGroups:
- !Ref myPrivateEfsSg
SubnetId: !Ref Subnet1
ElasticFileSystemMountTarget0Delete:
Condition: Delete
DeletionPolicy: Delete
Type: AWS::EFS::MountTarget
Properties:
FileSystemId: !Ref ElasticFileSystemDelete
SecurityGroups:
- !Ref myPrivateEfsSg
SubnetId: !Ref Subnet1
ElasticFileSystemMountTarget1Retain:
Condition: Retain
DeletionPolicy: Retain
Type: AWS::EFS::MountTarget
Properties:
FileSystemId: !Ref ElasticFileSystemRetain
SecurityGroups:
- !Ref myPrivateEfsSg
SubnetId: !Ref Subnet2
ElasticFileSystemMountTarget1Delete:
Condition : Delete
DeletionPolicy: Delete
Type: AWS::EFS::MountTarget
Properties:
FileSystemId: !Ref ElasticFileSystemDelete
SecurityGroups:
- !Ref myPrivateEfsSg
SubnetId: !Ref Subnet2
Outputs:
ElasticFileSystem:
Value: !If [ Delete, !Ref ElasticFileSystemDelete, !Ref ElasticFileSystemRetain ]
Export:
Name: !Join ['', [ !Ref ExportName, 'ElasticFileSystem' ]]
ElasticFileSystemDnsName:
Description: DNS name for the Amazon EFS file system.
Value: !Join [ '.', [ !If [ Delete, !Ref ElasticFileSystemDelete, !Ref ElasticFileSystemRetain ], 'efs', !Ref 'AWS::Region', 'amazonaws', 'com' ] ]
Export:
Name: !Join ['', [ !Ref ExportName, 'ElasticFileSystemDnsName' ]]
ElasticFileSystemMountCommand:
Description: Mount command for mounting the Amazon EFS file system.
Value: !Join [ '', [ 'sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 ', !Join [ '.', [ !If [ Delete, !Ref ElasticFileSystemDelete, !Ref ElasticFileSystemRetain ], 'efs', !Ref 'AWS::Region', 'amazonaws', 'com:/' ] ] ] ]
Export:
Name: !Join ['', [ !Ref ExportName, 'ElasticFileSystemMountCommand' ]]

Brining up two InfluxDB containers

myInfluxDB1:
Type: AWS::CloudFormation::Stack
Properties:
Parameters:
mySubnet: !ImportValue myPrivateSubnet1
myLogGroup: "/ecs/influxdb1-ecs"
myServiceName: "influxdb1"
myContainerName: influxdb1-container
mySecurityGroup: !ImportValue PrivateClusterSG
TemplateURL: !Sub 'https://s3-${AWS::Region}.amazonaws.com/influxdb-ecs-nested-stack-${AWS::Region}-${AWS::AccountId}/stack-templates/influxdb-ecs-cf-template.yml'
TimeoutInMinutes: 20
myInfluxDB2:
Type: AWS::CloudFormation::Stack
Properties:
Parameters:
mySubnet: !ImportValue myPrivateSubnet2
myLogGroup: "/ecs/influxdb2-ecs"
myServiceName: "influxdb2"
myContainerName: influxdb2-container
mySecurityGroup: !ImportValue PrivateClusterSG
TemplateURL: !Sub 'https://s3-${AWS::Region}.amazonaws.com/influxdb-ecs-nested-stack-${AWS::Region}-${AWS::AccountId}/stack-templates/influxdb-ecs-cf-template.yml'
TimeoutInMinutes: 20

Bringing up the container services can be a little bit tricky. The volume section is where the magic happens and the EFS volume is mounted to the container.

Also, have a look at how service discovery has been done and the AWS Route 53 DNS updates with the containers IP address. Again, make sure you have a AWS Cloudmap service name available that can be imported by the resource.

The template exports the end-point where the database can be reached.

LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Ref myLogGroup
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: influxdb-ecs
RequiresCompatibilities:
— EC2
NetworkMode: awsvpc
ExecutionRoleArn: !Sub arn:aws:iam::${AWS::AccountId}:role/ecsTaskExecutionRole
Memory: 512
Cpu: 256
ContainerDefinitions:
- Name: !Ref myContainerName
Image: influxdb:latest
Memory: 512
Cpu: 256
MountPoints:
— ContainerPath: /var/lib/influxdb
SourceVolume: influxdb_data_volume
PortMappings:
-
ContainerPort: 8086
HostPort: 8086
-
ContainerPort: 8083
HostPort: 8083
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref myLogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: !Ref myServiceName
Volumes:
-
Host:
SourcePath: !Sub
— /efs/${ServiceName}
— {ServiceName: !Ref myServiceName}
Name: “influxdb_data_volume”

ServiceDefinition:
Type: AWS::ECS::Service
Properties:
LaunchType: EC2
TaskDefinition: !Ref TaskDefinition
Cluster: Hydro-Cluster-Private
ServiceName: !Ref myServiceName
ServiceRegistries:
— RegistryArn: !GetAtt InfluxDbServiceDiscovery.Arn
DesiredCount: 1
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: DISABLED
SecurityGroups: [!Ref mySecurityGroup]
Subnets: [!Ref mySubnet]
InfluxDbServiceDiscovery:
Type: AWS::ServiceDiscovery::Service
Properties:
Name: !Ref myServiceName
DnsConfig:
DnsRecords: [{Type: A, TTL: “10”}]
NamespaceId: !ImportValue myPrivateServiceDiscoveryNamespace
HealthCheckCustomConfig:
FailureThreshold: 1
Outputs: MyInfluxDbServiceName:
Description: The Service discovery name
Value: !Join
— ‘.’
— — !GetAtt InfluxDbServiceDiscovery.Name
— !ImportValue myPrivateNamespace

Conclusion

It’s possible to bring up a time-series database in a redundant configuration.

In a next article we will look into using Grafana as a dashboard for our database.

Product Manager | Technology Consultant | Interim CTO | Entrepreneur | Investor

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store