Managing encrypted SQS/SNS communication for microservices using Terraform

Communication between microservices can be daunting.

At BetterPT we use SQS/SNS for cross-service communication between microservices which works really well for us. Because we must maintain HIPAA compliance we have to encrypt everything that transmits or contains protected health information (PHI) both in transit and at rest. KMS and SQS are perfect for this. Each microservice has workers polling queues and processing messages (stay tuned for an upcoming blog post about our HIPAA compliant microservices infrastructure running on top of Kubernetes).

This article will mostly focus on getting Terraform for multiple environments up and running. With step by step directions we will build a production ready encrypted SQS/SNS communication with subscriptions to the topics and all necessary IAM permissions.

You will need to have Terraform by HashiCorp installed and ready to go, as well as an AWS account.

Our state will be stored in a remote bucket, and could be used for multiple environments.

1. Create a new folder in which all the work will be done. I’ll create two files in this folder: main.tf, which will contain a provider and backend, and variables.tf.

If you plan on storing this project in a remote repository, make sure to add a .gitignore with the following:

# Local .terraform directories
**/.terraform/*
# .tfstate files
*.tfstate
*.tfstate.*
# .tfvars files
*.tfvars
.terraform/*

2. To variables.tf I’ll add the following code:

variable "region" {
  description = "The AWS region."
  default     = "us-east-1"
}
variable "prefix" {
  default = "blog"
}
variable "account_id" {
  description = "ID of your AWS account"
}
variable "owner" {
  default     = "DevOps"
  description = "The name of the team who manages it"
}
variable "support" {
  default     = "team@yourcompany.com"
  description = "Support contact"
}

3. Next we will add this code to a main.tf file:

provider "aws" {
  region = "${var.region}"
  // Note: If you simply leave out AWS credentials, Terraform will
  // automatically search for saved API credentials (for example, in
  // ~/.aws/credentials) or IAM instance profile credentials. This
  // option is much cleaner for situations where tf files are 
  // checked into source control or where there is more than one 
  // admin user. See details here. Leaving IAM credentials out of
  // the Terraform configs allows you to leave those credentials out
  // of source control, and also use different IAM credentials for
  // each user without having to modify the configuration files.
}
terraform {
  backend "s3" {
    encrypt = true
    bucket  = "" // You will need to replace this with just name of your bucket, not full host name
    key     = "remote-state"                // Name of the state 
    region  = "us-east-1"                   // backend does not take interpolations
  }
}

At this point we are ready to initialize our backend project. Let’s run these commands in our project root folder:

$ terraform init
$ terraform workspace list

You will see that you are currently in the default workspace. Let’s create a new dev workspace and select it:

$ terraform workspace new dev
$ terraform workspace select dev

Cool, our environment is setup and we are ready to start creating our awesome infrastructure.

Let’s start from SQS. I’ll create a module for it so we can reuse it.

Create a folder called modules and in it add new directory called sqs.

In the sqs folder we will start from from creating interface. Create a file called interface.tf, we will add all outputs and variables there.

Let’s add these variables to interface.tf:

/****************VARIABLES*****************************************/
variable “environment” {}
variable “name” {
  type = “string”
  description = “Name of the queue”
}
variable “key_id” {
  type = “string”
  description = “KMS key id for encryption”
}
variable “region” {
  description = “The AWS region.”
  default = “us-east-1”
}
variable “owner” {
  default = “DevOps”
  description = “The name of the team who manages it”
}
variable “support” {
  default = “team@yourcompany.com”
  description = “Support contact”
}
/****************VARIABLES*****************************************/

Now add main.tf file where we will be creating actual resources.

Let’s add a dead letter queue to it:

resource “aws_sqs_queue” “dead_letter_queue” {
  name = “${var.name}-dead-letter-queue-${var.environment}”
  message_retention_seconds = 432000
  kms_master_key_id = “${var.key_id}”
  kms_data_key_reuse_period_seconds = 300
  tags {
    Name = “${var.name}-dead-letter-queue”
    Owner = “${var.owner}”
    Support = “${var.support}”
    Environment = “${var.environment}”
  }
}

And a regular queue:

resource “aws_sqs_queue” “queue” {
  name = “${var.name}-queue-${var.environment}”
  delay_seconds = 0
  message_retention_seconds = 86400
  redrive_policy = “{\”deadLetterTargetArn\”:\”${aws_sqs_queue.dead_letter_queue.arn}\”,\”maxReceiveCount\”:100}”
  kms_master_key_id = “${var.key_id}”
  kms_data_key_reuse_period_seconds = 300
  
  tags {
    Name = “${var.name}-queue”
    Owner = “${var.owner}”
    Support = “${var.support}”
    Environment = “${var.environment}”
  }
}

As you can see this module will be creating two queues. They are configured with a re-drive policy and will be using KMS encryption. This is all we will need from SQS. Now we will add outputs for this module to interface.tf.

/*************OUTPUTS*******************************************/
output “dead_letter_queue_id” {
  description = “${var.name} dead letter queue URL”
  value = “${aws_sqs_queue.dead_letter_queue.id}”
}
output “dead_letter_queue_arn” {
  description = “${var.name} dead letter queue arn”
  value = “${aws_sqs_queue.dead_letter_queue.arn}”
}
output “queue_id” {
  description = “${var.name} dead queue URL”
  value = “${aws_sqs_queue.queue.id}”
}
output “queue_arn” {
  description = “${var.name} dead queue arn”
  value = “${aws_sqs_queue.queue.arn}”
}
output “queue_name” {
  description = “${var.name} dead queue full name”
  value = “${var.name}-queue-${var.environment}”
}
/*************OUTPUTS*******************************************/

Our module is ready.

We will definitely need multiple topics so let’s create a new module which will be creating a SNS topic and queues subscriptions to it.

In the modules directory I will create a new folder called sns, and again we will start with creating an interface in interface.tf:

/*************VARIABLES******************************************/
variable “environment” {}
variable “region” {
  description = “The AWS region.”
  default = “us-east-1”
}
variable “name” {
 type = “string”
 description = “Name of the queue”
}
variable “policy_name” {
  type = “string”
  description = “Name for policy SID”
}
variable “account_id” {}
variable “queue_names” {
  type = “list”
}
/*************VARIABLES******************************************/ 

Now let’s add resources in main.tf:

resource “aws_sns_topic” “topic” {
  name = “${var.name}-${var.environment}”
  display_name = “${var.name}-${var.environment}”
}
resource “aws_sns_topic_subscription” “subscription” {
  count = “${length(var.queue_names)}”
  topic_arn = “${aws_sns_topic.topic.arn}”
  protocol = “sqs”
  endpoint = “arn:aws:sqs:${var.region}:${var.account_id}:${var.queue_names[count.index]}-queue-${var.environment}”
  endpoint_auto_confirms = true
}

This module will create a SNS topic and will subscribe it to the queues which we will provide to it.

Outputs to interface.tf:

/*************OUTPUTS**************************************/
output “topic_arn” {
  description = “${var.name} dead letter queue URL”
  value = “${aws_sns_topic.topic.arn}”
}
/*************OUTPUTS**************************************/

SNS module is ready. Now in our project root directory we are going to add a new file called sqs.tf where will be creating queues. We will start by adding a KMS key which will be used for encryption of the queues and permissions for it.

Let’s start from creating a policy document for our key. In the sqs.tf file add the following:

data “aws_iam_policy_document” “sqs_key_policy” {
  policy_id = “sqs-sms-key-policy-${terraform.workspace}”
  
  statement {
    sid = “Enable IAM User Permissions”
    actions = [“kms:*”]
    principals = {
      type = “AWS”
      identifiers = [“arn:aws:iam::${var.account_id}:root”]
    }
    resources = [“*”]
  }
  
  statement {
    sid = “SNS decrypt permission”
    actions = [“kms:GenerateDataKey*”, “kms:Decrypt”]
    principals = {
      type = “Service”
      identifiers = [“sns.amazonaws.com”]
    }
    resources = [“*”]
  }
}

Now let’s add a key itself and give it our policy:

resource “aws_kms_key” “sqs-encryption-key” {
  description = “This key is used for encryption of SQS queues”
  policy = “${data.aws_iam_policy_document.sqs_key_policy.json}”
  tags {
    Name = “sqs-encryption-key-${terraform.workspace}”
    Owner = “${var.owner}”
    Support = “${var.support}”
    Environment = “${terraform.workspace}”
  }
}

Then we will add two queues — user and blog. We will use sqs module that we have created:

module “user_sqs” {
  source = “modules/sqs”
  region = “${var.region}”
  environment = “${terraform.workspace}” // name of the workspace we are in
  name = “user”
  key_id = “${aws_kms_key.sqs-encryption-key.id}”
}
module “blog_sqs” {
  source = “modules/sqs”
  region = “${var.region}”
  environment = “${terraform.workspace}”
  name = “blog”
  key_id = “${aws_kms_key.sqs-encryption-key.id}”
}

Now let’s create few topics in topics.tf:

module “new-user” {
  source = “modules/sns”
  name = “new-user”
  account_id = “${var.account_id}”
  environment = “${terraform.workspace}”
  queue_names = [“user”, “blog”]
}
module “new-blog” {
  source = “modules/sns”
  name = “new-blog”
  account_id = “${var.account_id}”
  environment = “${terraform.workspace}”
  queue_names = [“blog”]
}

You would think this is everything, right? It isn’t. We have created queues and subscriptions, but we still need to add permissions for queues. Let’s do it.

First let’s start by creating locals which will help us managing topics in the sqs.tf file:

locals {
  user_topic_arns = [“${module.new-user.topic_arn}”]
  blog_topic_arns = [“${module.new-user.topic_arn}”, “${module.new-blog.topic_arn}”]
}

Next we will add a queue policy for user queue:

resource “aws_sqs_queue_policy” “user_sqs_policy” {
  queue_url = “${module.user_sqs.queue_id}”
  policy = <<POLICY
{
  “Version”:”2012–10–17",
  “Id”: “sns-to-user-queue-${terraform.workspace}-sqspolicy”,
  “Statement”:[
    {
      “Sid”:”SNSToSQSPolicy${terraform.workspace}”,
      “Effect”:”Allow”,
      “Principal”:”*”,
      “Action”:”sqs:SendMessage”,
      “Resource”:”${module.user_sqs.queue_arn}”,
      “Condition”:{
        “ArnEquals”:{
          “aws:SourceArn”: ${jsonencode(local.user_topic_arns)}
        }
      }
    }
  ]
}
POLICY
}

And for for blog queue:

resource “aws_sqs_queue_policy” “blog_sqs_policy” {
  queue_url = “${module.blog_sqs.queue_id}”
  policy = <<POLICY
{
  “Version”:”2012–10–17",
  “Id”: “sns-to-blog-queue-${terraform.workspace}-sqspolicy”,
  “Statement”:[
    {
      “Sid”:”SNSToSQSPolicy${terraform.workspace}”,
      “Effect”:”Allow”,
      “Principal”:”*”,
      “Action”:”sqs:SendMessage”,
      “Resource”:”${module.blog_sqs.queue_arn}”,
      “Condition”:{
        “ArnEquals”:{
          “aws:SourceArn”: ${jsonencode(local.blog_topic_arns)}
        }
      }
    }
  ]
}
POLICY
}

And this is all we need. We are ready to run terraform init again, which will pull all modules. If it was successful you can run terraform plan which will show you what will be provisioned. If it looks good you can now run terraform apply. Be careful with apply as it will actually create these resources in your AWS account, which might result in some charges.

Now if you want to create same kind of infrastructure for different environment you need to do following steps:

$ terraform workspace new staging
$ terraform workspace select staging
$ terraform plan
$ terraform apply

The whole repository with code could be found here: https://github.com/xpolb01/terraform-encrypted-sqs-sns

Summary

Using Terraform has eliminated big part of management for our use case of SNS/SQS. It has made it easy to provision changes to permissions, subscriptions, and queues. By running all changes on our staging environment first allowed us to make sure that there will be no bugs introduced to production. It is simply an easy, efficient, and cool thing to work with.