Terraforming infrastructure for EventBridge to Kinesis Data Stream relay

In my previous post, we discovered how Kinesis Data Streams can be used to solve race-conditions and we benchmarked its performance against SQS FIFO Queues. We determined under test that Kinesis Data Streams can be a good choice depending on the nature of your application and shape of the data. Please read that post before continuing with this one, although it is not a prerequisite.

In this post, I would like to do a step-by-step walkthrough on how we set up an EventBridge to Kinesis Data Stream relay using terraform as our Infrastructure as Code tool of choice.

Infrastructure as Code (IaC) is a widely used process for configuring and managing infrastructure in a declarative way. There are many IaC tools to choose from but the most popular and the one we use at Cazoo is terraform. It is an open-source tool for automating infrastructure provisioning and it has many benefits over other IaC tools. My favourite is that by maintaining state of the current infrastructure, terraform allows incremental changes to be made safely and efficiently.

Steps taken

1. Create an AWS Kinesis Data Stream.
2. Create an AWS Cloudwatch event rule, assuming you already have an event bus in place.
3. Declare the target to set up the relay of an event from EventBridge to the newly created Kinesis Stream
4. Create a new role for this resource and attach policies for access to event bus and execution to `put` data into Kinesis Stream.
5. Run terraform commands required to validate our configuration, show all the changes proposed by the current configuration, apply the changes and finally clean up by destroying the infrastructure created.
6. Restructure our code by making use of terraform modules.

Declaring the Infrastructure

Create a `main.tf `file before we begin. We will use it to enter our code and declare the infrastructure we wish to provision on AWS.

Create an AWS Kinesis Data Stream

Let us first declare our AWS resource for our Kinesis stream. We will give it a name, shard count, a retention period of 7 days (in hours), tags, and the shard level metrics we care about. Shard count and retention_period are configurable and depend on the nature of your application, i.e. throughput and volatility of the data flowing through the pipeline.

Create an AWS Cloudwatch event rule

Now we can create our Cloudwatch event rule where we need to declare the pattern of the event we want to relay to our Kinesis Stream. This will provide us with an EventBridge Rule resource. The naming is different because EventBridge was previously known as Cloudwatch Events, but the functionality is identical.

Declare the AWS Cloudwatch event target

Next step is to create a resource for declaring the AWS Cloudwatch event target so that our events are relayed from EventBridge to our Kinesis Stream. Rule property is the name of the rule we created in the previous step, the target_id is the resource name of the resource used to create our Kinesis Stream, and arn points to the arn of the same resource.

Creating a new IAM role

Now we can create a role for this resource and attach policies to the role using the aws_iam_policy_attachment resource. We will also use aws_iam_policy_document data resource to generate an IAM policy document in JSON format to use in this resource for a) Cloudwatch access policy and b) Cloudwatch execution policy.

First let’s create our data resources to declare an access and execution policy. To do this create a `data.tf` file and enter the following code:

Back in our `main.tf` file we can now declare a role using the aws_iam_role resource attaching the access policy we created above in our `data.tf` file and an aws_iam_policy resource for attaching the execution policy.

Finally using aws_iam_role_policy_attachment resource we can attach the policy to the role we have created.

Our terraform code is now complete. This configuration will provision the required infrastructure to relay any event in your event bus with `detail-type = orderCreated` to the newly created Kinesis Data Stream `test_event_stream`.

Run terraform commands

If you are logged in your AWS account using aws-cli you can simply run the following to provision the infrastructure in your AWS account. AWS console can be used to check that the resources have been provisioned correctly.

Last tested using terraform v0.14.10
terraform init
terraform validate (to check the configuration is valid)
terraform plan (will show all the changes required by the proposed configuration)
terraform apply (create or update the infrastructure)
terraform destroy (destroy previously created infrastructure so you can clean things up after trying out the code in this post)

Restructuring our code

Let us take it one step further and structure our terraform code in a more efficient way by making use of terraform modules. Modules allow us to build an abstraction layer describing the part of the infrastructure we want to provision for a given part of our architecture. In our case we can place our EventBridge to Kinesis Data Stream relay infrastructure declaration in its own directory and call it from the main terraform file.

Create a terraform directory and create environments and modules subdirectories. The environment directory can contain subdirectories for each of the environments we wish to provision our infrastructure and deploy code to. The modules subdirectory can contain the terraform code for different parts of the infrastructure we want to provision. In our case the directory tree could look something like the following:

.
|__src
   |__terraform
      |__environments
      |  |__test
      |     |__main.tf
      |__modules
         |__orderEventStream
            |__data.tf
            |__main.tf

The code we declared above can go in `src/terraform/modules/orderEventStream/data.tf` and `main.tf` files respectively.

We can then call our module abstraction from the main terraform file in `src/terraform/environments/test/main.tf`. For example:

We can use the terraform commands as we did previously to validate, plan, and apply the infrastructure. You can use the `-chdir=DIR` Global option to switch to a working directory before executing the given subcommand:

terraform -chdir=myproject/src/terraform/environments/test init
terraform -chdir=myproject/src/terraform/environments/test plan
terraform -chdir=myproject/src/terraform/environments/test apply

Conclusion

In the previous post, we explored serverless options on AWS to allow us to consume events in the order in which they are emitted. In this post, we explored how we can configure and manage the infrastructure required in a declarative way using IaC tools and we provided an example using terraform.

It takes time and effort looking up different resources in order to correctly write the terraform configuration. I hope that bringing all these steps together in this post will provide a useful guide for anyone looking at using IaC to provision and manage their infrastructure on AWS.

Thank you for reading.