Configure KCL to read from a different AWS Account
TL;DR
Kinesis Client Library (KCL) is a library used to build Applications that process data from Kinesis Data Streams. In this tutorial we describe how to configure KCL in order to connect an Application running in an AWS Account different from the AWS Account where the Kinesis Stream is.
Preface
At upday we collect anonymous user data in order to improve content recommendations. For collecting this data we created a custom user events tracking system.
In a similar fashion, we send user data to Snowplow, our tracking/analytics solution. In order to simplify complexity, get rid of redundancy and save costs we started to investigate how we could reuse user data which are sent to Snowplow for creating user recommendations as well.
Our recommendation systems and our events tracking reside in the same AWS Account, therefore we don’t have any issues connecting them. This would not be the same if we use our Snowplow user data being located in another AWS Account.
Reading Kinesis Stream from a different AWS Account
Our event tracking system receives events sent by our upday app and puts them into a Kinesis Stream. These are Data Streams collecting and processing large Streams of data records in real time. As soon as we have those records inside the Stream, different recommendation systems are able to consume them.
The first challenge was to connect the recommendation systems, running on EC2 instances of our AWS Production Account, to a Kinesis Stream that resides in another AWS Account that hosts Snowplow related systems.
Below we describe the necessary steps to connect them.
Step by step tutorial
Delegate Access Across AWS Accounts Using IAM Roles.
- We need to create a role in the AWS account where the Kinesis Stream is. First out which is the Account ID of the AWS account where the Application is running. For that sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/. In navigation bar, choose Support, and then Support Center. The Account Number is in the upper right corner immediately below the Support menu. The account ID is a 12-digit number.
2. Then sign in the AWS Account where the Kinesis Stream is, open the IAM console at https://console.aws.amazon.com/iam/, click on Roles then click on the button Create Role.
3. Click on Another AWS Account option and add the Account Id where the Application is running and click on next
5. Select AmazonKinesisFullAccess policy and click on next
6. Add a role name and a description and click on create.
Configure Kinesis Client Library to assume the role external-kinesis-access
Kinesis Client Library (KCL) is available in different languages. For the sample we choose the Java implementation. When implementing a KCL consumer Application in Java, as explained in this tutorial Developing Amazon Kinesis Data Streams Consumers Using the Kinesis Client Library, we must complete the following tasks :
- Implement the IRecordProcessor Methods
- Implement a Class Factory for the IRecordProcessor Interface
- Create a Worker
- Modify the Configuration Properties
- Migrating to Version 2 of the Record Processor Interface
All these tasks will remain the same, the only task that needs to be changed is Modify the Configuration Properties. The fields which need to be configured are:
- Application name
- Stream name
- worker id
- credentials for DynamoDB (where KCL keeps track of the Application state)
- credentials for Cloudwatch (where KCL publish the metrics)
- credentials for Kinesis Stream itself.
In our sample we want to have the metrics and the Application state in the AWS Account where the Application is running and the Kinesis Stream in another AWS Account. For that just follow the steps bellow.
- First we create the credentials provider chain for Cloudwatch and DynamoDB
- Then we create the credentials provider chain for Kinesis that is in the other Account.
3. Finally we configure the Kinesis Client Library, passing the correct credentials provider chain as a parameter for each service.
Conclusion
After this experiment we found out that we can consume events that reside in a different AWS Account, making one step forward deprecating our custom event tracking system.
