How I Built A Cascading Data Pipeline Based on AWS (Part 2)

Automatic, scalable, and powerful

Photo by Mehmet Ali Peker on Unsplash

Previously, I shared my experience in developing a data pipeline using AWS CloudFormation technology. It is not an optimal approach, though, because it leaves behind 3 more issues awaiting resolution:

1. The deployment has to be imposed manually which could increase the chances of errors;
2. All resources are created in one single stack, without proper boundaries and layers; as the development cycle goes on, the resource stack will be heavier, and managing it will be a disaster;
3. Many resources are supposed to be sustained and reused in other projects.

In short, we are going to increase the manageability and reusability of this project, in an agile manner.

Solution

AWS enables users to implement 2 types of CloudFormation structural patterns: cross-stack reference and nested stacking. Cross-stack reference stands for a designing style of developing cloud stacks separately, and usually independently, while the resources among all stacks can be interrelated based on the reference relationship. Nested stacking means a CloudFormation stack composed of other stacks. It is achieved by using the AWS::CloudFormation::Stack resource.