Go Macro, then Micro
This post shows you how we improved our application’s maintainability, observability, and onboarding time, using StepFunction, applying the concept “go macro, then micro”.
Suppose you read here how we distribute content with our serverless application, and here how we monitor and observe our application. In that case, you will understand that although the Syndication application is stable and requires minimum maintenance, we keep an eye on AWS services features and metrics to keep the applications somehow up to date and more straightforward in terms of maintainability.
Recently we had a minor problem for a case that was not supported, and with the team, we needed to verify and understand how to apply this flow to the existing logic. We were in a situation to open a part of code written years ago. As you can imagine, even with the best documentation possible and unit tests, we had some problems remembering the reasons for certain business logic that looked outdated to us.
Let me summarize again what the Syndication Application does. Conceptually speaking, there are three primary operations that a syndication application must handle:
- Ingest
- Update
- Takedown
Each operation typically generates a package made of:
- Video(s)
- Metadata format(s) (XML, XLS, CSV)
- Image(s)
Once the package is generated, it will be delivered to media companies and social media platforms. The application is made of different Microservices, and the one that we are talking about today is this one:
At this point, when a delivery request arrives, there is some business logic to run to understand what type of package should be generated. This part consists of one Lambda function that will decide if the entry event should be processed from one of the three Lambda functions:
- CREATE
- UPDATE
- AGGREGATION
Without going into details in each of these Lambda functions, we can say that the business logic is made up of few software design patterns:
- Factory
- Strategy
- Iterators
- Template Methods
In these Lambda functions, we access a configuration with AWS AppConfig, and they just work. Because of this, they got little attention over time.
Apart from the problem, what bothers me is that we have to remember the flow and some edge cases, bringing to the title “Go Macro, then Micro”.
The Lambda functions are not poorly written and are well covered with tests, but we lost one critical feature observability with time.
Luckily for us in the meanwhile, AWS announced Step Functions Workflow Studio, and this allowed me to visually build the new flow and use it as a tool for the presentation to the team. Sometimes an image is more effective than many words.
The first objection that I got presenting this was “Too many Lambda functions”.
We indeed moved from a handful of Lambda functions to X time more, but we gained much more:
- Observability
- Maintainability
- Extensibility
- Reduce onboarding time
Observability
For example, one event for us could be CREATE for Partner1 and UPDATE for Partner2. A Macro Lambda hides the logic and, the architecture diagram alone does not tell the whole story. Moving to this design, we have now a comprehensive view of the flow.
Maintainability
The code inside the Lambda functions is now minimal (Micro). We removed boilerplate code like If, Loops, and code needed to implement patterns like a factory, a strategy with an effect in more straightforward unit tests.
This flow is now more accurate for non-technical people, which is a benefit that we always underestimate. After all, our applications follow some ubiquitous language that we use with business people, and the flow can now represent this language.
Extensibility
Using AWS Step Function, we can identify individual service responsibilities without the need to understand the details. For example, we can now extend a branch for a specific partner without worrying about the base flow.
Reduce onboarding time
For onboarding time, I aim to solve both:
- New partner onboarding
- New developer team member
Assuming that we have ten partners with this design, I now have ten Lambda functions for the CREATE path and 10 Lambda functions for the UPDATE path.
The base code is shared through the npm package or lambda layer (whatever fits best). However, thanks to Template Methods patterns, we can now leverage that specific partner’s behavioural logic without going inside the original code to see how and what it is doing, allowing a prompt understanding of the partner behaviour thanks to the removal of the boilerplate code.
Conclusion
Serverless architecture diagrams can be scary if you are moving from a classic cluster application. Still, once you start, you will understand that they have tremendous advantages. We moved from cluster to serverless, and we had Macro Lambda functions first, and with time and evolutions of the AWS services, we are developing the architecture and embracing the Micro Lambda functions concept unleashing all other features like maintainability, observability, onboarding time, making the next step to make our application feature ready.
