Easy configurations management with AWS AppConfig Hosted Configurations
by Fabio Fava
Syndication team is part of the Digital Media Distribution of ProSiebenSat.1 Digital, responsible for syndicating/distributing ProSieben formats and contents to other media companies and social media platforms.
Our microservices architecture is serverless, running on AWS Lambda, Step Functions, SQS, AppSync, DynamoDB, Kinesis. In order to support multiple customers, we make a large usage of configuration files to define how a package is structured for a specific customer, which images format and resolution to provide, etc.
Configuration files are accessed by different microservices during the workflow, and for this reason we needed a centralized service where to store them and access them quickly.
2. From S3 to AppConfig
On the first implementation of the Syndication service, we decided to follow the KISS principle. For this reason, we tried to address the configuration requirements by storing configuration files in an S3 bucket and sharing them among all the AWS Lambda functions.
After some analysis, we concluded that this was not the best solution and was only temporary: you need to assign to each lambda role a policy to access the S3 bucket, create a VPC endpoint for S3 to keep the network traffic internal and enable S3 Bucket versioning to keep control and history of the changes done to the configurations, in case of rollback. Together with such limitations, which are also costly (S3 versioning, S3 get operations, etc), we also needed a management console to check the history of our changes, for example to allow our Quality Assurance team to check the configurations content, instead of downloading files from the bucket or check in a git repository.
Hence, we started to evaluate other services to replace our S3 Bucket initial solution. We evaluated four different approaches: S3 (the solution to replace), Parameters Store, Lambda Layer, AppConfig Hosted Configurations.
Below you find a table with the comparison of the different services evaluated:
We initially had a look at the Parameters Store provided by AWS Systems Manager but unfortunately our configurations can be bigger than the service limits (4KB-8KB limit), so we had to discard this service.
Lambda layer was another candidate. This approach is very easy to implement on Lambda but has a big disadvantage: we had to load all configuration files into the Layer, instead of picking up what you need every time. Not very efficient for our use case, especially if the number of configurations will grow up in the future.
When we tried AWS AppConfig, we were surprised about its versatility, features and easy integration with AWS Lambda.
To be precise, we evaluated AWS AppConfig Hosted Configurations:
AWS AppConfig announces the launch of hosted configurations
Posted On: AWS announces the launch of AWS AppConfig hosted configurations, a new feature that simplifies on-boarding…
AWS AppConfig key features that were relevant for us are:
- Provides a centralized location to store and manage configurations.
- Easy access from AWS Lambda through Lambda integration.
- UI that could be accessed by other teams, like QA.
- JSON validation.
- 64KB max configuration size.
- Deployment strategies, versioning.
To be fair, there are some limitations to define the AppConfig structure and configurations deployment in an automated way through CloudFormation. Hosted Configurations requires to include the configuration as plain text in the template, with a limit of 4KB.
There is a way to work around this by using CloudFormation transformations: we stored our data in an S3 bucket (only temporary), link the URI to the template to be processed during the deployment, and delete them after this was completed.
On the other hand, the integration with the AWS Lambda is extremely easy:
AWS AppConfig works as a Lambda Integration: each request from a function directed to AppConfig goes directly to a localhost HTTP server, which is forwarding the message to AppConfig, retrieving the information and send it back. Caching is also possible, to improve the performances. Moreover, everything is encrypted at rest and at transit.
In concrete, to retrieve a specific configuration you just need to send an GET request from your Lambda to localhost, using the following URL convention:
Considering the multiple advantages vs. disadvantages, we decided to go for the AppConfig service because it perfectly addresses our requirements.
More information about AppConfig and how to integrate it in your AWS Lambda functions are available here:
AWS AppConfig integration with Lambda extensions
An AWS Lambda extension is a companion process that augments the capabilities of a Lambda function. An extension can…
We are really satisfied with the introduction of AWS AppConfig in our architecture. We can now validate and deploy configuration changes from a central location without interruptions, also having easy access in case of troubleshooting.
More in general, this service is useful also for other purposes rather than simply storing configurations: feature toggle use cases or deploy-monitor-rollback approaches.
More information are available here: