How I design 12-factor cloud native app on GCP.
The 12-Factor App defines a methodology for developing and deploying web applications, especially SaaS (software-as-a-service) apps.
These best practices are based on a few specific parameters for deployment of cloud-native applications:
- It uses declarative formats for setup automation
- It works with a clean contract with the operating system for greater portability between environments
- It limits the difference between development and production, for continuous deployment
- And it allows for scaling up and down without significant changes to tooling, architecture, or development practices.
The 12-factor methodology is programming language agnostic and works with any combination of backing services.
Looking at the parameters for 12-Factor app, I chose microservice architecture to build an app using Python, Docker, Git, Cloud Build, Terraform, Pytest, Sphinx etc. So that it will be easily scalable, portable, deployable.
1. Codebase
One codebase tracked in revision control, many deploys
12-factor app advocates that every application should have its own codebase. Multiple codebases for multiple versions must be avoided, also multiple apps sharing the same code is a violation of 12-factor.
I maintained the code in Git and also recommends to use Gitflow.
2. Dependencies
Explicitly declare and isolate dependencies
This principles maintains that you should never rely on the implicit existence of system-wide packages. The full and explicit dependencies specification is applied uniformly to both development and production.
I maintain the python library dependency in requirements.txt file. When code builds, it uses pip install -r requirements.txt command to deploy the python libraries from pypi server.
3. Config
Store config in the environment
An application and its configuration should be completely independent. It varies between deployments (development, qa, staging, production, etc)
The configuration includes database details, credentials to connect to external services, hostnames, etc.
These configurations should be stored separate from code.
The 12-factor app stores config in environment variables.
I maintained the environment configuration in the folder config/<environment>.yaml and set this as environment variables while deployment.
Please note, never store credentials and sensitive information in code repository.
4. Backing Services
Treat backing services as attached resources
A backing service is one that requires a network connection to run like Cloud SQL, memcached. If the location or connection details of such a service changes, you should not have to make code changes. Instead, these details should be available in the configuration.
The attachment/detachment of backing services makes 12-factor app loosely coupled.
I maintained the Database and Memcached details in configuration.
5. Build, release, run
Strictly separate build and run stages
The application must have a strict separation between the build, release and run stages.
- Build stage: Transform the code into an executable package.
- Release stage: Get the build package from the build stage and combines with the configurations of the deployment and make your application ready to run
- Run stage: It is like running your app in the execution environment.
I have used Google Cloud’s CI/CD pipeline in cloud build.
1. Builds Docker image based on the application code
2. Pushes the docker image to Google Cloud Repository
3. Deploys the code into Cloud Run.
6. Processes
Execute the app as one or more stateless processes
All the 12-factor apps have to be stateless which means any information that we want store/save has to be done with backend resources and these resources are attached to the app at runtime.
Sticky sessions, caching user session in memory of the app’s process, is against the 12-factor principle.
Session data should be stored in a databases like Memcached or Redis.
I have used, GCP memcached for caching.
7. Port binding
Export services via port binding
The 12-factor app is completely self-contained and executes standalone. The web server is packaged as a library and bundled with the application and do not require a separate server like Apache
The web app exports HTTP as a service by binding to a port, and listening to requests coming in on that port.
I used flask web framework which has gunicorn web server listening at 5000 port.
8. Concurrency
Scale out via the process model
12-factor app principles advocates to opt for horizontal scaling instead of vertical scaling.
By adopting the containerisation, applications can be scaled horizontally as per the demands.
9. Disposability
Maximize robustness with fast startup and graceful shutdown
In an automated deployment environment, the apps should bring up and bring down as quickly as possible.
Startup Time : When you deploy new code, you want the new version to start right away and be able to deal with traffic immediately.
Graceful Shutdown: If there is any issue, it should send the SIGTERM signal from the process manager to shut down gracefully.
The 12-factor app is against the sudden death.
With the adoption of containerisation into the deployment process of micro-services, starting and stopping of docker container can be instantaneous.
I used Cloud Run (Serverless Containerised application).
10. Dev/prod parity
Keep development, staging, and production as similar as possible
The 12-factor app is designed for continuous deployment by keeping the gap between development and production small.
Containers made is possible to build once and ship to multiple target environments.
Also, I used Terraform as a IaC tool to build an infrastructure so that same environments used in development and test or staging as are used in production.
11. Logs
Treat logs as event streams
Logs provides health of your applications. It is important to decouple the collection processing and analysis of logs from the core logic of your apps. Your application should not be concerned with the storage and management of these logs.
Generally the logs are written to a file on disk. It should be treated as an event stream.
I have used google-cloud-logging library to add logs into containerised application. The container logs written to these supported locations are automatically associated with the Cloud Run service, revision, and location.
12. Admin processes
Every application has some admin processes to run while deployment. All these processes should be checked into the source code repo to have consistency across environments.
These are usually one-off processes and should be decoupled from the application. These should be automated and repeatable, not manual processes.
We have setup Cloud Scheduler to regular backup of the datastore entities to cloud storage.
These 12-factor principles become a common set of principles for all cloud-native apps.
Other factors that is not mentioned but I would like to add are
13. Automated Testing
Automated testing is important part of DevOps methodology. It tests all the release candidates before deploying to the production. TDD approach is recommended.
I used pytest to automated unit tests.
14. Documentations
Documentations is also important part which developer miss most of the time. Code written once and read multiple times so we documented code is much needed.
I integrated sphinx with GitHub to automatically generate documents based on the doc string.
Sample documentations https://twelve-factor-app.readthedocs.io/en/latest/
15. System Monitoring
Get Performance metrics, System logs and Health checks using tools like Prometheus, ELK, etc.
Create Monitoring dashboard and get alerts on the threshold.
This is important part of Cloud native application to monitor the health of the system.
I used Cloud Operations tools for Monitoring, Alerting, Debugging..
16. Security
Security is also important aspect in building / maintaining Cloud native app. Maintain Role Based Access Control to the app and follow the
Principle of Least Privileges
Also get the Audit trail for security.
I used Cloud Audit logs for monitoring “who accessed what data, and when”.
Conclusion
The 12-factor design helps you decouple components of the application so that each component can be deployed to the cloud using continuous deployment and scaled up or down seamlessly. The design principles also help maximize portability to different environments.
Because the factors are independent of any programming language or software stack, 12-factor design can be applied to a wide variety of applications to achieve scalability , maintainability and portability.
