Get to know the Twelve Factor App

Photo by Christina @ wocintechchat.com on Unsplash

Introduction

In today’s world, applications are often delivered as web applications or software-as-a-service (SaaS). The Twelve-Factor App is a methodology used to build scalable modern applications. Introduced in 2011 by developers at Heroku, this methodology is based on their extensive experience in developing and deploying hundreds of applications. Understanding the twelve factors is essential for developers today as it represents the best practices for building applications with maximum portability and continuous development support.

1. Codebase

One codebase tracked in revision control, many deploys

In the Twelve-Factor App methodology, an application must have a single codebase tracked in a version control system like Git, Mercurial, or Subversion.

Key Points:

• One codebase per application.
• Multiple deployments from the same codebase.
• All deployments share the same codebase, although they may be at different versions.

Following this principle makes code management easier as every change to the code can be tracked effectively.

Incorrect Example:

An e-commerce application has one repository for product pages and another repository for the checkout page. This violates the twelve-factor principle because there should be one repository encompassing all the application code. These separate repositories complicate management and deployment and increase the risk of inconsistencies.

Correct Example:

A social media application has a single code repository. The development team uses this repository to create copies of the application for development, staging, and production. All environments use the same codebase, although the active versions might differ.

2. Dependencies

Explicitly declare and isolate dependencies

When developing applications, various libraries or modules are often needed. In the Twelve-Factor App methodology, all dependencies must be explicitly declared and isolated from the operating system.

Key Points:

• Declare all dependencies in a manifest file like requirements.txt.
• Isolate dependencies using tools like virtualenv to ensure the application does not rely on system-level libraries.
• Ensure consistency between development and production environments by using the same dependency declaration and isolation methods.

This principle ensures that the application can run consistently in various environments and is easy to manage and maintain.

Incorrect Example:

Creating a Python application without declaring dependencies in requirements.txt and installing libraries directly on the operating system with commands like pip install requests flask, leading to dependencies on system libraries that may not be present or may have different versions in other environments. Without using virtualenv, the application may use libraries from the operating system, which might not be compatible or consistent with the application's requirements.

# Direct system installation (Incorrect)
pip install requests flask

Correct Example:

When creating a Python application, use requirements.txt to declare all dependencies. For example, if you need the requests and flask libraries, list them in requirements.txt. Additionally, use virtualenv to create an isolated environment so that no system libraries are used by the application. With the command pip install -r requirements.txt, all dependencies will be installed in the isolated environment.

# requirements.txt
requests==2.25.1
flask==1.1.2

# Creating a virtual environment
python -m venv venv
source venv/bin/activate

# Installing dependencies
pip install -r requirements.txt

3. Config

Store config in the environment

When developing applications, configuration differences between deployments are likely. Examples of configurations include:

• Resource addresses like database URLs
• API credentials

The Twelve-Factor App emphasizes strict separation of configuration from code. By using environment variables for configuration, applications become more secure, portable, and easier to manage. This also ensures that the application code remains consistent across all deployments, while configurations can be tailored for each specific environment.

Benefits:

• Security: Sensitive information like credentials are not stored in the code, allowing the code to be open without exposing secrets.
• Portability: Configurations can be easily changed between deployments without altering the application code.

Incorrect Example:

Storing configuration as constants in the code. For example, writing database credentials directly in the code:

# Incorrect: Storing configuration in code
DB_USERNAME = "admin"
DB_PASSWORD = "supersecret"

Correct Example:

Storing configuration in environment variables. This way, configurations can be changed without modifying the application code.

# Setting environment variables
export DB_USERNAME="admin"
export DB_PASSWORD="supersecret"

# Accessing environment variables in code
import os
DB_USERNAME = os.getenv("DB_USERNAME")
DB_PASSWORD = os.getenv("DB_PASSWORD")

4. Backing Services

Treat backing services as attached resources

Backing Services are services consumed over the network as part of an application’s normal operations. Examples include databases (like MySQL or CouchDB), messaging/queueing systems (like RabbitMQ or Beanstalkd), SMTP services for email (like Postfix), and caching systems (like Memcached). Twelve-factor app code does not distinguish between local and third-party services.

Benefits:

• Flexibility: Easily swap local backing services with third-party managed ones without changing the code.
• Portability: Easily move or deploy applications to different environments by changing environment variables.
• Easy Management: Easily replace a malfunctioning backing service with another service without modifying the application code.

Source: 12factor.net

For example, if your application uses a local MySQL database, you might decide to switch to a MySQL database managed by Amazon RDS for performance or reliability reasons.

Incorrect Example:

Hardcoding connection details in the application code.

# Local database connection
DATABASE_URL = "mysql://localhost:3306/mydatabase"

When switching to Amazon RDS, you need to change the application code and redeploy:

# Amazon RDS connection
DATABASE_URL = "mysql://user:password@rds.amazonaws.com:3306/mydatabase"

Correct Example:

Storing connection details in environment variables.

# Setting environment variable for local database
export DATABASE_URL="mysql://localhost:3306/mydatabase"

When switching to Amazon RDS, just change the environment variable without altering the application code:

# Setting environment variable for Amazon RDS
export DATABASE_URL="mysql://user:password@rds.amazonaws.com:3306/mydatabase"

5. Build, Release, Run

In the application development process, there are three main stages that code must go through before becoming a runnable application:

1. Build Stage: The build stage transforms the code into an executable bundle. The output of this stage is a build, which includes all compiled code and dependencies.
2. Release Stage: The release stage combines the build from the build stage with the necessary configuration for a specific deployment. The output is a release, ready to run in the execution environment.
3. Run Stage: The run stage involves running the application in the execution environment.

The Twelve-Factor App strictly separates the build, release, and run stages. Code changes cannot be made at runtime because they cannot propagate back to the build stage. Deployment tools usually offer release management tools, like the ability to roll back to a previous release. Each release should have a unique ID, such as a timestamp (e.g., 2011–04–06–20:32:17) or an incrementing number (e.g., v100), and cannot be modified once created. Builds are initiated by developers when new code is deployed, while runtime execution can happen automatically, such as during a server reboot or when a crashed process is restarted. Therefore, at the run stage, the number of components or processes should be minimized. This is crucial because if issues prevent the application from running, they can cause downtime at inopportune times, like in the middle of the night when no developers are available. The run stage should be simple and stable, with few moving parts. Conversely, the build stage can be more complex, as any errors can be quickly addressed by developers during deployment.

Incorrect Example:

Build Stage:

• Action: Developer makes changes directly in the production environment.
• Example: Developer edits files on the production server and runs build commands there, such as npm run buildon the production server.

Release Stage:

• Action: No separation between build and release.
• Example: After the build is completed, the developer immediately runs the build without saving and identifying the release with a unique ID. Consequently, there is no way to roll back if an error occurs.

Run Stage:

• Action: Too many processes running at runtime.
• Example: Developer adds numerous additional components or services running at runtime, such as complex logging servers or resource-intensive monitoring scripts, causing the application to become unstable and hard to maintain.

Correct Example:

Build Stage:

• Action: Developer runs commands to compile code and package it into a bundle.
• Example: In a Node.js project, the developer runs npm run build to compile the code, optimize assets, and prepare the bundle for deployment.

Release Stage:

• Action: The created build is combined with the latest configuration to create a unique release.
• Example: Using tools like Capistrano, releases are stored in a subdirectory with a unique name based on a timestamp, such as 2024-06-12-14:30:00. The developer ensures this release is ready to run in the target environment.

Run Stage:

• Action: The created release is run in the production environment.
• Example: Running the application on the production server with commands like docker run myapp:2024-06-12-14:30:00. If any issue arises, only this release is affected and can be quickly replaced with the previous release.

6. Processes

Execute the app as one or more stateless processes

In the Twelve-Factor methodology, an application is run as one or more stateless processes. This means that an application should not store any data that needs to persist in the process memory or filesystem. Instead, persistent data should be stored in stateful backing services like a database.

By running the application as stateless and share-nothing processes, it ensures the application is easier to scale and more reliable, as each process can be replaced without losing critical data. All data that needs to persist should be stored in well-managed backing services like databases or external caching systems.

Incorrect Example:

Stateful Application:

• Sticky Sessions: A web application stores user session data in local memory and expects subsequent requests from the same user to be routed to the same server. If that server restarts or if the request is routed to a different server, the session data is lost or unavailable, causing issues for the application.

Local Memory Storage:

• Non-Persistent Cache: The application stores important calculation results in local memory and relies on that data for subsequent requests. If the process restarts, this data is lost, and the application cannot continue operating correctly.

Correct Example:

Stateless Application:

• Using a Database for Data Storage: A web application uses PostgreSQL to store user information. Each HTTP request is processed without relying on data stored in the application’s local memory. If the server restarts, no data is lost as everything is stored in the database.

Using Cache for Temporary Operations:

• Processing Large Files: The application downloads a large file, processes it, and stores the results in the database. The file is only temporarily stored in the local filesystem during this process and is deleted afterward.

7. Port Binding

Export Services via Port Binding

Port binding is a concept where an application directly connects to a port to provide services without relying on an external web server. Imagine owning a restaurant at home. In the traditional model, you need a chef (like Apache or Tomcat) to come to your house every day to cook. Without the chef, your restaurant can’t operate.

However, with port binding, the restaurant no longer needs an external chef because you have the cooking skills yourself and can serve food to customers. Customers only need to know the restaurant’s address (URL and port), like http://localhost:5000/, to come and enjoy the menu. Though not entirely analogous, this is a simplified analogy of port binding in real life.

In practice, this means our application must have its web server directly integrated. For instance, a Python application uses Tornado, a Ruby application uses Thin, or a Java application uses Jetty. All of this happens within the application’s code, allowing the application to bind to a port and handle requests.

With port binding, our application becomes more self-contained and easier to deploy across various environments without depending on specific web server configurations. In a local development environment, we can access the service provided by the application through a URL like http://localhost:5000/. In deployment, a routing layer handles requests from a public-facing hostname to the port-bound web processes.

Port binding also allows one application to become a backing service for another application by providing the URL as a resource handle in the configuration of the consuming application. This makes the application more flexible, self-contained, and easier to manage, simplifying the management of various services in complex application architectures.

8. Concurrency

Scale out via process model

In the Twelve-Factor methodology, applications are organized into separate processes to handle different tasks. For example, one process handles web requests (HTTP requests), while another handles background tasks (such as data processing).

Benefits of Concurrency:

1. High Scalability: Easily add or remove processes according to workload demands.
2. Workload Isolation: Different types of work can be isolated into separate processes, increasing reliability.
3. Flexibility: Applications can run different types of processes on different physical machines, allowing for unlimited scaling as workload increases.
4. Efficient Management: Process management, crash handling, and user-initiated restarts or shutdowns are managed by the operating system’s process manager.

Source: 12factor.net

In the diagram, we see several types of processes running concurrently:

• web.1 and web.2 processes handle HTTP requests.
• worker.1 to worker.4 processes handle background tasks.
• clock.1 process handles scheduled tasks.

By separating tasks into different processes, the application becomes easier to scale and manage, ensuring reliable and efficient performance.

Incorrect Example:

Combining all tasks into one large process, making it difficult to scale and prone to failure, where a single failure can bring down the entire application.

Correct Example:

Using separate processes to handle different types of workloads, such as web processes for HTTP requests and worker processes for background tasks, which can be managed and scaled independently.

9. Disposability

Maximize robustness with fast startup and graceful shutdown

In the Twelve-Factor methodology, application processes should be able to start and stop quickly and easily. This means the application can rapidly adapt to changes in workload, code updates, or configuration changes.

Benefits of Disposability:

1. Fast Scalability: Processes can quickly start to increase capacity as workload increases.
2. Rapid Updates: New code or configurations can be deployed quickly because processes can start and stop quickly.
3. Reliability: The application becomes more robust as it can easily be moved to new physical machines if needed.

The core of disposability is fast startup and graceful shutdown. Fast startup means processes should start within seconds to be ready to handle requests or tasks. This aids in quick releases and scaling. Graceful shutdown means when a process receives a SIGTERM signal, it should:

1. Stop accepting new requests.
2. Complete any ongoing requests.
3. Exit cleanly.

By designing processes that can start and stop quickly and handle sudden terminations, the application becomes more reliable and easier to scale.

Web Process Example:

Web processes should stop listening for new requests on the service port and complete any ongoing requests before fully shutting down.

Worker Process Example:

Worker processes should return the current job to the work queue if they receive a shutdown signal. For example, in RabbitMQ, the worker can send a NACK (negative acknowledgment).

Robustness Against Failures:

Processes should be designed to withstand sudden failures, like hardware crashes. They should handle sudden terminations by utilizing reliable queuing backends.

10. Dev/Prod Parity

Keep development, staging, and production as similar as possible

To keep applications stable and easy to manage, it is important to make the development, staging, and production environments as similar as possible. This means the tools and configurations used in development should be nearly identical to those used in production. Often, there are significant gaps between development (where a developer makes live edits to a local deploy of the app) and production (a running deploy of the app accessed by end-users). These gaps manifest in three areas:

1. Time Gap: Code written by developers may take days, weeks, or even months to reach production.
2. Personnel Gap: Developers write the code, while ops engineers deploy it.
3. Tools Gap: Developers might use a stack like Nginx, SQLite, and OS X, while the production deploy uses Apache, MySQL, and Linux.

The Twelve-Factor App is designed for continuous deployment by keeping the gap between development and production small. Addressing the three gaps:

1. Reduce Time Gap: Developers should write code and have it deployed within hours or even minutes.
2. Reduce Personnel Gap: Developers who write the code should be closely involved in deploying it and monitoring its behavior in production.
3. Reduce Tools Gap: Keep development and production environments as similar as possible.

Benefits:

1. Fast Deployment: Code written can be deployed quickly without delays.
2. Increased Collaboration: Developers who write the code are also responsible for deployment, understanding potential issues better.
3. Tool Consistency: Using the same tools across all environments minimizes the risk of incompatibility or unexpected bugs.

Keeping development, staging, and production environments consistent ensures that your application remains stable and ready for continuous deployment. This reduces the risk of unexpected issues and ensures that what is tested in development will work properly in production.

Incorrect Example:

• Time Gap: Code written today is deployed next week.
• Personnel Gap: Developers write the code but are not involved in deployment.
• Tools Gap: Using SQLite in development but MySQL in production.

Correct Example:

• Time Consistency: Code written in the morning is deployed to production in the afternoon.
• Personnel Consistency: Developers who write the code are responsible for deployment and monitoring.
• Tool Consistency: Using PostgreSQL in both development and production environments.

11. Logs

Treat logs as event streams

Logs are records of events that happen within an application, similar to a journal that records all activities. Instead of storing logs in files, a twelve-factor app sends logs directly to stdout (standard output) so that they can be monitored directly by developers during development.

When the application runs in staging or production environments, logs from all processes are collected by the execution environment and sent to a final storage location for viewing or archiving. These logs can then be analyzed further to understand the application’s behavior over time.

Benefits:

• Visibility and Monitoring: Provides direct visibility into the application’s behavior.
• Ease of Analysis: Allows for deep analysis to find specific events or see trends over time.
• Active Alerts: Can set alerts based on certain conditions, helping to quickly respond to issues.

Incorrect Example:

• Writing Directly to Files: The application tries to write logs to its own files and manage those files.
• Not Capturing Logs: Logs are not collected or analyzed, making it hard to monitor and understand the application’s behavior.

Correct Example:

• Local Development: Developers view logs directly in the terminal to monitor the application’s behavior during development.
• Production: Logs are collected and directed to log analysis systems like Splunk for long-term monitoring and analysis.

12. Admin Processes

Run admin/management tasks as one-off processes

In an application, there are times when specific tasks need to be performed just once, such as updating database structures or fixing incorrect data. These tasks should be run in the same environment as the main application to ensure consistency.

Examples include:

• Database Migrations: Running commands like manage.py migrate in Django or rake db:migrate in Rails.
• REPL Shell: Running an interactive console to inspect the application’s models directly against the live database. Most languages provide a REPL by running the interpreter without arguments (e.g., python or perl), or with specific commands (e.g., irb for Ruby or rails console for Rails).
• One-time Script: Running one-off scripts committed to the application’s repository (e.g., php scripts/fix_bad_records.php).

All these tasks should run in the same environment as the main application, using the same code and configuration. For instance, if the Ruby web application runs with bundle exec thin start, the database migration should also use bundle exec rake db:migrate.

Benefits:

• Consistency: Ensures all processes run in the same environment, avoiding synchronization issues.
• Ease of Use: Simplifies running admin tasks without changing the application environment.
• Flexibility: Allows developers to run one-off tasks whenever needed, both in development and production environments.

By understanding and applying these principles, we can ensure that admin and management tasks are executed consistently and reliably, maintaining the application’s integrity and stability.

Conclusion

The Twelve-Factor App methodology provides comprehensive guidelines for building scalable, portable, and manageable applications. By adhering to these principles, developers can ensure that their applications can adapt to various environments, support continuous development, and avoid many common issues that often arise during application development. Understanding and implementing each of these factors will result in more robust, easier-to-deploy applications, ready to scale and evolve as needed. Implementing this methodology is not just about following rules but adopting best practices compiled by experts and proven effective in real-world scenarios, thus ensuring the quality and reliability of the built applications.

Thank you for reading. Happy learning 😁

Source: 12factor.net

Support me here 😊