Say Goodbye to CI/CD: Rethinking Pipelines for Modern Applications

Source: iStockphoto/rmatejmo

In today’s rapidly evolving software landscape, the traditional approaches to Continuous Integration and Continuous Deployment (CICD) are facing a reckoning. The cloud-first era demands a paradigm shift, and this blog post introduces an innovative CICD approach tailored for cloud and containerized environments. This new model addresses the limitations of traditional CICD, focusing on adaptive, efficient, and cloud-native practices to enhance build and release processes.

Rethinking Traditional CICD

The conventional single-pipeline monolithic CICD models have served us well, but they falter in cloud-centric settings. They struggle to handle the diverse and dynamic needs of modern applications, especially in cloud and container technologies. As a result, we often end up with complex, hard-to-maintain pipelines that don’t integrate well with various cloud services and containerized architectures.

The Need for Change

Recognizing these challenges, we propose a new CICD model that emphasizes modularity, flexibility, and cloud-native practices. This model, termed ‘Build & Release Pipelines’, is designed to streamline the build and release process in cloud environments and for containerized applications. It treats artifacts and environments as first-class citizens, fostering rapid, high-quality software delivery.

The new Build & Release Pipelines model presents a significant evolution in the CICD paradigm, particularly in how it addresses the principle of separation of concerns compared to traditional CICD models. Here’s a breakdown of this distinction:

Traditional CICD Model: Limited Separation of Concerns

Monolithic Pipeline: In traditional CICD, we often see a single, monolithic pipeline handling both integration and deployment tasks. This can lead to a complex and inflexible setup, where changes in one part of the pipeline can impact others, making maintenance and scaling challenging.

Tight Coupling: Components like build, test, deployment, and environment configuration are closely intertwined. This tight coupling means that adjustments for specific environments or deployment strategies can become cumbersome.

Limited Adaptability: With the pipeline stages being closely linked, adapting to different types of applications (like microservices vs monoliths) or different deployment environments (cloud, on-premises, hybrid) can be less efficient.

New Build & Release Pipelines Model: Enhanced Separation of Concerns

This model clearly distinguishes between the Build and Release Pipelines. The Build Pipeline focuses solely on integrating and preparing software artifacts. In contrast, the Release Pipeline is dedicated to the deployment and management of these artifacts across various environments.

Specialized Focus:

• Build Pipelines: Concentrate on producing consistent and reliable artifacts (Deployable Objects, Configuration Data, etc.). This process is isolated from the complexities of deployment environments.
• Release Pipelines: Focus on the deployment and operational aspects, such as environment-specific configurations, deployment strategies, and post-deployment verification. This separation allows for more targeted optimizations and adaptations in each pipeline.
• Increased Flexibility: By decoupling build and release activities, the new model allows for greater flexibility. Different applications or services can have tailored Build and Release Pipelines that suit their specific requirements without affecting each other.
• Scalability and Maintenance: This approach simplifies scaling and maintaining pipelines. Changes in the deployment process or environment configurations don’t require modifications in the build process, and vice versa. This leads to more manageable and less error-prone processes.
• Enhanced Traceability and Accountability: With a clearer separation, tracking changes, pinpointing issues, and assessing the impact of modifications become more straightforward. Each pipeline can be audited and optimized independently, leading to better overall performance and reliability.

The New Model Explained

• Build Pipelines (CI): The focus here is on efficiently producing artifacts like Deployable Objects, Configuration Templates, SBOM, and Deployment Processes. These elements form an immutable set known as an Artifact Bundle, which can be deployed consistently across environments. This approach ensures traceability and reliability, with each Artifact Bundle being versioned and metadata-rich.
• Stages in the Build Pipeline: The Build Pipeline includes Creation, Validation, Publication, and Reporting stages, each customizable to project requirements.
• Release Pipelines (CD): In this event-driven workflow, the efficient release of Artifact Bundles is paramount. Triggered by successful builds, approvals, or scheduled events, the Release Pipeline promotes Artifact Bundles across development, staging, and production environments, ensuring software readiness at each stage through automated checks or manual approvals.
• Enhanced Environments: In our model, environments are a first-class citizen. Each environment undergoes four stages: Render, Deploy, Verify, and Report. This structured approach ensures that deployments are not only efficient but also secure and reliable.

Build Pipelines (CI)

Efficient Production of Artifacts

Focus: Build Pipelines are geared towards producing Deployable Objects, Configuration Templates, SBOM, and Deployment Processes. These components form an Artifact Bundle, which is immutable and can be consistently deployed across different environments.

Stages: The pipeline consists of Creation, Validation, Publication, and Reporting stages. Each stage is fine-tuned to meet specific project requirements.

Artifact Bundles: These are versioned sets, including semantic versioning and detailed metadata for enhanced tracking and auditing. This ensures both traceability and reliability in the software development process.

Build Pipeline Logical View/Robert Kozak

Below is an example Build Pipeline in github actions:

name: Build Pipeline

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  build_and_publish:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout Repository
      uses: actions/checkout@v2

    # AWS Credentials Setup
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: your-aws-region

    # Creation Stage
    - name: Build Docker Image with Artifact Bundle
      run: |
        # Retrieve configuration from AWS Parameter Store
        # Retrieve secrets from AWS Secrets Manager
        # Include these configurations in the Docker image
        docker build -t myapp:${{ github.sha }} .

    # Validation Stage
    - name: Run Trivy Vulnerability Scanner
      run: trivy image myapp:${{ github.sha }}

    - name: Run Tests
      run: docker run myapp:${{ github.sha }} npm test

    # Publication Stage
    - name: Push to Artifactory
      run: |
        docker tag myapp:${{ github.sha }} ${{ secrets.ARTIFACTORY_URL }}/myapp:${{ github.sha }}
        docker push ${{ secrets.ARTIFACTORY_URL }}/myapp:${{ github.sha }}

    # Reporting Stage
    - name: Send Logs
      run: |
        echo "Send logs to your log provider"

Release Pipelines (CD)

Efficient and Safe Release of Artifacts

Workflow: The Release Pipeline is an event-driven process, ensuring the safe and efficient release of Artifact Bundles. It is triggered by events such as successful builds, manual approvals, or scheduled releases.

Promotion Model: This pipeline employs a promotion model through different environments — development, staging, and production. It incorporates a mix of automated checks and manual approvals to ensure the thorough evaluation and readiness of software at each stage.

Environment Stages: In this model, environments undergo four key stages: Render, Deploy, Verify, and Report. This structure ensures deployments are efficient, secure, and reliable.

Example Release Pipeline for microservice deployment/Robert Kozak

Environments: A Key Component in the New Model

In the Build & Release Pipelines model, environments are elevated to a central role, reflecting the diversity and complexity of modern deployment scenarios. This focus brings several advantages, particularly in terms of separation of concerns and operational efficiency.

First-Class Citizens: Unlike traditional CICD models where environments are often an afterthought or merely settings within a pipeline, in this new model, environments are treated as first-class citizens. This means each environment is distinctly recognized and managed, with its own specific configurations and requirements.

Environment Specificity: The model allows for each environment (development, staging, production, etc.) to have its own unique workflow and settings. This is essential for cloud-first and containerized deployments where environments can vary significantly in terms of scale, security requirements, and other factors.

Scalability and Flexibility: By treating environments as integral parts of the pipeline, scaling up or modifying deployment strategies becomes more streamlined. Each environment can evolve independently, adapting to new requirements or technologies without disrupting the overall pipeline.

Improved Reliability and Security: With clear separation and dedicated stages, each environment can be more rigorously tested and secured. This approach minimizes the risk of deployment errors and security vulnerabilities, particularly important in complex, multi-environment deployments.

Stages Within Environments: Each environment in the Release Pipeline goes through four distinct stages — Render, Deploy, Verify, and Report. This structured approach ensures that deployments are not only efficient but also secure and reliable.

• Render: This stage involves updating templates and configurations with environment-specific values. It ensures that the right settings are applied for each environment, enhancing the adaptability of the deployment process.
• Deploy: The actual deployment of code to the environment happens here. This stage is where the prepared artifacts from the Build Pipeline are released into the environment.
• Verify: Post-deployment, this stage conducts tests and scans to ensure everything is working correctly. It’s crucial for maintaining the reliability and stability of the application in the new environment.
• Report: The final stage updates deployment statistics, application metrics, version numbers, and more. This information can be sent to various platforms like Slack, Jira, and dashboards, providing transparency and insights into the deployment process.

Robert Kozak

Conclusion

In conclusion, the evolution from traditional CICD to the innovative Build & Release Pipelines model marks a significant step forward in addressing the complex demands of modern, cloud-centric software development. This new model transcends the limitations of the monolithic, tightly-coupled pipelines of the past, offering a modular, flexible, and environment-centric approach that aligns perfectly with the dynamics of cloud and container technologies.

Key Advantages of the New Model:

• Enhanced Modularity and Flexibility: By separating the Build and Release Pipelines, we enable greater specificity and adaptability in handling diverse software architectures and deployment environments. This separation not only simplifies the process but also allows for specialized optimizations in each pipeline.
• First-Class Treatment of Environments: Elevating environments to a central role in the pipeline reflects the intricate nature of modern deployment scenarios. This approach allows for more effective management of environment-specific configurations and requirements, leading to improved scalability, reliability, and security.
• Increased Efficiency and Reliability: The structured stages within each pipeline and environment ensure efficient and secure deployments. The clear demarcation of responsibilities enhances traceability, reduces error rates, and ensures that each part of the process is optimized for its specific purpose.
• Better Adaptation to Cloud-First Strategies: The new model is inherently designed for cloud and containerized deployments, ensuring that the pipelines are not only robust but also agile enough to embrace the evolving landscape of cloud technologies and practices.

By adopting the Build & Release Pipelines model, organizations can expect a transformative improvement in their software delivery processes. This model not only aligns with current technological trends but also anticipates future shifts, ensuring that businesses remain at the forefront of software development practices. As we bid farewell to the traditional CICD approaches, we welcome a new era of efficiency, adaptability, and excellence in continuous integration and deployment.