Python DevSecOps YAML Pipeline on Azure DevOps

Introduction:

DevSecOps is a development practice that integrates security at an early stage (shift left) of the software development lifecycle to deliver robust and secure applications and save looping and re-work time. In this article we’re going to build a YAML pipeline on Azure which includes the following:

1. Build and push a Dockerized Django app.
2. Run SCA (Software Composition Analysis) and SAST (Static application security testing) scans.
3. Run automated Unit tests.
4. Deploy the Dockerized app to a staging Kubernetes cluster.
5. Run DAST (Dynamic application security testing) scan and Integration tests on the deployed app.
6. Upload the scan results to DefectDojo vulnerability management platform.
7. Deploy the Dockerized app to a production Kubernetes cluster.

for the sake of showing how security scans are performed and how they generate reports I used an “intentionally vulnerable” Django application known as “pygoat”, I’ve forked the repo, done some fixes and added my code which you can find in the link below,

BasharDlaleh/pygoat-devsecops-github

Let’s get down to business!

We’ll start off by building the project:

trigger:
- main

pool: 
  vmImage: ubuntu-latest

variables:
  tag: "$(Build.BuildId)"
  image: "bashar1993/pygoat"

stages:
 
- stage: build
  jobs:
    - job: build_and_push_app
      displayName: Build and Push App
      steps:

      - task: UsePythonVersion@0
        inputs:
          versionSpec: '3.8'
          addToPath: true
          architecture: 'x64'
        displayName: 'Setup Python'

      - script: |
          pip install -r requirements.txt
        displayName: 'Install Dependencies'
      
      - task: Docker@2
        inputs:
          containerRegistry: 'dockerhub'
          repository: '$(image)'
          command: 'buildAndPush'
          Dockerfile: '**/Dockerfile'
        displayName: 'Build and Push Docker image'

Now that our image is pushed to our repo, we can add the next stage, which is the test stage, we’ll start with the SCA scan which usually scans your project’s dependencies and for that we’ll use a python package called Safety which does that by pointing it to the requirements.txt file,

- stage: test
  dependsOn: build
  jobs:
    
    - job: run_sca_analysis
      displayName: Run SCA Analysis
      steps:
      
      - task: CmdLine@2
        inputs:
          script: |
            pip install safety
            safety check -r requirements.txt --continue-on-error --output json > $(Pipeline.Workspace)/dependency-check-report.json
        displayName: 'Safety Dependency Check'

the — continue-on-error flag is important in many scanning tools, so the pipeline doesn’t fail even if some vulnerabilities were found by the scanner.

After the scan is complete, we need to publish the scan report to the pipeline artifacts so we can use it in a later job,

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Pipeline.Workspace)/dependency-check-report.json'
    artifact: 'dependency-check-report'
    publishLocation: 'pipeline'

In the SCA job we should also scan the container image we built for vulnerabilities because container images also have their own dependencies just like our code, for that we use Docker Scout which is an official tool from Docker for scanning images,

- task: CmdLine@2
  inputs:
     script: |
       # Install the Docker Scout CLI
       curl -sSfL https://raw.githubusercontent.com/docker/scout-cli/main/install.sh | sh -s --
       # Login to Docker Hub required for Docker Scout CLI
       docker login -u $(DOCKER_HUB_USER) -p $(DOCKER_HUB_PAT)
       # Get a CVE report for the built image and fail the pipeline when critical or high CVEs are detected
       docker scout cves $(image):$(tag) --only-severity critical,high --format sarif --output $(Pipeline.Workspace)/image-scan-report.json
   displayName: Image Scanning

- task: PublishPipelineArtifact@1
  inputs:
     targetPath: '$(Pipeline.Workspace)/image-scan-report.json'
     artifact: 'image-scan-report'
     publishLocation: 'pipeline'
  condition: succeededOrFailed()
  displayName: 'Publish Image Scan Report'

Now we will add another job in the test stage to run our unit tests using python pytest package and of course we’ll also publish the test results,

- job: run_unit_tests
  displayName: Run Unit Tests
  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.8'
      addToPath: true
      architecture: 'x64'
    displayName: 'Setup Python'

  - script: |
      pip install -r requirements.txt
    displayName: 'Install Dependencies'

  - script: |
      python -m pip install pytest-azurepipelines pytest-cov
      python -m pytest introduction/tests/unit/ --junitxml=$(Pipeline.Workspace)/TEST-output.xml --cov=. --cov-report=xml
    displayName: 'UnitTests with PyTest'
      
  - task: PublishPipelineArtifact@1
    inputs:
      targetPath: '$(Pipeline.Workspace)/TEST-output.xml'
      artifact: 'unit-test-results'
      publishLocation: 'pipeline'
    condition: succeededOrFailed()
    displayName: 'Publish UnitTest Report'

Next, we’ll run SAST scan which scans our source code for vulnerabilities, we’ll use a popular python package called Bandit to do that,

- job: run_sast_analysis
  displayName: Run SAST Analysis
  steps:
  - task: CmdLine@2
    inputs:
      script: |
        pip3 install --upgrade pip
        pip3 install --upgrade setuptools
        pip3 install bandit
        bandit -ll -ii -r ./introduction -f json -o $(Pipeline.Workspace)/sast-report.json --exit-zero
    displayName: 'Bandit Scan'
      
  - task: PublishPipelineArtifact@1
    inputs:
      targetPath: '$(Pipeline.Workspace)/sast-report.json'
      artifact: 'bandit-sast-report'
      publishLocation: 'pipeline'
    condition: succeededOrFailed()
    displayName: 'Publish SAST Scan Report'

After all scans have been done, and because it’s hard to check and manage all results in the pipeline, we’ll upload the results to a popular vulnerabilities management platform called DefectDojo where you can manage your vulnerabilities in one place,

You can either use DefectDojo paid SAAS or host your own instance.

DefectDojo/django-DefectDojo at dev (github.com)

DefectDojo works by creating an engagement which represents a pipeline run and then uploading the reports to that engagement, will do that by calling its API,

- job: upload_reports
  dependsOn: 
  - run_sast_analysis
  - run_unit_tests
  - run_sca_analysis
  displayName: Upload Reports
  variables:
    DEFECTDOJO_ENGAGEMENT_PERIOD: 7
    DEFECTDOJO_ENGAGEMENT_STATUS: "Not Started"
    DEFECTDOJO_ENGAGEMENT_BUILD_SERVER: "null"
    DEFECTDOJO_ENGAGEMENT_SOURCE_CODE_MANAGEMENT_SERVER: "null"
    DEFECTDOJO_ENGAGEMENT_ORCHESTRATION_ENGINE: "null"
    DEFECTDOJO_ENGAGEMENT_DEDUPLICATION_ON_ENGAGEMENT: "false"
    DEFECTDOJO_ENGAGEMENT_THREAT_MODEL: "true"
    DEFECTDOJO_ENGAGEMENT_API_TEST: "true"
    DEFECTDOJO_ENGAGEMENT_PEN_TEST: "true"
    DEFECTDOJO_ENGAGEMENT_CHECK_LIST: "true"
    DEFECTDOJO_NOT_ON_MASTER: "false"
    DEFECTDOJO_PRODUCTID: 1
    DEFECTDOJO_SCAN_MINIMUM_SEVERITY: "Info"
    DEFECTDOJO_SCAN_ACTIVE: "true"
    DEFECTDOJO_SCAN_VERIFIED: "true"
    DEFECTDOJO_SCAN_CLOSE_OLD_FINDINGS: "true"
    DEFECTDOJO_SCAN_PUSH_TO_JIRA: "false"
    DEFECTDOJO_SCAN_ENVIRONMENT: "Default"
    DEFECTDOJO_ANCHORE_DISABLE: "false"
    DEFECTDOJO_SCAN_TEST_TYPE: "SAST and SCA Scan"
  steps:
  - task: DownloadPipelineArtifact@2
    inputs:
      buildType: 'current'
      itemPattern: '**'
      targetPath: '$(Pipeline.Workspace)'
    displayName: 'Download Reports From Pipeline Artifacts'
        
  - task: CmdLine@2
    inputs:
      script: |
        TODAY=`date +%Y-%m-%d`
        ENDDAY=$(date -d "+$(DEFECTDOJO_ENGAGEMENT_PERIOD) days" +%Y-%m-%d)
        ENGAGEMENTID=`curl --fail --location --request POST "$(DEFECTDOJO_URL)/engagements/" \
          --header "Authorization: Token $(DEFECTDOJO_TOKEN)" \
          --header 'Content-Type: application/json' \
            --data-raw "{
              \"tags\": [\"AZURE-DEVOPS\"],
              \"name\": \"pygoat-$(System.DefinitionId)\",
              \"description\": \"$(Build.SourceVersionMessage)\",
              \"version\": \"$(Build.SourceBranch)\",
              \"first_contacted\": \"${TODAY}\",
              \"target_start\": \"${TODAY}\",
              \"target_end\": \"${ENDDAY}\",
              \"reason\": \"string\",
              \"tracker\": \"$(Build.Repository.Uri)/\",
              \"threat_model\": \"$(DEFECTDOJO_ENGAGEMENT_THREAT_MODEL)\",
              \"api_test\": \"$(DEFECTDOJO_ENGAGEMENT_API_TEST)\",
              \"pen_test\": \"$(DEFECTDOJO_ENGAGEMENT_PEN_TEST)\",
              \"check_list\": \"$(DEFECTDOJO_ENGAGEMENT_CHECK_LIST)\",
              \"status\": \"$(DEFECTDOJO_ENGAGEMENT_STATUS)\",
              \"engagement_type\": \"CI/CD\",
              \"build_id\": \"$(System.DefinitionId)\",
              \"commit_hash\": \"$(Build.SourceVersion)\",
              \"branch_tag\": \"$(Build.SourceBranch)\",
              \"deduplication_on_engagement\": \"$(DEFECTDOJO_ENGAGEMENT_DEDUPLICATION_ON_ENGAGEMENT)\",
              \"product\": \"$(DEFECTDOJO_PRODUCTID)\",
              \"source_code_management_uri\": \"$(Build.Repository.Uri)\",
              \"build_server\": $(DEFECTDOJO_ENGAGEMENT_BUILD_SERVER),
              \"source_code_management_server\": $(DEFECTDOJO_ENGAGEMENT_SOURCE_CODE_MANAGEMENT_SERVER),
              \"orchestration_engine\": $(DEFECTDOJO_ENGAGEMENT_ORCHESTRATION_ENGINE)
        }" | jq -r '.id'` &&
        echo ${ENGAGEMENTID} > ENGAGEMENTID.env
    displayName: 'Create DefectDojo Engagement'

  - task: CmdLine@2
    inputs:
      script: |
        TODAY=`date +%Y-%m-%d`
        ENGAGEMENTID=`cat ENGAGEMENTID.env`
        array=('type=("Dependency Check Scan" "SARIF" "Bandit Scan")' 'file=("dependency-check-report/dependency-check-report.json" "image-scan-report/image-scan-report.json" "bandit-sast-report/sast-report.json")')
        for elt in "${array[@]}";do eval $elt;done
        for scan in 0 1 2; do \
          curl --fail --location --request POST "$(DEFECTDOJO_URL)/import-scan/" \
            --header "Authorization: Token $(DEFECTDOJO_TOKEN)" \
            --form "scan_date=${TODAY}" \
            --form "minimum_severity=$(DEFECTDOJO_SCAN_MINIMUM_SEVERITY)" \
            --form "active=$(DEFECTDOJO_SCAN_ACTIVE)" \
            --form "verified=$(DEFECTDOJO_SCAN_VERIFIED)" \
            --form "scan_type=${type[$scan]}" \
            --form "engagement=${ENGAGEMENTID}" \
            --form "file=@$(Pipeline.Workspace)/${file[$scan]}" \
            --form "close_old_findings=$(DEFECTDOJO_SCAN_CLOSE_OLD_FINDINGS)" \
            --form "push_to_jira=$(DEFECTDOJO_SCAN_PUSH_TO_JIRA)" \
            --form "test_type=$(DEFECTDOJO_SCAN_TEST_TYPE)" \
            --form "environment=$(DEFECTDOJO_SCAN_ENVIRONMENT)"
        done
    displayName: 'Upload Reports To DefectDojo'

• in the first task we’re downloading the scan results artifacts we uploaded earlier.
• in the second task we’re using DEFECTDOJO_URL and DEFECTDOJO_TOKEN which you would normally add in the pipeline variables as secrets.
• in the third task we’re looping on the 3 scans we downloaded and uploading them one by one using curl.

Next, we add the deploy_test stage which deploys the app to a staging environment where we can run dynamic tests, this stage is usually run manually by an approver after reviewing the previous stages so we will add it as a deployment rather than a job,

# for this you need to create a test environment in azure pipelines and add approvers
- stage: deploy_test
  dependsOn: test
  jobs:
  - deployment: deploy_to_test_k8s_cluster
    displayName: Deploy To Test K8S Cluster
    environment: 'pygoat-test'
    strategy:
     runOnce:
       deploy:
         steps:
         - download: none  # this prevents the deployment from automatically downloading the published artifacts
         - checkout: self  # this forces the deployment to checkout the source code which doesn't happen by default in deployments like in jobs
         - task: replacetokens@5
           inputs:
             targetFiles: '$(System.DefaultWorkingDirectory)/k8s-*.yaml'
             encoding: 'auto'
             tokenPattern: 'azpipelines'
             writeBOM: true
             actionOnMissing: 'fail'
             keepToken: false
             actionOnNoFiles: 'continue'
             enableTransforms: false
             enableRecursion: false
             useLegacyPattern: false
             enableTelemetry: true
           displayName: 'Replace K8S Image Tag'

         - task: KubernetesManifest@1
           inputs:
             action: 'deploy'
             connectionType: 'kubernetesServiceConnection'
             kubernetesServiceConnection: 'Kubernetes'
             namespace: 'test'
             manifests: 'k8s-*.yaml'
           displayName: 'Deploy Manifests'
    
         - task: CmdLine@2
           inputs:
             script: 'sleep 30'
           displayName: 'Sleep For 30 Seconds'

In order for this stage to work you need to create a Kubernetes service connection in your Azure DevOps project settings which connects your pipeline to your cluster.

Kubernetes service connection

of course, the better choice here is to use a GitOps tool like ArgoCD to pull your yamls instead of exposing your cluster to the pipeline, but I chose to be lazy :)

Next, we add a new stage for DAST scan and Integration tests (we use Selenium) which both run against a running application in the testing environment,

- stage: dast
  dependsOn: deploy_test
  jobs:
  - job: run_integration_tests
    displayName: Run Integration Tests
    steps:
# for containerized apps you could run the DAST and Integration tests against a container running inside the pipeline itself instead of deploying to an external staging url    
    # - task: CmdLine@2
    #   inputs:
    #     script: 'docker run -d -p 8000:8000 $(image):$(tag)'
    #   displayName: 'Run Container Inside Pipeline'

    - script: |
        python -m pip install -r requirements.txt
        python -m pip install pytest-cov
        python -m pytest introduction/tests/integration/ --junitxml=$(Pipeline.Workspace)/selenium-test-output.xml --cov=. --cov-report=xml
      condition: succeededOrFailed()
      displayName: 'Integration Tests with Selenium'

    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: '$(Pipeline.Workspace)/selenium-test-output.xml'
        artifact: 'selenium-test-results'
        publishLocation: 'pipeline'
      condition: succeededOrFailed()
      displayName: 'Publish Selenium Report'

Now we add a new job for running DAST scan for which we use the popular OWASP ZAP scanner,

- job: run_dast_scan
    displayName: Run DAST Scan
    steps:    
    - task: owaspzap@1
      inputs:
        scantype: 'targetedScan'
        url: 'http://pygoat.example.com' # replace with your staging URL here
      displayName: 'OWASP ZAP Scan'
    
    - task: CopyFiles@2
      condition: succeededOrFailed()
      inputs:
        SourceFolder: 'owaspzap/'
        TargetFolder: '$(Pipeline.Workspace)'
    
    - task: PublishPipelineArtifact@1
      condition: succeededOrFailed()
      inputs:
        targetPath: '$(Pipeline.Workspace)/report.json'
        artifact: 'owasp_zap_report'
        publishLocation: 'pipeline'
      displayName: 'Publish ZAP Report'

we could also upload the ZAP results to DefectDojo just like we did before.

Finally, the last stage is deploy_prod in which you deploy your packaged app to the production Kubernetes cluster,

# for this you need to create a production environment in azure pipelines and add approvers
- stage: deploy_prod
  dependsOn: dast
  jobs:
  - deployment: deploy_prod
    displayName: Deploy To Prod K8S Cluster
    environment: 'pygoat-prod'
    strategy:
     runOnce:
       deploy:
         steps:
         - download: none
         - checkout: self
         - task: KubernetesManifest@1
           inputs:
             action: 'deploy'
             connectionType: 'kubernetesServiceConnection'
             kubernetesServiceConnection: 'Kubernetes'
             namespace: 'default'
             manifests: 'k8s-*.yaml'
           displayName: 'Deploy Manifests'

That’s it, you have a full CI/CD DevSecOps pipeline.

Note that for SCA and SAST scans many people prefer to use popular tools like Snyk and SonarQube which both work well with many programming languages/frameworks and integrate easily with Azure DevOps using service connections, but I chose to use python packages to keep it simple and because using a tool designed specifically for your stack is sometimes a better choice than using a general-purpose tool, but that totally depends on your use case and your tech stack!