Enterprise Cloud Security: Application Development

Part III: Developing secure applications

This article is part of Enterprise Cloud Security Series with Part I: Introduction introducing the space and how it differs from on-premise security. Part II: Foundation covers the security consideration for building the cloud foundation. Part III: Application Development focuses on building security into application development process.

Cloud security reference architecture

The security of application can, at a high level, be classified in to the following

1. Application is being developed with security in mind and is secure
2. Application development infrastructure is secure
3. Application is secure while running
4. Application runtime infrastructure is secure.

Developing a secure application requires that application is designed and developed to be secure on a secure development platform (Reflections on Trusting Trust).

Application security in DevOps

CNCF has done a great job of putting together security white-paper which is based on NIST Application Security Container Guide, Center for Internet Security (CIS), NIST Security Strategies for Micro-service and OpenSCAP. It covers the typical lifecycle of development, distribution, deployment and runtime of applications. Due to its focus on cloud-native development, it is a bit light on some aspects like unique considerations for securing SaaS and PaaS services.

Secure DevOps

The overall approach of the DevOps is to ensure that security is achieved as the following types of artifacts flow through the process

1. Application — represents a collection of all the artifacts that are part of the runtime.
2. Configuration — consists of SaaS, PaaS, IaaS, and application configuration that drives the provisioning and runtime behavior of the application.
3. Code — primarily deployed to IaaS (Infrastructure as a Service), CaaS (container as a service), FaaS (Function as a Service), and PaaS (Platform as a Service) as part of the application
4. Image — provides the platform on which the code runs. This typically includes VM and container images.

As part of the DevOps, the following stages form the part of the process

1. Develop— involves the developers writing code, configuration, and sometimes putting together the image on which the solution is going to run. Developers are also responsible for testing and may also be either responsible for or involved in threat modeling given their understanding of the domain and technology involved.
2. Distribute — focuses on packaging the outcome of the development process to either use it within or share it with other teams, business units, and organizations for use. This typically involves more comprehensive testing to ensure that artifacts are “production-ready” for release. The packaged artifacts are typically pushed to the registry for staging purposes.
3. Registry staging — involves storing the artifacts for future use by downstream systems and users.
4. Deployment — focuses on setting up the application in a landing zone designated for the application.
5. Runtime — consists of an application and associated infrastructure that provides the service that the application was designed for.

Across these stages, the various aspects of security must be monitored and addressed to ensure that the application is “secure-by-default”. This “shift-left” approach to application security can reduce the time, effort, and cost of fixing these security issues during runtime. The following aspects of security are typically addressed as part of DevOps processes.

Application

1. Threat modeling — focuses on identifying, communicating, and understanding threats and mitigations within the context of protecting something of value i.e. data and/or service. This is an important part of the development process to provide appropriate guidance for users to deploy applications securely. During the deployment phase, threat modeling is typically performed to identify all the runtime threats based on the deployment architecture.
2. Code review — the changes to code, configuration, and images should be reviewed to ensure that it aligns with functional and non-functional requirements and enterprise guidelines.
3. Testing — ensure that the code, configuration, and images are aligned with functional and non-functional (including security) requirements. Depending on the requirements, the testing would typically involve unit testing, system testing, integration testing, performance testing, and others during the entire development lifecycle. With regards to security, penetration testing is one of the typical testing performed to ensure that implementation is secure by design. Security teams may develop automated smoke tests that can be incorporated into testing framework to ensure adequate coverage.
4. Third-party artifact analysis — looks at all the third-party artifacts like VM and container images, libraries that are pulled from external sources like maven repositories, docker container repository, and cloud marketplace and performs vulnerability analysis, signing and publishing to enterprise registry for use by application teams.
5. Security policy validation — involves checking all the components that form part of the runtime environment including code, configuration, image, network against pre-defined enterprise policies to ensure that the application is secure after deployment.
6. Log Flow validation — ensures that all the application and infrastructure logs are set up correctly to flow into the SIEM and/or performance/ telemetry platform after deployment.
7. Resiliency validation — focuses on verifying whether the deployed application is deployed such that it is aligned with the resiliency (failover, disaster recovery) requirement identified.
8. Forensic validation — ensures that deployed infrastructure is setup appropriately (e.g. installation of any agent, access to the environment from forensic automation tools) to quickly perform evidence collection for forensic investigations in case of any incident.

Image

1. Docker file vulnerability scan — focuses on identifying the various artifacts that form part of each layer of container images and validating the vulnerability associated with artifact.
2. Kubernetes manifest scan — looks for security and configuration issues with Kubernetes workload.
3. VM Image vulnerability scan — looks for potential points of exploits in the VM image.
4. Image hardening — strips unnecessary software and changes configuration to limit potential vulnerabilities that can be exploited by attackers.
5. Image signing and validation— adds digital fingerprint to the image which can be tested later to verify trust.
6. Public registry access — needs to be managed to ensure that unapproved or insecure images or image layers are not pulled in as part of any of the phases.
7. Marketplace restriction — ensures that images being developed are approved or aligned with enterprise policies
8. Image encryption — ensures that any intellectual property stored on the image is protected.

Code

1. Static Analysis (SAST) — focuses on a white box testing by analyzing source code before it is compiled to find vulnerabilities that may be exploitable at runtime depending on the deployment architecture and other factors
2. Dependency License Scan — reviews all the dependency associated with code to ensure that corresponding licenses align with enterprise policies and limit exposure.
3. Artifact and package signing — adds digital fingerprint to packages and artifacts created as part of compilation and packaging processes.
4. Dependency vulnerability scan — identifies any existing exploit associated with dependencies that are being used by code.
5. External library access — ensures that only authorized libraries are used.
6. Public Repo access scan — looks for references to public and unauthorized repositories in the code, configuration, and build process.
7. Pre-defined code templates — developed and validated for security to reduce reinvention of boiler-plate security code.

Configuration

1. Infrastructure as code scan— IaaC (Infrastructure as a Code) focuses on defining (and provisioning) infrastructure using a configuration or code. The configuration (e.g. cloud formation, terraform, azure ARM, GCP deployment manager) can be scanned using predefined and custom policies to ensure that infrastructure is secure (e.g. disk is encrypted, logs are enabled, access is configured, etc) and configured correctly.
2. Pre-defined configuration templates — developed and validated by security team across Infrastructure as Code, approved framework and libraries can reduce the possibility of incorrect configuration seeping in.
3. Runtime policy validation — is the runtime equivalent of IaaC where the same policies are validated against the runtime to ensure that the deployed infrastructure aligns with security and business policies.
4. Shared secret scanning — looks for shared secret, keys, private certificates in registry.

DevOps platform security

Security of the DevOps platform consists of securing all the technical components, enterprise processes with the idea that resistance to developers’ flow increases as the application moves from development to production. Each enterprise has to find the right balance of the developer’s freedom with security and operational controls needed to ensure the achievement of business objectives.

Sample CI/CD flow

DevOps consists of different components that need to be secured.apply the V.L.A.D.R. criteria to structure the approach below.

Code repository/Source Control Management

Depending on the delivery model (IaaS, PaaS, or SaaS) and applicable shared responsibility, the enterprise may be responsible for managing the basic platform security aspects like technology vulnerabilities, logging, and resiliency. Code repository poses a unique set of security challenges like process vulnerabilities, access, and data which should be considered appropriately.

Code repository is a type of database since it contains business and trade secrets, algorithms that can result in a loss if exposed. Keeping that in mind, code repositories should be handled similarly to data within the enterprise. This would typically involve separating the code repository based on sensitivity. Organization policies to secure data at rest by encryption with organization managed keys may mean that SaaS code repository can not be used for sensitive code.

Access to the code repository forms another important part of the process. Strong controls with regards to user’s access should be adequately be supplemented by control over identities (e.g. API keys) and access of scripts and other automation processes (e.g. automatic key rotation, service identity for authentication). It is important to ensure that API keys and similar tokens have same level of access control as user/service accounts (e.g. SAML authentication may enforce device authentication which may be missed by API keys). All the access should be based on the principle of least privilege with, where possible, use of UEBA (User & Entity Behavior Analysis) to continuously fine-tune the access and with periodic access reviews. In addition to that privileged shared/service accounts like root or administrator accounts should be on-boarded to privileged access management system to have better control over people who have access to them and monitor the operations performed. Additional network access control may be required to be put in place for sensitive repositories to reduce the attack surface.

Code repository in most of the cases also stores other aspects of the DevOps (e.g. CI/CD pipeline definition, application configuration, SAST/DAST configuration, vulnerability exceptions) which can become a source of process vulnerabilities. It is important to ensure that appropriate pre-commit hooks, scanning processes are in place to detect changes to CI/CD pipeline that circumvent enterprise processes, expose shared secrets/API keys/private certificates through code base, ignore false-positive vulnerability without exception, use build and release tasks that are not standard/authorized and so on. The enterprise should review the developers’ toolsets and use-cases to identify these process vulnerabilities and deploy controls to address them.

Registry

Besides basic platform security aspects like technology vulnerabilities, logging, and resiliency that enterprise may be responsible for depending on the delivery model and corresponding shared responsibility, registry specific process vulnerabilities and access model along with data continue to be the focus.

Depending on the degree of control in place, applications may not have access to a public repository during the development and distribution phase. The repository may operate as a proxy for external dependencies like libraries, VM Images, container images, and tool plugins (e.g. approved terraform providers). In addition to that additional controls may be in place to scan the external dependencies and generated artifacts for vulnerabilities, sensitive information.

Access to the registry is another aspect that plays an important role. This access can be of significant importance when the repository operates as a proxy to public repositories. The ability to add or remove the external repository allowed can have a significant impact.

Most registries also support the ability to sign and encrypt the artifacts which provide data-level security.

CI/CD Automation

CI/CD automation platform provides a way to execute the build process and deploy the artifacts. This platform needs to ensure that the process being executed is aligned with enterprise policies (e.g. application should not have critical severity vulnerability before deployment in production) and deployment is being with least privilege access. In addition to that the build and packaging tools should be from trusted sources and publishers (including extensions like terraform provider plugins) to ensure that artifacts being generated are not compromised during the build process. Due to the involvement of secrets/keys, it is important that automation is executed on the platform with limited accessibility and with an adequate level of encryption of data at rest and in motion. In addition to that adequate environment hygiene (e.g. delete all the files after build, encrypt and store state files using BYOK on alternate storage system) should be maintained during and after execution of build and deployment process to ensure that details about the application including secrets/service account passwords are not present on the file system after the CI/CD pipeline has been executed. Securing CI/CD runner is explored in detail in the article here.

Adding security to development and operational processes in any enterprise can be a significant daunting task which is further exacerbated by the changes in culture within most enterprises. Maintaining this balance between developer’s choice and enterprise security posture can be a significant challenge with the rollout of DevOps within organizations.

Changes
1. Added reference to difference between SAML for user authn vs API key
2. Added link to CI/CD agent security
3. Added reference to use of unauthorized tasks in CI/CD pipeline