Decathlon’s path towards container security

This post covers Decathlon’s views on container security, the main challenges associated with it, and possible ways to tackle them.

A little bit of context

Since Docker’s initial release in 2013, containers have had a huge growth as an IT technology, to the point where Gartner estimates that within 2022, 75% of global organizations will run containerized applications in production¹. With an increasing number of teams adopting the DevOps philosophy, orchestration is the logical next step on the path to a containerized world.
When discussing container orchestration, it only takes a few seconds before Kubernetes name is uttered. Kubernetes has indeed positioned itself as the go-to solution on the matter². Mentioning Kubernetes as such might convey the misleading idea that its usage is uniform across teams or companies. A look at the CNCF landscape shows that one can not be further from the truth. The Kubernetes ecosystem is abundant. Depending on your choice of distribution (often linked to your cloud-provider) and on the tools you deploy, your Kubernetes installation may vary dramatically from another.
The quick adoption rate of containers, the ease of deployment they provide, the complexity of Kubernetes itself, the diversity of the tools composing its ecosystem are so many factors making the securing of container workloads a vital and exciting challenge for today’s security teams.

Here are Decathlon’s takes on the matter.

An emergent and complex problematic

Container security as a subset of cloud security

Following our full-cloud program³, we massively adopted containers as the main deployment technology for compute workloads. As such, our container security posture is deeply entangled with our global Cloud Security Posture⁴, and both have to be addressed together.

A need of visibility

A Cloud Workload Protection Platform (CWPP) reveals its full potential in a multi-cloud environment since using each cloud provider security solution lacks standardisation. A third-party CWPP brings an aggregated, centralised view on workloads.

Doing so, it allows security teams to shed some light on the issues to remediate, and provides educational and actionable insights to the operational teams.

A matter of governance and prioritization

Although automation lies at the heart of what we build at Decathlon, humans are still one of the pillars of our company. It is thus our job as security teams to define the framework within which our colleagues will use containers.

Writing an Information System Security Policy (ISSP) with a focus on containers establishes a set of rules for everyone to follow. For that, security standards⁵ can help. Eventually, sharing a RACI matrix clarifies the different responsibilities throughout the organization.

Once those rules are in place, a CWPP brings visibility to the level of container security and enables the teams to prioritize the actions taken to improve it.

Where to start? Let’s dive in!

Rely on your cloud providers

Using managed Kubernetes services (GKE, EKS, AKE, etc.) presents the benefit of delegating part of the security to the cloud provider, mainly for the control-plane and for nodes. For those components, and to mitigate the associated risks, our responsibility then falls down to regularly updating our distribution, and to enforcing comprehensive hardened configurations on it.

Tighten your RBACs

Too large a set of permissions would allow an attacker to successfully get knowledge of a cluster usage or worst deploy an attack. Reviewing Roles and ClusterRoles is the first step to ensure each user has privileges strictly limited to what they are supposed to do. Use short-lived credentials for users, and fine-grained permissions for service accounts in order to further tighten possibilities.

Rework your images and their registry

Following the principle “Build it once, run it everywhere”, Docker images are used at every step, from development to production. Therefore, working on their security has a broad impact.

On the other hand, an overly permissive Docker image can give a significant edge to an attacker.
Here is a non-exhaustive list of what can be done in an attempt to mitigate risks:

• Use the slimmest base image possible;
• Refine write access to your registry: disable manual push to enforce using your CI/CD process to make an image available;
• Scan your images at build-time for vulnerabilities;
• Sign your images;
• Limit the deployable images: restrict deployable images to a verified, secured, signed set of images;
• Ensure HTTPS-only exposition of your registry;
• Limit the pull possibilities from your registry: use your cloud providers’ features⁶, or a Kubernetes secret⁷ for authentication and authorisation.

Watch and restrict your runtimes

CWPP provides you with a view of what is currently happening on your clusters, alerting you in case of privilege escalation, suspicious binaries being run within images, reverse shells, etc.

Being alerted is the first step to remediation.

You can also rely on Kubernetes features in order to set additional boundaries earlier on.

Pod Security Policies — or the proposed replacement⁸ once they are removed⁹ — allow you to:

• Prevent privilege escalation within a Pod;
• Use a read-only file system in containers;
• Prevent containers from running as root;
• Limit kernel capabilities.

Isolate your workloads

Limiting the surface of an attack is one way to mitigate its impact. One way to achieve that within a Kubernetes cluster is to use NodeSelectors, Namespaces, and NetworkPolicies.

• NodeSelectors would allow you to schedule targeted workloads on specific nodes;
• Namespaces provide logical separation between workloads (allowing finer-grained RBAC);
• NetworkPolicies setup communication rules between the different deployed resources.

Using a combination of the three makes up a good solution to limit the impact of an attack.

Involve your users

Prevent human error

As the GitOps model advocates a more code-based approach to infrastructure, deployment, and operation, it allows to spot errors at an earlier stage than before, through peer review for instance.

Moreover, the Kubernetes AdmissionController concept enables blocking the deployment of resources that do not comply with a set of rules. This makes the above recommendations easier to enforce than just relying on our teams’ goodwill.

Overall, security becomes more proactive than reactive.

Shift Left, even more!

Following the Shift Left approach, security is taken into consideration at earlier stages of deployment. It could be taken even further!
Some tools can scan Infrastructure as Code repositories for potential misconfigurations. This could be applied to Kubernetes clusters and we could be protected from potential attack vectors even before they are in place.

Take a step back, anticipate your next step

The complexity of container technologies in general, and of Kubernetes in particular associated with their fast adoption rate across the industry makes securing them a challenge that may seem overwhelming at first glance. However, we tried to lay out possible solutions to tackle it.

Leveraging CWPP features to detect and alert, we are able to focus our attention on actions yielding the best improvements of our security posture.

Even though security is an everyday concern, especially so with containers, taking a step back to anticipate future containers’ usage and the associated securisation steps might be worth it.

In that regard, serverless containers¹⁰ and the distro-less approach are of interest.

[1]: Gartner’s “Forecast Analysis: Container Management (Software and Services), Worldwide”
[2]: Kubernetes adoption, security, and market trends report 2021
[3]: Terraform at Decathlon: From On-Prem to 100% Cloud
[4]: How Decathlon implemented a CSPM | Fabien Bloume | Decathlon Technology
[5]: Securing Kubernetes, Secure containerized environments with updated threat matrix for Kubernetes, Kubernetes Hardening Guidance
[6]: Using Amazon ECR Images with Amazon EKS — Amazon ECR, Configuring access control | Artifact Registry documentation
[7]: Kubernetes documentation: Use a private registry
[8]: PodSecurity admission (PodSecurityPolicy replacement) · Issue #2579 · kubernetes/enhancements
[9]: https://kubernetes.io/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/
[10]: Cloud Run: Container to production in seconds, AWS Fargate, Azure Container Instances, Elastic Container Instance (ECI): Serverless Containers