Pangeo and Kubernetes, Part 3: Thoughts on Security

Written by Joe Hamman, Scott Henderson, and Rob Fatland, and Amanda Tan Lehr.

Tldr; Pangeo’s public cloud-hosted JupyterHubs are not enterprise grade platforms. We recommend not storing access keys, passwords, or private data on these hubs.

In previous posts we’ve discussed our design of Pangeo deployments on Kubernetes and some rough estimates of how much it costs to operate one in production. In this post, we provide some thoughts and discussion on the subject of security. We should note that none of us are experts in this area. If you are coming here to learn about the core concepts of security on Kubernetes, we recommend you start with an authoritative source, such as this book.

Philosophy

At the time of writing, Pangeo on Kubernetes is a research project. It is being developed, in large part, by individuals who work at institutions where the primary focus is on scientific research. That said, as a multi-user web-based platform, we do our best to ensure access is limited to only registered users in order to mitigate risks. While we have not ignored the subject of security as we have developed Pangeo on Kubernetes, we have, at various times, chosen to focus on functional development in lieu of security features. We argue that this freedom has served us well. Pangeo has advanced rapidly precisely because we were not preoccupied with security constraints. Finally, now that the project reaching a higher level of maturity, we are inviting contributions from individuals and organizations that have expertise in securing such systems.

Terms of Service

Because of the research nature of the Pangeo Project at present, we don’t provide any Terms of Service or a Service Level Agreement (these are legal terms that tend to govern the relationship between users and service providers). Yes, there are people using the Pangeo system for day-to-day research tasks but the systems are still in a beta state. We see three main reasons why we don’t provide level of service guarantees:

1. Much of our funding available to the Pangeo developer community is intended to explore the use of these systems. This funding comes in increments of 1–3 years and under this structure, there is little incentive to hardening our systems at the expense of development velocity.
2. We are not staffed to implement or maintain enterprise grade service or security. For this we would need experienced DevOps engineers. This is simply out of scope for our current grant-funded projects. But we welcome outside contributions and regularly reach out to university IT staff and other software professionals for assistance.
3. We don’t have a legal entity (non-profit, company, etc.) to broker service agreements. As it stands, the Pangeo Project is a distributed collaboration between many individuals and institutions. None of the individuals or institutions are well-positioned to broker these types of service agreements.

Security specifics

The Pangeo Project is fortunate to build upon many tools and services that provide high levels of security. What tools do we use to secure deployments of Pangeo?

1. We use JupyterHub to manage user authentication and web proxying. Most of our JupyterHubs use the industry standard Open Authorization (OAuth) protocol via the services provided by GitHub, Google, or Globus.
2. We use standard Kubernetes tooling to limit access to cluster resources beyond the cluster API endpoint. The Kubernetes clusters are commonly provisioned using tools provided by individual cloud vendors. For example, on AWS, we use eksctl and on GCP we use Google’s SDK.
3. We use Kubernetes roll based access control (RBAC) to limit users ability to control the Kubernetes cluster from within.
4. We use IAM tools from cloud providers (AWS, GCP) to control access to Cloud account resources from the Kubernetes cluster.
5. We use encryption technologies (e.g. GitCrypt) to manage versioned access keys to a variety of systems.

JupyterHub architecture diagram.

Discussion

Despite our best efforts, we have had a few minor incidents over the past two years. These incidents have provided valuable lessons learned and have led to further hardening of our systems. Without providing details on the specific incidents, we are able to highlight the sort of things we’ve seen go wrong and how we’ve adapted:

• Running spurious networking applications. In the early days, we ran our JupyterHubs with open authentication. Anyone could log in if you had a GitHub account. After about a year, we found someone running things they shouldn’t have been and we moved to a more secure authentication system.
• Running crypto-currency mining applications. Cloud providers are quick to catch this behavior and we now ban a long list of IP addresses that have been known to misuse these systems.
• Unauthenticated access to Kubernetes Clusters. Kubernetes itself is a relatively new, and fast moving project. The security around public API endpoints is rapidly changing and we’re doing our best to keep pace with recommended security settings (i.e. this tutorial from Google, or this best-practices guide). We hope that Cloud-providers and others working in this sphere continue revising relevant documentation on the Zero2JupyterHub on Kubernetes project.

In summary

The goal here was to provide some thoughts on the Pangeo Project’s approach to security on its public infrastructure. Organizations that are interested in deploying Pangeo in a more secure environment should not read this post as a deterrent from using the platform. Rather, they should be aware that the default deployment specification does not yet feature enterprise grade security.

Finally, we also seek to entrain new contributions from experts in the area of security to further harden the Pangeo platforms. We are starting a new Pangeo Working Group focused on cloud deployments. If this subject interests you, we encourage you to join our first meeting on December 9th (4:30p GMT, 8:30a PT).