Federated AWS Access

The Chan Zuckerberg Initiative (CZI) leverages technology, community-driven solutions, and collaboration to accelerate progress in Education, Justice & Opportunity, and Science. CZI’s Infrastructure Engineering team is responsible for building shared infrastructure that serves the philanthropy’s efforts in these core areas. Though each team is working toward specific goals, they share many common technical challenges: they all handle sensitive data, manage compute resources, and onboard new engineers often.

We manage our cloud infrastructure on AWS, so we need a secure way for engineers, data scientists, and I.T. admins to get work done in AWS. As we iterated on our AWS access model, we had the following design requirements in mind:

1. Credentials must be short-lived
2. Credentials must be stored securely
3. Credentials must be tied to each user’s centralized identity
4. Every action within AWS is logged for observability
5. Requires multi-factor authentication
6. Authentication is not clunky, so employees don’t feel burdened by the new workflow

Requirements 1–5 are about avoiding weakest-link-in-the-chain scenarios where an adversary finds static credentials. There are numerous solutions for those requirements from Segment’s aws-okta, to Hashicorp’s Vault. We gave a lot of them an earnest try, but many were complicated to use or were deprecated.

Requirement 6 is what this post is about. In this post we describe how we designed the authentication and configuration for aws-oidc, our library for managing AWS credentials.

Design

Our original command-line access method was risky for several reasons:

• Each employee had their access key information stored in plaintext on their laptop or dev server
• A bot would pester us to run a script to rotate those plaintext keys
• AWS Onboarding/Offboarding was disjoint from CZI’s process, often resulting in slow onboarding or delayed access revocation.

We initially wanted to use aws-okta to revamp our workflow. Unfortunately, aws-okta was deprecated shortly after we started rolling it out. We took the aws-okta deprecation as an opportunity to design a lightweight authentication tool that solves security challenges represented in our design requirements. aws-okta and aws-oidc are both command-line tools, but our users reported their delight with aws-oidc’s accessibility. Are we speaking out of personal bias? Possibly, but we cut down our average onboarding time from several hours to several minutes.

When we talk about this project, we usually split it into two parts: authentication and configuration. Authentication describes the backend, how we combine different services (AWS and Okta) to ensure credentials get loaded in a secure way. Configuration touches on the user experience and our work for requirement 6.

Authentication

Each time an engineer assumes an AWS role, aws-oidc checks if the identity token in the cache isn’t expired. If it isn’t expired, then aws-oidc refreshes it by taking the employee through the OIDC protocol with the Okta Identity Provider (IDP).

Figure 1: Workflow for loading credentials into the command-line environment

A key part of our setup is that members of the same Okta group can assume the same set of roles. Each group gets assigned to one or more Okta apps, which dictate the AWS roles the group members could assume.

Figure 2: Our Okta + AWS setup

All of our AWS roles are federated and reference information about the Okta Identity Provider. Here is the policy document we embedded into our AWS roles based on our module in our Terraform library:

Notice there are three aspects of this policy that work with Okta:

• Principal: You need a Federated field that refers to the Identity Provider’s Amazon Resource Name (ARN)
• Action: AssumeRoleWithWebIdentity needs to be enabled for this role
• Condition: You need to have a key (formatted like <Identity Provider URL>:aud) and value (OAuth client ID) that will help inform which roles the user can assume

The Condition aspect is the logic behind the black arrows in Figure 2. It explains why the Infrastructure Engineer can assume the Engineering Infra Role for two accounts, or why a Security Admin can assume two different roles for one account.

Client Configuration

The information from Figure 2 to streamline the AWS authentication flow is an opportunity to suggest setups for employees so they can fully take advantage of AWS. This insight helped us consider a few ideas to make AWS authentication more intuitive. We started with a simple website that displays a suggested AWS config file, and ended up with a command line interface (CLI) that steps the user through a survey to customize their AWS config file.

The user’s perspective is pretty straightforward. They can query the configuration web server through the configure command, answer questions about what roles they would like to configure, and then receive an aws-oidc enabled AWS config file. This is what the user will see, line-by-line, in the Command Line:

$ aws-oidc configure --issuer-url <identity provider url> --client-id <webserver deployment client ID> --config-url <webserver deployment destination>
? Please input your default AWS region: us-west-2
? How would you like to configure your AWS config? Configure one role at a time? (advanced)
? Select the AWS account you would like to configure for this profile: stagingAcct (000000000000)
? Select the AWS role you would like to configure for this profile: developerRole
? What would you like to name this profile: default
? Would you like to configure another profile? No

If we want to modify a client’s access to AWS, we can simply reassign their membership with the right Okta groups, and ask them to run the configure command again. That process will refresh their config file with the updated list of roles.

Future Plans

We will be maintaining this project for a while, so we’d love to see how other teams would use aws-oidc for their infrastructure needs. We’re keeping an eye out for issues, contributions, and any other forms of feedback. You can learn more about contributing.

If you liked this article, consider joining our team. We’re hiring!