Future-Proof Your Multi-Cloud Strategy: Secure AWS Access for Google Cloud Workloads

Introduction

Is your application running on Google Cloud? Are you using static credentials to access AWS resources? If the answer is yes, it’s not just a risk — it’s a substantial security threat. Static, long-lived credentials, like usernames and passwords, are vulnerable to leaks. If they fall into the wrong hands, they could be exploited to access business-critical data, potentially causing financial and legal damage.

In this post, we’ll introduce a solution to reduce this risk significantly. We’ll show you how to use OpenID Connect Federation (OIDC Federation) to switch to dynamic, short-lived credentials. These credentials refresh automatically, eliminating the need for manual rotation. This is a more secure way for your Google Cloud applications to access AWS resources.

Multi-cloud environments are on the rise, driven by the need to optimize costs and reduce vendor lock-in. However, as you integrate multiple clouds, your infrastructure becomes more complex, and the attack surface expands. This underscores the importance of putting security at the heart of your multi-cloud strategy.

Our use-case

We had a similar realization while building the Marketing Tech Platform at Lightricks. This platform uses software and AI to boost our marketers’ efficiency. We integrate with various third-party services to pull marketing data, which is then analyzed to optimize our marketing campaigns.

We deploy our applications with Google Kubernetes Engine (GKE) on Google Cloud (GCP). Some of them need to access data from AWS Kinesis. So, how do we ensure this access is secure? Let’s start by understanding how not to do it.

Figure 1: Multi-cloud workload for Marketing Tech Platform at Lightricks

Access With Static Credentials: Why is this bad?

If your app runs on AWS, securing access is straightforward — attach an IAM role with the necessary permissions. However, things get more complicated on GCP because Google doesn’t support AWS IAM natively.

A common workaround is to generate static credentials by creating an AWS IAM user, storing the access keys in your GCP application, and using them to access AWS services. However, this approach is dangerous because the keys can leak, which could lead to unauthorized access, data breaches, or ransom attacks.

Now, let’s explore how to establish a secure connection from GCP to AWS without storing keys.

Access with Dynamic Credentials using OIDC Federation — The Better Way

OpenID Connect (OIDC) is an authentication protocol based on OAuth 2.0. It removes the need for clients to manage passwords and allows them to get temporary credentials dynamically without storing them. It also provides a way to refresh them on expiration. Sounds promising!

OIDC Federation allows an application in GCP to assume an AWS IAM role. Assuming a role is the process by which an application temporarily gains the permissions and access rights defined by an AWS IAM role.

AWS already supports the OIDC Federation. Since our application is deployed on GCP, it can use Google as an identity provider to request temporary credentials from AWS. Here’s how this works:

1. Our application requests an identity token from GCP’s metadata server
2. Next, it calls the AWS Security Token Service (STS) to exchange identity token for temporary AWS credentials
3. Finally, it uses these temporary credentials to access the target AWS Service (Kinesis in this case)

We can capture this entire flow to generate temporary credentials in a script (credential generation script). But who will invoke this script? Do we also need to keep track of credential expiration, given these credentials are temporary?

Figure 2: Authentication flow between GCP and AWS

AWS SDK — The Orchestrator

Our application triggers the credential generation script through AWS SDK, which handles the entire authentication flow. Not just this, it also refreshes the credentials when they expire. So we don’t need to write any explicit code to refresh credentials.

In summary,

• AWS SDK triggers the credential generation process to obtain temporary credentials with a configurable validity duration
• It caches these credentials in memory until they expire, minimizing unnecessary AWS API calls.
• The SDK uses these credentials to establish a session with AWS.
• When the credentials expire, the SDK automatically refreshes them.

How does AWS SDK trigger the credential generation process?

We use the AWS Config file to map the AWS SDK to the credential generation script, using its credentials_process setting.

# Sample AWS config file setting 
credential_process = /path/to/credential_generation_script

AWS SDK uses this mapping when initializing a session with AWS. Here’s a sample Python code with Boto3, AWS SDK for Python.

from boto3 import Session

# Initialize a session with AWS
session = Session() ### No explicit credentials passed. So Boto3 tries to find this in AWS config file.

# Initialize a kinesis client
kinesis_client = boto3_session.client("kinesis")

# Connection successful. Consume events from kinesis and execute business logic

Figure 3: Authentication flow orchestrated by AWS SDK

Prerequisites to implement authentication flow

First, you must configure a few AWS and GCP resources to make this authentication flow work. This section discusses the steps required for an app deployed with Kubernetes on GKE. Here, we’ll use Terraform as our Infrastructure as Code (IaaC) tool to provision and manage our infrastructure. You can achieve similar results with GCP/AWS console, CLI tools, or other IaaC tools.

GCP Configuration Steps

1. Create a Google Cloud Service Account (GSA): A service account is a special account meant for non-human users. We’ll attach this to our application to provide it with a distinct Google identity. We’ll also use this service account ID to create a trusted entity in AWS. AWS can use this entity to authenticate requests coming from our application. Creating a dedicated service account is recommended to better control the permissions linked to this account

provider "google" {
  project = "your-project-id"  # Replace with your Google Cloud project ID
  region  = "your deployment region"  # Replace with your deployment region
}

resource "google_service_account" "your-gsa" {
  account_id   = "your-gsa"  # Replace with your desired GSA name
  display_name = "Service account for Workload Identity"
}

2. Create a Kubernetes Service Account (KSA). We can’t directly attach Google service accounts to our pods in GKE. Kubernetes has its own IAM system, which is different from GCP’s. So, we need to create a KSA first to give our pod a unique identity in the Kubernetes cluster.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: your-ksa  # Replace with your desired service account name
  namespace: default  # Replace with the namespace where you want to create the service account

3. Enable workload identity for your GKE cluster: Workload identity acts as a bridge between a Kubernetes Service Account and a Google Service Account. When we enable workload identity, our pods can impersonate a Google Service account. This approach is also more secure. It allows Kubernetes pods to assume the identity of a Google Cloud service account, without a static key. Workload identity creates and uses short-lived, dynamic keys under the hood on your behalf.

resource "google_container_cluster" "primary" {
  name     = "your-gke-cluster"
  location = "your deployment region"  # Replace with your deployment region

  workload_identity_config {
    identity_namespace = "your-project-id.svc.id.goog"
  }

  # Other necessary cluster configurations
}

4. Map Kubernetes service account to Google Service Account

• Create necessary GCP IAM Policy Binding
  Assign iam.workloadIdentityUser to your Kubernetes Service Account

resource "google_project_iam_binding" "workload_identity_binding" {
  role    = "roles/iam.workloadIdentityUser"
  members = [
    "serviceAccount:your-project-id.svc.id.goog[default/your-ksa]",  # Replace with your KSA and namespace
  ]
}

• Annotate Kubernetes Service Account: Update your ServiceAccount object to link to Google Service Account.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: your-ksa  # Replace with your KSA name
  namespace: default  # Replace with your KSA namespace
  annotations:
    iam.gke.io/gcp-service-account: your-gsa@your-project-id.iam.gserviceaccount.com  # Replace with your GSA’s email

5. Deploy and Test: Apply all the above configurations and deploy your pod with the newly created Kubernetes Service Account

apiVersion: apps/v1
kind: Deployment
metadata:
  name: your-deployment  # Replace with your deployment name
  namespace: default  # Replace with your namespace
spec:
  replicas: n # Replace with your replicas
  selector:
    matchLabels:
      app: your-app  # Replace with your app label
  template:
    metadata:
      labels:
        app: your-app  # Replace with your app label
    spec:
      serviceAccountName: your-ksa  # Use the KSA created and annotated earlier
      containers:
      - name: your-container  # Replace with your container name
        image: your-image  # Replace with your container image
        # Additional container specifications...

AWS Configuration steps

1. Create an AWS IAM Role For OIDC Federation: Create an IAM role and select Google as an identity provider. Add your GCP service account ID as the audience so that only the intended application assumes the role. You can make this trust policy more restrictive by using StringEquals conditions. Check out all the available IAM condition keys

provider "aws" {
  region = "your_aws_region"  # Replace with your desired AWS region
}

resource "aws_iam_role" "aws_role_for_gcp_access" {
  name = "your-aws-role"  # Replace with your desired AWS role name

  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Principal" : {
          "Federated" : "arn:aws:iam::<your-aws-account-id>:oidc-provider/accounts.google.com"  # Replace <your-aws-account-id> with your AWS account ID
        },
        "Action" : "sts:AssumeRoleWithWebIdentity",
        "Condition" : {
          "StringEquals" : {
            "accounts.google.com:aud": "azp-value", # Replace with azp value from your identity token
            "accounts.google.com:oaud": "aud-value", # Replace with aud value from your identity token
            "accounts.google.com:sub": "sub-value" # Replace with sub value from your identity token
          }
        }
      }
    ]
  })
}

1. Principal: Specify accounts.google.com as the federated identity provider.
2. Action: sts: AssumeRoleWithWebIdentity indicates that the allowed action is to assume the role using a web identity token
3. Condition: Specifies the IAM keys that should match the claims in the identity token issued by Google. This identity token is a JWT Token

Figure 4: Mapping of JWT Claims to AWS IAM Trust Policy Condition Keys

• accounts.google.com:aud: The azp claim in the JWT (JSON Web Token) from Google. Indicates which application or service (referred to as the “client” or “authorized party”) is allowed to use this token
• accounts.google.com:oaud: Maps to aud field in the token. Represents the intended recipient of the token.
• accounts.google.com:sub: The sub (subject) claim in the JWT. Represents the subject. Typically a user id or the client id if no user is involved

2. Add permissions to the role to access your target service (Kinesis here). Set up read-only access to Kinesis as below

resource "aws_iam_role_policy" "gke_policy" {
  role   = aws_iam_role.aws_role_for_gcp_access.id
  policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "kinesis:GetRecords",       # Allows retrieving records from the stream
          "kinesis:GetShardIterator", # Allows getting a shard iterator
          "kinesis:DescribeStream",   # Allows describing the stream
          "kinesis:ListStreams"       # Allows listing all Kinesis streams
        ],
        "Resource": "arn:aws:kinesis:us-west-2:<your-aws-account-id>:stream/<your-stream-name>"
        # Replace <your-aws-account-id> with your AWS account ID
        # Replace <your-stream-name> with your Kinesis stream name
      }
    ]
  })
}

Implementing the authentication flow

Phew, that was quite a few prerequisites. We finally have everything to implement the authentication flow discussed in Figures 2 and 3.

1. Implement the credential generation script
   This script has the following responsibilities:

• Get GCP Identity token: Get the identity token from GCP’s metadata server
• AWS Security Token Service (STS) API call: Call STS’s AssumeRoleWithWebIdentity API to get temporary AWS credentials
• Pass the temporary credentials to AWS SDK (Boto3): Boto3 requires the script to print the credentials as JSON to STDOUT.

Here’s a sample Python implementation for generate_credentials.py

#!/usr/bin/python3

import json
import requests
import boto3

METADATA_SERVER_URL = 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity'
AWS_ROLE_ARN = 'arn:aws:iam::<your-aws-account-id>:role/your-aws-role'  # Replace with your AWS role ARN
AUDIENCE = 'your-custom-audience'  # Replace with your custom audience expected by AWS

def get_identity_token():
    """
    Retrieves the identity token from the GCP metadata server.
    This token will be used to authenticate with AWS.
    """
    response = requests.get(
        METADATA_SERVER_URL,
        params={"format": "full", "audience": AUDIENCE, "licenses": "FALSE"},
        headers={"Metadata-Flavor": "Google"},
    )
    
    return response.text

def get_temporary_credentials_from_aws(identity_token: str):
    """
    Assumes the specified AWS role using the identity token retrieved from GCP.
    Returns temporary AWS credentials.
    """
    sts_client = boto3.client("sts")
    response = sts_client.assume_role_with_web_identity(
        RoleArn=AWS_ROLE_ARN,
        RoleSessionName="gcp-to-aws-session",  # A descriptive session name
        WebIdentityToken=identity_token,
        DurationSeconds=3600, # Validity duration for credentials in seconds
    )
    return response['Credentials']

def main():
    # Step 1: Get the identity token from GCP
    identity_token = get_identity_token()

    # Step 2: Use the identity token to get temporary AWS credentials
    credentials = get_temporary_credentials_from_aws(identity_token)

    # Step 3: Format the credentials for output
    aws_temporary_credentials = {
        "Version": 1,
        "AccessKeyId": credentials["AccessKeyId"],
        "SecretAccessKey": credentials["SecretAccessKey"],
        "SessionToken": credentials["SessionToken"],
        "Expiration": credentials["Expiration"].isoformat(),
    }

    # Step 4: Output the credentials in JSON format
    print(json.dumps(aws_temporary_credentials, indent=4))

if __name__ == "__main__":
    main()

Breakdown of the script: Get the identity token

We’ll use the metadata server to get the identity token. This identity token is a Google-signed JSON Web Token (JWT). Fetching the identity token is a simple HTTP API call to the metadata server. Read more about the request parameters like audience, format, and license

METADATA_SERVER_URL = 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity'
AUDIENCE = 'your-custom-audience'  # Replace with your custom audience expected by AWS
def get_identity_token():
    response = requests.get(
        METADATA_SERVER_URL,
        params={"format": "full", "audience": AUDIENCE, "licenses": "FALSE"},
        headers={"Metadata-Flavor": "Google"},
    )
    
    return response.text

Breakdown of the script: Calling the AssumeRoleWithWebIdentity API

AWS Security Token Service (STS) is a web service that exposes AssumeRoleWithWebIdentity API to request temporary credentials with limited privilege. We’ll use Boto3’s assume_role_with_web_identity interface.

def get_temporary_credentials_from_aws(identity_token: str):
    """
    Assumes the specified AWS role using the identity token retrieved from GCP.
    Returns temporary AWS credentials.
    """
    sts_client = boto3.client("sts")
    response = sts_client.assume_role_with_web_identity(
        RoleArn=AWS_ROLE_ARN,
        RoleSessionName="gcp-to-aws-session",  # A descriptive session name
        WebIdentityToken=identity_token,
        DurationSeconds=3600, # Validity duration for credentials in seconds
    )
    return response['Credentials']

Example Response

{
  "Version": 1,
  "AccessKeyId": "AKIAQ4EXAMPLE1234", # Temporary access key
  "SecretAccessKey": "2kRv3JQklzEXAMPLENewKEY/abCDEFxyz", # Temporary access secret
  "SessionToken": "Fwo.....Bp6l9bGZFsTIGjLAJ2dOBHY7CzK41rscEBnNEWTOKEN==", # Unique session id
  "Expiration": "2024-08-31T04:45:30Z" # Expiration time for these credentials
}

2. Make this script executable

This is necessary for AWS SDK to invoke credential generation script (generate_credentials.py)

chmod +x /path/to/your/generate_credentials.py

3. Create AWS Config File

Set the path to the credential generation script. We’re using a custom profile (your-custom-profile-name), but you can use the default profile, too. Learn more about configuring profiles here.

[profile your-custom-profile-name]
credential_process = /path/to/your/generate_credentials.py

4. Test access: Initialize a session with AWS

Initialize a session with AWS and test access to AWS resources for your application.

from boto3 import Session

# Initialize a session with AWS
session = Session(profile_name='your-custom-profile-name') ### No explicit credentials passed. So Boto3 tries to find this in AWS config file.

# Initialize a kinesis client
kinesis_client = session.client("kinesis")

# Connection successful. Consume events from kinesis and execute business logic
# Example: List all Kinesis streams 
response = kinesis_client.list_streams() 
print(response) # Should print all streams

Observability

AWS CloudTrail can be used to monitor AssumeRoleWithWebIdentity calls made to AWS STS. Each time your application assumes a role using this method, CloudTrail logs the event, capturing key details like the identity provider, audience, and session name. Reviewing these logs helps you track access activity, troubleshoot issues, and ensure your AWS environment remains secure by identifying any unauthorized or unexpected role assumptions.

Conclusion

Now you know how to set up a robust authentication flow for your GCP to AWS workload. Your application can assume an AWS role using web identity federation and access AWS resources using short-lived credentials.

If you have a multi-cloud/hybrid-cloud setup, it is essential to invest in securing this workload to reduce risks and achieve operational efficiency. Try out this approach and share your experience with us.