Protecting your Application on Cloud Run with API Gateway and Identity Aware Proxy

Photo by Tim Hüfner on Unsplash

I’m a big fan of serverless. The less I have to take care about hosting infrastructure, the happier I am. Because I can focus more of my time on generating value for my users. But since I’m always conscious about security it also gives me peace of mind to know, that my services are behind a secure layer that somebody else maintains for me. Personally I like to use platform features to secure service to service communication. In service meshes mTLS is a great option to solve this. In Cloud Run we can rely on Google Cloud IAM to provide this level of security to us.

In Google Cloud with Identity-Aware Proxy (IAP) there is a great solution to protect your web applications against undesired access. Similarly API Gateway is a great, lightweight option for exposing APIs. It takes care of verifying JWT tokens for you and can be securely integrated into your stack thanks to Google IAM. Combining IAP and API Gateway though can be challenging. In this post, we are going through the necessary steps together, you will need some experience with Spring and Spring Security. You can start with a Spring application from Spring Initalizr with some OAuth dependencies. I put a full example plus Terraform for deployment here:

blog-examples/lb_apigw_spring_boot_example at main · cgrotz/blog-examples

Add Identity-Aware Proxy to a Spring Boot App

Let’s start by adding support for IAP to our application. IAP provides the login token via the x-goog-iap-jwt-assertion header. This header contains a JWT that is signed by IAP. So let’s start by adding JWT verification to our application.

spring:
  security:
    oauth2:
      resourceserver:
        jwt:
          issuer-uri: https://cloud.google.com/iap
          jwk-set-uri: https://www.gstatic.com/iap/verify/public_key-jwk

And configure a different header name in the securityFilterChain:

http
    .oauth2ResourceServer()
        .bearerTokenResolver(new HeaderBearerTokenResolver("x-goog-iap-jwt-assertion"));

Now our application is ready to accept and verify IAP tokens.

Adding API Gateway support to a Spring Boot App

Now adding API Gateway support should be easy. We need to add the issuer-uri and jwk-set-uri config to the application.yaml.

spring:
  security:
    oauth2:
      resourceserver:
        jwt:
          issuer-uri: <issuer-uri>
          jwk-set-uri: <jwk-set-uri>

And of course we need a TokenResolver for API Gateway as well, since API Gateway provides the original login token via the x-forwarded-authorization header. A custom BearerTokenResolver could look like this:

http
    .oauth2ResourceServer()
        .bearerTokenResolver(new HeaderBearerTokenResolver("x-forwarded-authorization"));

Combining API Gateway and Identity-Aware Proxy

Sadly at the moment we can’t simply combine the two options. Spring Security OAuth only supports one JWT Identity Provider configuration at a time. But we can combine the BearerTokenResolver and init a custom AuthenticationManagerResolver in the SecurityFilterChain.

@Bean
  public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
    JwtDecoder decoder1 = createJwtDecoder(issuerUri, jwksUrl);
    JwtDecoder decoder2 = createJwtDecoder("https://cloud.google.com/iap",
        "https://www.gstatic.com/iap/verify/public_key-jwk");
    JwtAuthenticationProvider provider1 = new JwtAuthenticationProvider(decoder1);
    JwtAuthenticationProvider provider2 = new JwtAuthenticationProvider(decoder2);
    JwtIssuerAuthenticationManagerResolver authenticationManagerResolver =
        new JwtIssuerAuthenticationManagerResolver(context -> {
          if (context.startsWith(issuerUri)) {
            return provider1::authenticate;
          } else if (context.startsWith("https://cloud.google.com/iap")) {
            return provider2::authenticate;
          } else {
            throw new RuntimeException("Unsupported Issuer " + context);
          }
        });
http
        .authorizeHttpRequests(authorize -> authorize
            .anyRequest().authenticated()
        )
        .oauth2ResourceServer(oauth2 -> oauth2
            .authenticationManagerResolver(authenticationManagerResolver)
        );
    return http.build();
  }
private JwtDecoder createJwtDecoder(String issuer, String jwkSetUri) {
    OAuth2TokenValidator<Jwt> jwtValidator = JwtValidators.createDefaultWithIssuer(issuer);
    NimbusJwtDecoder jwtDecoder = NimbusJwtDecoder.withJwkSetUri(jwkSetUri)
        .jwsAlgorithm(SignatureAlgorithm.ES256)
        .jwsAlgorithm(SignatureAlgorithm.RS256)
        .build();
    jwtDecoder.setJwtValidator(jwtValidator);
    return jwtDecoder;
  }

Also, the tokens of the two options are differently formatted. IAP will extract the custom claims from the original token and place them in the gcip claim. This results in differently formatted Authentication objects. We can harmonise this of course with a custom Converter<Jwt, AbstractAuthenticationToken> (you get the full example in the linked source code).

Calling other services

When calling other services, we also want to use IAM to protect service to service traffic. This means that we need to place the user’s context, for example the original JWT, in which the calls should be made in either the payload or a header. In Spring we can simply customise the RestTemplate using a RestTemplateCustomizer and adding an interceptor to add an IAM token.

interceptors.add((request, body, execution) -> {
      // This behaviour could also be limited to the .a.run.app domain to only forward the Identity Token to other Cloud Run services
      GoogleCredentials adCredentials = GoogleCredentials.getApplicationDefault();
      if (adCredentials instanceof IdTokenProvider) {
        IdTokenProvider idTokenProvider = (IdTokenProvider) adCredentials;
        IdToken idToken = idTokenProvider.idTokenWithAudience(
            "https://" + request.getURI().getHost(), null);
        request.getHeaders().add(HttpHeaders.AUTHORIZATION, "Bearer " + idToken.getTokenValue());
      }
      return execution.execute(request, body);
    });

Deploying the solution

The architecture of the example

At the moment, there are a few things that make the deployment complex:

1. API Gateway traffic is not considered internal from Cloud Run, hence Cloud Run services that are fronted by an API Gateway need to have ingress all configured. Of course when setting ingress all you want to have IAM authorisation in front of your Cloud Run service.
2. IAP traffic is considered load balancer traffic, hence you can set ingress internal+loadbalancing, but it requires you to set roles/run.invoker to allUsers

Those two configurations are due to security concerns exclusive to each other. You want to have at least one layer of protection in front of your services. The easiest solution is, to deploying your micro-services twice, with the different configurations.

If you don’t want to do that (or due to some organisation policy/constraint can’t do this), you can also consider using an envoy based reverse proxy as a backend for the Load Balancer hosted on a VM or Managed Instance Group for the IAP Load Balancer instead of using the serverless network endpoint group. The proxy can add the Identity token to your requests, allowing you to deploy all Cloud Run services uniformly with ingress set to all and IAM limited to certain service accounts. You can find an example for the envoy configuration here:

blog-examples/envoy-cloudrun-proxy at main · cgrotz/blog-examples

So, this completes this small excursion in hosting a Spring Boot application on Cloud Run with API Gateway and IAP in front.