GCP API Gateway for Cloud Run authenticated backend with Firebase Auth JWT Tokens

An exercise in finding the right Authorization Header to Authenticate Users

When securing APIs with Firebase Authentication through GCP API Gateway, I faced some unexpected issues — the JWT token was not going through as expected was what I finally figured, but it took a frustrating, sleepless day to get to that discovery. Luckily, Gemini was able to help me fix something that should have been a non-issue. Gemini was very, very helpful in being a pair-investigator, a la Sherlock and Watson, questioning the results, the options, the paths to investigate/debug, trying out different things, and finally, it was elementary. I believe that it would have taken me significantly longer to identify the issue had it not been for Gemini. Or, as I’m prone to when faced with a lack of time, abandoned the whole excursion as a too-time-consuming-to-be-valuable task and moved on. However, the little progresses we made with the investigation kept me going. There were occasions when I was so tired and sleepy, but since I outsourced the heavy thinking and code generation to Gemini, I had enough brain battery to be able to evaluate the suggestions and continuously make incremental progress.

p.s. before I continue, let me note here that this is not a tutorial but a post-mortem of a debugging process in a hyper-specific scenario. Hopefully, it will give somebody who is in the same position as I was a quick answer, but it also outlines an investigation and debugging approach for similar situations.

The Symptom: Firebase Auth failed with Invalid Firebase Token

My setup involved an Angular v19 frontend sending Firebase ID tokens to a Python backend on Cloud Run, fronted by GCP API Gateway. The backend consistently failed verification, throwing this error:

Invalid Firebase token.

Small debugging tip at this point: be generous and expansive with your logging. Example below. I only had minimal logging initially and that lost me time.

    try:
        # Verify the ID token
        log_debug(f"Verifying Firebase token: {id_token}")
        decoded_token = auth.verify_id_token(id_token)
        log_debug(f"Decoded token: {decoded_token}")
        uid = decoded_token['uid']
        log_debug(f"Token verified for UID: {uid}")
    except auth.InvalidIdTokenError as e:
        log_error(f"Invalid Firebase token: {e}", exc_info=True)

The Seeming Problem: Correct Token Sent, Wrong Audience Received

After comparing the logs of the token sent from the front end and the one received at the backend, it was clear that they were different. my-project-id was my Firebase Project ID. This is also the correct expected value of the audience (aud) field. Yet, the backend received a token that showed aud as the backend's own service URL. With help from Gemini, I wrote an interceptor for the API call to ensure that what was finally getting sent was the correct token holding the correct audience and issuer fields. My frontend interceptor logs confirmed I was sending a token via user.getIdToken() with the correct aud: "my-project-id"and issuer. My off-the-cuff suspicion at this stage was that the token was getting mangled somehow or maybe it used a different encoding. Of course, I soon ruled that out because the data was being decoded and some non-nonsensical, yet readable values were showing, but it was just the wrong values.

Debugging Steps

I followed several steps:

• Verified Frontend: Confirmed Angular code and Firebase config (projectId: "my-project-id") were correct. Tokens logged before sending were valid. I used the javascript library jwtDecode to pick out and re-verify the parts of the token being sent using the http interceptor. I’d also checked the token format at some point on https://jwt.io/.

if (existingAuthHeader.toLowerCase().startsWith('bearer ')) {
   const existingToken = existingAuthHeader.substring(7); // Get token part
   try {
          const decoded: any = jwtDecode(existingToken);
          console.log('[DebugAuthHeaderInterceptor] DECODED existing token AUD:', decoded.aud);
          console.log('[DebugAuthHeaderInterceptor] DECODED existing token ISS:', decoded.iss);
          console.log('[DebugAuthHeaderInterceptor] RAW existing token:', existingToken);
   } catch (e) {
          ...
   }
}

• Verified Backend: Confirmed Python firebase-admin SDK initialization used the correct project (my-project-id) and verify_id_token was called.
• Analyzed API Gateway Config: Checked the OpenAPI spec. The securityDefinitions and other fields looked correct. An extract:

securityDefinitions:
  firebase_auth: 
    type: oauth2
    flow: implicit
    authorizationUrl: ""
    x-google-issuer: "https://securetoken.google.com/my-project-id"
    x-google-jwks_uri: "https://www.googleapis.com/robot/v1/metadata/x509/securetoken@system.gserviceaccount.com" 
    x-google-audiences: "my-project-id"

paths:
  /user-account:
    get:
      summary: Get user account information
      operationId: getUserAccount
      security:
        - firebase_auth: [] 
      x-google-backend:
        address: https://get-user-account-number-location.run.app/get-user-account
        path_translation: CONSTANT_ADDRESS 

• Logged Backend Headers: Added detailed logging in Python to see all incoming headers and decode the specific token being passed to verify_id_token. I also used PyJWT in the backend to see each part of the decoded token.

unverified_claims = jwt.decode(
    id_token,
    options={"verify_signature": False, "verify_aud": False, "verify_exp": False}
)
log_debug(f"Manually decoded claims (unverified): {unverified_claims}")
log_debug(f"--> Manual AUD claim check: {unverified_claims.get('aud')}")
log_debug(f"--> Manual ISS claim check: {unverified_claims.get('iss')}")

The Breakthrough: Discovering X-Forwarded-Authorization

The detailed backend logging gave me clues to the issue:

• The correct Firebase user token (aud: "my-project-id") was arriving at the backend, but in a header named X-Forwarded-Authorization.
• The standard Authorization header either contained a different token (likely a service-to-service token with aud: <your-cloud-run-service-url>).

My Python code was still checking the standard Authorization header, hence the error.

Why The Documentation Confusion?

It appears the standard GCP API Gateway documentation for Firebase authentication (https://cloud.google.com/api-gateway/docs/authenticating-users-firebase) might be incomplete or doesn't explicitly detail this header-forwarding behavior in all scenarios.

Crucial information explaining why the Authorization header gets modified and the original token moved is found in the Cloud Endpoints documentation regarding OpenAPI extensions, specifically the section on x-google-backend (https://cloud.google.com/endpoints/docs/openapi/openapi-extensions#jwt_audience_disable_auth). This documentation states that when ESPv2 (used by API Gateway) authenticates to the backend service, it copies the original Authorization header value to X-Forwarded-Authorization and overrides the original Authorization header. It explicitly advises the backend to look for the user's JWT in the X-Forwarded-Authorization header in this situation. Anyone using API Gateway with backend authentication should consult both sets of documentation.

The Solution: Read the Correct Header

The fix was to simply to modify the Python backend code to read the token from the correct header:

auth_header = request.headers.get('X-Forwarded-Authorization', '')
id_token = auth_header[7:]  # Remove 'Bearer ' prefix

# Verify the token
user_record, error_response = verify_firebase_token(id_token)

By reading from X-Forwarded-Authorization, the backend now gets the correct user token originally sent by the frontend, and verify_id_token succeeds.

Key Takeaways

When debugging authentication issues with GCP API Gateway, remember that headers might be modified. Log extensively on the backend to see the actual state of incoming requests, and don’t assume standard headers always arrive untouched. Sometimes, the crucial detail lies in related documentation, like needing to check Cloud Endpoints behavior (https://cloud.google.com/endpoints/docs/openapi/openapi-extensions#jwt_audience_disable_auth) even when primarily using API Gateway guides (https://cloud.google.com/api-gateway/docs/authenticating-users-firebase).