Turning News Into Data — Day 9

September 1 2022

Turns out I was using the --service-account flag the wrong way

All I needed to do was add this environment variable to gitlab

GC_CLOUDRUN_SVC_ACCOUNT_KEY=<email here>

And then use the gcloud run deploy command like this

gcloud run deploy <OTHER ARGS HERE> --service-account ${GC_CLOUDRUN_SVC_ACCOUNT_EMAIL}

The email is the email that is displayed on the service account IAM page. Then the deploy_dev stage works like this:

deploy_dev:
  image: google/cloud-sdk:alpine
  stage: deploy to cloudrun dev
  before_script:
    - echo ${GC_CLOUDRUN_SVC_ACCOUNT_KEY} > ${HOME}/key.json
    - gcloud auth activate-service-account --key-file ${HOME}/key.json
  script:
    - >
      gcloud run deploy ${CI_PROJECT_NAME} 
      --image ${GC_REGION}-docker.pkg.dev/${GC_PROJECT}/${GC_IMG_REPO}/${CI_PROJECT_NAME}:latest 
      --project ${GC_PROJECT} --region ${GC_REGION} --allow-unauthenticated --platform managed 
      --service-account ${GC_CLOUDRUN_SVC_ACCOUNT_EMAIL}
  after_script:
    - rm ${HOME}/key.json

Now the actual deployment isn’t working for some reason though

I’m assuming it is from dockerize waiting for other services which do not really exist in the cloudrun environment so I’m going to remove it. Also adding the environment variable for NEWS_API_KEY to git CICD variables

It works!

And this is what I mean by cloudrun’s nicely integrated interface

You can extend the logs even more and also push them to a centralized logger somewhere if you want

Anyway, here’s the final .gitlab-ci.yml file:

stages:
  - quality assurance
  - build, push image
  - deploy to cloudrun dev
quality_assurance:
  image: python:3.9-alpine
  stage: quality assurance
  script:
    - echo "this is where the unit and integration tests will go"
# in future host own runners and avoid docker in docker (dind)
build_image:
  image: google/cloud-sdk:alpine
  stage: build, push image
  services:
    - docker:dind
  variables:
    DOCKER_DRIVER: overlay
    microservice: ${CI_PROJECT_NAME}
    DOCKER_HOST: tcp://docker:2375
  before_script:
    - echo ${GC_ARTIFACT_SVC_ACCOUNT_KEY} > ${HOME}/key.json
    - gcloud components install docker-credential-gcr
    - gcloud auth activate-service-account --key-file ${HOME}/key.json
    - gcloud auth configure-docker --quiet
    - gcloud auth configure-docker ${GC_REGION}-docker.pkg.dev --quiet
  script:
    - docker build -t ${CI_PROJECT_NAME}:latest .
      --build-arg GC_PORT=${GC_PORT}
      --build-arg NEWS_API_KEY=${NEWS_API_KEY}
    - docker tag ${CI_PROJECT_NAME}:latest ${GC_REGION}-docker.pkg.dev/${GC_PROJECT}/${GC_IMG_REPO}/${CI_PROJECT_NAME}:latest
    - docker push ${GC_REGION}-docker.pkg.dev/${GC_PROJECT}/${GC_IMG_REPO}/${CI_PROJECT_NAME}:latest
  after_script:
    - rm ${HOME}/key.json
deploy_dev:
  image: google/cloud-sdk:alpine
  stage: deploy to cloudrun dev
  before_script:
    - echo ${GC_CLOUDRUN_SVC_ACCOUNT_KEY} > ${HOME}/key.json
    - gcloud auth activate-service-account --key-file ${HOME}/key.json
  script:
    - >
      gcloud run deploy ${CI_PROJECT_NAME} 
      --image ${GC_REGION}-docker.pkg.dev/${GC_PROJECT}/${GC_IMG_REPO}/${CI_PROJECT_NAME}:latest 
      --project ${GC_PROJECT} --region ${GC_REGION} --allow-unauthenticated --platform managed 
      --service-account ${GC_CLOUDRUN_SVC_ACCOUNT_EMAIL}
  after_script:
    - rm ${HOME}/key.json

The Dockerfile :

## build stage
FROM tiangolo/uvicorn-gunicorn-fastapi:python3.9 as build
WORKDIR /app
# tell python not to write to .pyc files and instead write bytes
ENV PYTHONWRITEBYTECODE 1
# force stdout and stderr to be unbuffered
ENV PYTHONBUFFERED 1
COPY . /app
RUN pip wheel --no-deps --wheel-dir /app/wheels -r requirements.txt
## main stage
FROM tiangolo/uvicorn-gunicorn-fastapi:python3.9
# add env variables here and also in gitlab-ci as build args
ARG GC_PORT
ENV GC_PORT=${GC_PORT}
ARG NEWS_API_KEY
ENV NEWS_API_KEY=${NEWS_API_KEY}
ENV PYTHONBUFFERED 1
WORKDIR /app
COPY --from=build /app/wheels /wheels
COPY --from=build /app .
RUN pip install --no-cache /wheels/*
EXPOSE ${GC_PORT}
WORKDIR /app/src
CMD gunicorn --bind 0.0.0:${GC_PORT} -w 4 -k uvicorn.workers.UvicornWorker app.server:app

And the updated dev_scripts/launch_container.sh file:

#!/bin/sh
if [ -z ${GC_PORT} ]; then
    echo 'Run the following command first:'
    echo ''
    echo 'export $(cat .env | xargs)'
    exit
fi
echo "Building container image named ${APP_NAME}"
docker build -t ${APP_NAME} . --build-arg GC_PORT=${GC_PORT} --build-arg NEWS_API_KEY=${NEWS_API_KEY}
# you need -it to make interactive so you can run bash scripts directly within docker
echo ""
echo "Deploying image as container"
docker run -it -dp ${GC_PORT}:${GC_PORT} ${APP_NAME}