Run your app locally as if you were on Google cloud
Any applications have issues; Applications running on Google Cloud also. And, to debug them, you have to run them locally, add breakpoints, and deep dive in the code.
Some issues come from security context or configuration. To reproduce them, you have to test with the exact same context locally to reproduce the exact same behavior as if you were on Google Cloud.
How to run securely and locally an app with the exact same security context as on Google Cloud?
That means 2 things:
- The app must run with the same security context, permissions and principal.
- The app requires no code change/update, or doesn’t implement a condition that changes the app behavior if it runs locally or in the cloud.
The Google Cloud authentication mode
Before exploring the different solution, let’s have a look at the Google Cloud environment security mechanism.
The Google Cloud client libraries that you use automatically detect the runtime environment to obtain the credentials. The mechanism is named ADC: Application Default Credential.
On Google Cloud, every service has a metadata server which exposes endpoints, and some to generate short lived tokens and without exposing any secrets.
By default, the default service account of the service is used to generate those tokens.
However, many services offer to customise the service account to use. It’s a good practice to use a custom service account on that service and to grant only the required permissions on it for a dedicated workload.
If you have several workloads that use the same service, you can have 1 custom service account per workload and narrow the granted permissions.
It’s the least privilege principle.
The default service accounts have too wide permissions like Project Editor and it’s dangerous to use such powerful permissions by default everywhere!
If you use a service account key file on Google Cloud environment, I recommend you upgrade your code and application to remove it. The service account key files are only files and it’s dangerous and difficult, to manage them safely. I already wrote an article on that topic.
The most secure way to keep secrets secret, is not to have secrets!
Service account use is a bad practice, even if you watch videos, follow tutorials or read documentation EVEN COMING FROM GOOGLE CLOUD!
The 3 legacy ways to achieve it
To reproduce the ADC locally, and running the application in the exact same security condition as on Google Cloud, there are 3 existing and not satisfying solutions.
Let’s discover which they are, and why they aren’t good.
Use service account key files
It’s the most common way to achieve it in the Google Cloud security environment. Lot of articles and videos use, and abuse, of it. Mainly because it’s easy, and legacy. It’s also the ugliest!
The principle is :
- Download a service account key file locally. Save it as JSON
- To define an environment variable
GOOGLE_APPLICATION_CREDENTIALthat points to the absolution path of the key file on your workstation. - Run your application.
The ADC mechanism inside the Google Cloud client libraries detects automatically the environment variable and use it as credential source. No need to change the code.
The main problem of that solution is the service account key file itself. It’s a key file, a secret. Not many in the development teams are aware of that, and the impact that can have a leakage.
You download a secret on your workstation. Is it well secured? Will you delete the file after your tests? Will you do that with your production environment? Are you sure enough?
In addition, because you directly use the service account identity, it’s impossible to differentiate who has performed the request: The application on your workstation or on Google Cloud?
In the audit logs, there is no difference, it’s the same identity. That’s a serious problem in the case of internal security issues, it’s impossible to trace the source of the issue/attack/data thief.
That solution isn’t really acceptable in terms of security.
Grant your user account the same permissions
To prevent the service account key file download, you can use your own credentials, and grant the same permissions as the Google Cloud service account on your own user account.
The ADC mechanism inside the Google Cloud client libraries detects your user account credentials stored in the “well-known” locations (usual local directories), and loads them as source credentials. No need to change the code.
Ok, you have the same permission as the service account on the project, but YOU aren’t the service account. Your principal isn’t the same.
In addition, if the service account email has been granted on another resource, in another project (on a BigQuery dataset hosted in another project), your user account won’t, and you won’t be able to reproduce the exact same workload.
For a small app limited to 1 project, the solution can work. But we can’t rely on it for all apps.
Impersonate service account in your code
To have the exact same permission in all the project and to have the same credential as the Google Cloud service account on your local environment,
- You must use the service account credential.
- You mustn’t use a service account key file.
Here comes the service account impersonation feature.
The impersonation is the ability to use your user credential and to generate OAuth tokens on behalf of the service account.
It’s a powerful feature that allows reuse of service account permissions without downloading a secret key.
In addition, your principal is logged in the impersonation delegate chain to know the originator of the request. You have a clear view of WHO is using the service account.
The problem comes from the integration with the Google Cloud client libraries. The feature is supported but you need to add a piece of code to activate the impersonation.
When you want to test locally your app because you have security issues, and the first thing that you do is to change the security logic, it’s obviously absurd!
The new impersonation way
So, if we could combine all the pros without the cons, we will have the perfect solution: impersonate a service account but without changing the code and letting the Google Cloud client library manage that for us.
That feature exists!
It’s a brand new feature and few know it. The idea is to configure the account impersonation outside of the app code, with the gcloud CLI. For that, you can use that command
gcloud auth application-default login \
--impersonate-service-account=<service account email>Pretty simple, no?!
Then run your app, and it will run with the same service account credentials as your Google Cloud environment by impersonation! The Google Cloud client libraries are “awesome”!
Of course you must be a Service Account User on the service account to be able to impersonate it.
Awesome? quite. Simple? Yes. Does it work? Not sure!!
Indeed, I initially tested it with a Golang app and it didn’t work. It was too new and the feature wasn’t implemented in the Google Cloud Golang client libraries. I implemented it and it was merged in October.
So now, it works for Golang (perform a go get golang.org/x/oauth2 to download the latest version on your local environment). It also works in Java.
From my latest tests, it wasn’t the case in Python (feel free to contribute if you need that feature!)
Update security context by configuration.
The reproducibility on your local environment is really critical to quickly understand and resolve the problems, but security is certainly more important when you work with sensitive applications in production.
That table summarizes the 4 solutions
Solutions are multiple, some are legacy, others are quicker. However, keep security in mind at any stage of the development, and even after the development!
The Service Account impersonation with gcloud CLI hasn’t any red flag but is not yet perfectly supported by the client libraries of all the languages.
Anyway, try it first!!
Google Cloud security products are gaining more and more features that help you to avoid bad practices and to keep your environment clean and safe!
