How to track active users in Google Cloud Platform (GCP)
I mean, all of your active users. Across your whole organization. Not just some particular project.
Why would I need to do this? In the case you were wondering this, I can tell you my story: in my case, I received the business requirement to force the 2-step verification adoption for advanced users: this is, not just regular G-Suite users (accessing gmail and drive services), but also services like Google Cloud Platform (GCP).
Probably you know you can do this by checking the integrated activity log in GCP logging. Nevertheless, this can take a while when the number of projects in your organization is considerable. And although this amount can be somewhat small, when you add a new project you must extend your search to the new added project. In summary, it is not manageable at scale (or in time).
What I discovered is a nice feature, regarding exporting different activity logs, as an export sink, directly to logging. And I know, you can do this from ages ago: what’s different then? well, you can do this across your whole organization via aggregated sinks. This is, no matter the project is. This is manageable at scale.
How come I never realized this before? you could be asking to yourself. Well, a simple answer is this method is not available on the web console. If you are interested on this, you must use either the API or the gcloud command. And I am going to help you with that, with a short quickstart.
What to Do
I am going to provide you the formula for a BigQuery sink (I think the most suitable and efficient sink of them all). You can extend this solution easily by checking the documentation and seeing the requirements for the other options (look for the destination section, where you will find sinks such as Cloud PubSub and Cloud Storage)
Requisites for BigQuery sink:
- a sink! (for example, create previously some BigQuery dataset, in some active project: this is what I am going to use in the rest of the article, but you can set whatever sink you are comfortable with)
- permissions for creating the sink: ideally, Logging/Logs Configuration Writer in the project where the sink will live. And yes, by having Owner role you are ok, but this does not follow the least privilege principle.
- permissions for writing logs: a dedicated service account is going to be created for writing the logs. When you create the sink, you will be given the details on this (see the example below), and you will need to grant the BigQuery Data Editor on the previous dataset.
TL;DR: I just want the formula/solution
Ok, just run this:
In the previous command you find:
- YOUR_SINK_NAME: it is self explanatory. Give a comprehensive name.
- PROJECT_ID: where the sink lives
- DATASET_ID: where the logs are going to be written
- ORGANIZATION_ID: this is the numerical id for you organization. How to pick it up easily? grab it directly from whatever Google Cloud link, it is a query parameter!
- LOG_FILTER: a query that specifies what we want to export (this is a export sink, remember!). You can specify things that you normally set in the advance filter box in the web console. For example (and my recommendation) is you to setup this filter if you want all relevant activity logs:
Once you run the gcloud command, and if it succeeds, you will be given this output (intentionally I redacted several words and identifiers):
Created [https://logging.googleapis.com/v2/organizations/${ORGANIZATION_ID}/sinks/${YOUR_SINK_NAME}].Please remember to grant `serviceAccount:o123456789012-345678@gcp-sa-logging.iam.gserviceaccount.com` the BigQuery Data Editor role on the dataset.More information about sinks can be found at https://cloud.google.com/logging/docs/export/configure_export
What else can I get by using this solution?
If you check the official documentation (even for the alpha and beta versions) for the command, google logging sinks create you can see that you can focus on logs for:
- your organization (what we review in this article!)
- a particular project (what you probably already knew!)
- a particular folder (nice if you use folders -if not, you should take a look into this-)
- a particular billing id (nice to have if you manage separate billing accounts)
What can I do with this solution?
Ok, in the case you lack of ideas, here’s something that maybe can be of interest: if you have access to the GSuite Admin for your organization, you can the security user reports, where for example you get nice things such as last login, password strength, and 2-step verification enrollment. This report you can export to a Google Sheet.
So, I have access to a security auditing of my users (in Google Sheets), and (after reading this article) detailed information of Google Cloud users in BigQuery.
If you didn’t get it yet: you can cross reference the information. Either by implementing an external query of the security auditing in BigQuery, or (for more visual users) implement a DataStudio report by connecting both sources directly, and implementing a Blended Source of them (spoiler alert: the email is your join key). With this, you can audit security measures for advanced users who access GCP regularly in your organization (it can be a security breach to have GCP users with weak passwords, for example).
One more thing
If you use the alpha version of the command, you can set nice things such the partitioning for BigQuery. Probably this version will be promoted to GA, but at this time it is only available as this:
I must highlight that another (very) interesting option is to enable DLP on your sink:
If you face a similar situation, where you discover something interesting, I invite you to share it with the community.
