Collecting Gsuite Enterprise logs with Google API Python Client
Attempting to collect Gsuite enterprise logs via Google API for the first time can be a very painful experience. This is primarily because the steps and the instructions for it are scattered around dozens of pages of documentation, absolutely not consolidated at all!
Between those dozen docs and a whole lot of very specific Stack Overflow posts, it can easily be a couple of weeks (if you are lucky. More if you are not!) and a whole lot of back-and-forth with Google API support before that annoying error 401 — unauthorized goes away and you are successfully able to get the logs that you need. This blog post is an attempt to make that journey relatively easier.
The instructions here are for the Google API Python Client but the steps are easily applicable to Google API Client in any language.
Create a Project
Go to https://console.cloud.google.com
If you already have other projects created by / accessible to your user account:
you will see a page with Project Info, APIs requests graph, etc. In that case, click on the dropdown on the top next to ‘Google Cloud Platform’ that has three triangular dots and says the project name.
Click New Project . Give it a name of your choice and make sure your organization’s name is in the ‘Location’ box.
If you don’t already have any projects created by/accessible to your user account:
click on the dropdown on the top next to ‘Google Cloud Platform’ that says Select a project.
Click New Project. Give it a name of your choice and make sure your organization’s name is correct in the ‘Organization’ box as well as the ‘Location’.
Create a service account and a service account key in your project
Click on the Navigation Menu in the top left corner. Go to APIs & Services > Credentials.
Make sure the project that you just created is selected on the top.
Click on Create Credentials > Service account key
In the dropdown beneath service account, click on new service account. Give your service account a meaningful name. Select Project > Owner role. Enter a meaningful service account ID. In the key type select JSON. Click Create.
This should download a credentials.json file on your laptop.
This file and all the info in this file is super duper important. You will use this info later in many places so, store it in a secure place.
Make sure you add your colleagues as co-owners for your project. You can do this by clicking on Navigation Menu > IAM & Admin > IAM > Add
Enable Google Admin SDK Library
Cick on Navigation Menu > APIs & Services > Library > type ‘Admin sdk’ in the search box > Select Admin SDK > Enable
Authorize your project with the necessary access
**You will be able to complete the following steps only with an admin account**
Go to http://admin.google.com/
Select Security from the list of controls. If you don’t see Security listed, select More controls from the gray bar at the bottom of the page, then select Security from the list of controls. If you can’t see the controls, make sure you’re signed in as an administrator for the domain.
Select Show more and then Advanced settings from the list of options.
Select Manage API client access in the Authentication section.
In the Client Name field enter the value of client_id from the credentials.json file that we’d previously downloaded and saved when we created the service account.
In the One or More API Scopes field enter https://www.googleapis.com/auth/admin.reports.usage.readonly, https://www.googleapis.com/auth/admin.reports.audit.readonly
Click Authorize
Impersonate an admin user account
The script that makes the API call will need to impersonate an admin user in order to successfully fetch the logs. The most secure way to do this would be to create a Google account with admin privileges. Enable 2 step verification for it with a Yubikey and then destroy it. This will make sure that nobody can interactively access this account because there is never going to be a need to do that.
Make the API call and get those logs!
Install Google Python API Client
#For Python2
pip install --upgrade google-api-python-client#For Python3
pip3 install --upgrade google-api-python-client
Create a File called google_logger.py that looks like the following
from google.oauth2 import service_account
from googleapiclient.discovery import build API_SERVICE = 'admin'
API_VERSION = 'reports_v1'
SCOPES = ['https://www.googleapis.com/auth/admin.reports.usage.readonly', 'https://www.googleapis.com/auth/admin.reports.audit.readonly']#make sure credentials.json is in the same directory as where this code is run
SERVICE_ACCT_FILE = 'credentials.json' #enter the admin account id that you want to impersonate
DELEGATION_ACCT = 'admin@xyz.com'#service_account.Credentials.from_service_account_info() works too
creds = service_account.Credentials.from_service_account_file(SERVICE_ACCT_FILE, scopes=SCOPES)
delegate_creds = creds.with_subject(DELEGATION_ACCT)
service = build(API_SERVICE, API_VERSION, credentials=delegate_creds)#applicationName can be any listed in the parameters section here https://developers.google.com/admin-sdk/reports/v1/reference/activities/listlogs = service.activities().list(userKey='all',applicationName='admin').execute()print(logs)
Conclusion
You can find the detailed API reference here: https://developers.google.com/admin-sdk/reports/v1/reference/
This is the most basic version of this script. You can (and should) add mechanisms to store the service account credentials JSON file securely and not store it on disk in plaintext.
This API supports pagination and if there are more pages to your result, you will get a ‘nextPageToken’ when you make the request for the first time. You can pass it in the ‘pageToken’ parameter in your subsequent request. Similarly, you can also define the start and/or end time for which you want the logs.
Hope this is helpful!
