Importing GCP Projects into your Organization with Python

Problem: “Legacy Projects” (pre-Organization)

If you use GCP with G Suite (including Cloud Identity) accounts, you are already familiar with how using Organizations makes it easier to centrally manage all your Google Cloud resources.

However, if you started using GCP before the Organization resource became generally available, you may have a number of projects that were created by your G Suite users before the Organization resource for your domain existed. These projects are currently out of your control.

Your users can opt-in to migrate existing projects into the organization, as long as have granted them the Project Creator role on your organization. But as the GCP Organization administrator, there is no easy way for you to find all of these “legacy projects” and “import” them into your Organization resource.

Solution: Automate with Python

Automating this process (even for thousands of users or projects) is pretty easy as long as you can create a service account with delegated domain-wide authority in your G Suite domain. (You may need to contact your G Suite administrator if that person is not you.)

Process

This is the general process to follow in order to discover and import all of your domain’s legacy projects into your Organization resource.

  1. Retrieve the list of all users in your G Suite domain using a service account with domain-wide authority and a user with the Super Admin role.
  2. As each user in the domain, retrieve a list of projects and their IAM policy bindings to find projects owned by the user that have no organization or folder as their parent.
  3. For each of the projects found in step #2 (as the user that owns the project) add the service account as an owner of the project.
  4. As the service account, add the project to the Organization resource.

Service Account Setup

In order to do this with Python, we need a service account with the right level of access to both the G Suite domain and the GCP Organization resource.

  1. Create a service account (select Enable G Suite Domain-wide Delegation)
  2. Save the service account JSON file (ex. serviceaccount.json)
  3. Delegate domain-wide authority for the service account to use the following scopes: https://www.google.apis.com/auth/admin.directory.user, https://www.google.apis.com/auth/cloud-platform
  4. Enable APIs in the GCP console: Admin SDK, Cloud Resource Manager API
  5. Grant service account the Project Creator role in the Organization resource.

Authenticating with Python

Now that we have a service account with all the right permissions, we can create credentials and prepare to make an authorized call as a user with the Super Admin role in our G Suite domain.

from oauth2client.service_account import ServiceAccountCredentials
superadmin = 'google@mydomain.com'
scopes = [
'https://www.google.apis.com/auth/admin.directory.user',
'https://www.google.apis.com/auth/cloud-platform'
]

credentials = ServiceAccountCredentials.from_json_keyfile_name(
'serviceaccount.json',
scopes
)
admin_creds = credentials.create_delegated(superadmin)

Retrieving all Domain Users

Now we can use these admin credentials to make an authenticated call to the Admin SDK Directory API in order to get a list of all users in the domain.

Note: Uses the Admin SDK Directory API users.list method.

from apiclient.discovery import build

admin = build('admin', 'directory_v1', credentials=admin_creds)
params = {
'customer': 'my_customer',
'fields': fields,
}
users = admin.users()
request = users.list(**params)
domain_users = []
while request is not None:
users_list = request.execute()
domain_users += users_list.get('users', [])
request = users.list_next(request, users_list)

Retrieving all Projects

With a list of all domain users, we can now “become” each user to get a list of all of their projects.

Note: Uses the Cloud Resource Manager projects.list method.

# loop through all of the users in the domain
for user in domain_users:
email = user['primaryEmail']
    # create credentials as the user
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'serviceaccount.json',
scopes
)
user_creds = credentials.create_delegated(email)
    # connect to cloud resource manager as the user
crm = build(
'cloudresourcemanager',
'v1',
credentials=user_creds
)
    # get a list of all projects visible to the user
projects = crm.projects()
request = projects.list(**params)
    user_projects = []
while request is not None:
projects_list = request.execute()
user_projects += projects_list.get('projects', [])
request = projects.list_next(request, projects_list)

Find User Projects to Import

For each user, once we’ve gotten the list of visible projects, we need to narrow the list down to only the active projects that have no parent (organization or folder) and are owned by the user. This will require us to get the IAM policy bindings for each active project with no parent.

Note: Uses the Cloud Resource Manager projects.getIamPolicy endpoint.

# loop through each of the projects visible to the user
for project in user_projects:
    # skip projects that are not active
if project['lifecycleState'] != 'ACTIVE':
continue
    # skip projects that already have a parent
if not project.get('parent', {}):
continue
    # get the IAM policy bindings for the project
project_id = p['projectId']
params = {
'resource': project_id,
'body': {},
}
policy = crm.projects().getIamPolicy(**params).execute()
bindings = policy.get('bindings', [])
    owner = False
    # find out if the user is an owner of the project
for b in bindings:
        # skip bindings other than owner
if b['role'] != 'roles/owner':
continue
        # see if user is one of the owners
if 'user:%s' % (email) in b['members']:
owner = True
    # if user is owner, add the project to import list
if owner:
        # add bindings to the project record
project['bindings'] = bindings
        # add the service account as a project owner...
        # add the project to the organization...

Add Service Account as Project Owner

Now that we know which projects need to be imported into the Organization resource, we still need the right permissions in order to be able to add the project. Once again we do this as the user.

We can’t add the super admin as an owner via the API. Adding an owner to the project generates an email notification that must be accepted by the user before they can be granted the role (this can only be done from the console).

Adding a user as owner with the API is only possible if the project is already in an organization and the user being added as an owner is in the same domain as the organization. Service accounts can be made owners of a project directly without any restrictions.

Note: Uses the Cloud Resource Manager projects.setIamPolicy endpoint.

# set service account user name
user = 'serviceAccount:%s' % (credentials.service_account_email)
newbindings = []
update = False
# check bindings for owner role and update if necessary
for b in project['bindings']:
if b['role'] == 'roles/owner' and user not in b['members']:
b['members'].append(user)
update = True
newbindings.append(b)
# update the project IAM policy if necessary
if update:
body = {'policy': {'bindings': newbindings}}
params = {'resource': project_id, 'body': body}

# set the IAM policy for the project
crm.projects().setIamPolicy(**params).execute()

Add Project to the Organization Resource

Now that the service account is an owner (and it’s already a Project Creator on the Organization) we can use the service account’s credentials to add the project to the organization.

# set the organization ID
organization_id = '123456789012'
# create credentials as the service account
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'serviceaccount.json',
scopes
)
# removing bindings from record before updating
del project['bindings']
# add a "parent" to the project record
project['parent'] = {
'type': 'organization',
'id': organization_id
}
params = {
'projectId': project_id,
'body': body,
}
crm = build(
'cloudresourcemanager',
'v1',
credentials=credentials
)
# update project parent to the organization_id
crm.projects().update(**params).execute()

Done! Each active project that is owned by one of your G Suite users but had no parent is now a member of your Organization resource and you can begin managing ALL of your users’ GCP resources!

Example: import_projects.py

I’ve added a new tool called import_projects.py, which bundles in all the functionality described in this post, to my gcp-tools GitHub Repo. Running it with a properly configured service account and the email address of a super admin in your G Suite domain will allow you do all of the following:

  1. Retrieve all users in your domain
  2. For each user, retrieve all of their projects
  3. Filter active “legacy projects” (owned by your users but not in your organization)
  4. Display the list of projects to import
  5. Request permission to proceed with importing projects
  6. For each project to import, add the service account as an owner, and
  7. Add the project to the organization.

Here is an example of it in use (note, google@mydomain.com is the name of a user with Super Admin access to the mydomain.com G Suite domain):

> ./import_projects.py google@mydomain.com
Retrieving users from Google Admin SDK Directory API...
Found 13 users in domain.

Scanning all users for projects without a parent...
lukwam@mydomain.com:
* Test Project 1: test-project01 [367391543796] (ACTIVE)
* Test Project 2: test-project02 [746509631927] (ACTIVE)
* Test Project 3: test-project03 [216921026845] (ACTIVE)

Found 3 projects to import:
* test-project01 (lukwam@mydomain.com)
* test-project03 (lukwam@mydomain.com)
* test-project02 (lukwam@mydomain.com)

Preparing to move 3 projects into org: mydomain.com...
---> Are you sure you want to continue? [y/N]: y

Organization: mydomain.com [685481217344] (customer: C0392o3bz)

* test-project01:
+ added gcp-tools@ltk-gcp-tools.iam.gserviceaccount.com as project owner.
+ added project to organization 685481217344.

* test-project02:
+ added gcp-tools@ltk-gcp-tools.iam.gserviceaccount.com as project owner.
+ added project to organization 685481217344.

* test-project03:
+ added gcp-tools@ltk-gcp-tools.iam.gserviceaccount.com as project owner.
+ added project to organization 685481217344.

Done.

Full documentation for import_projects.py is available in the GitHub Wiki for this project. Bugs and feature requests can be submitted via GitHub Issues.

References:

Show your support

Clapping shows how much you appreciated Lukas Karlsson’s story.