Most people started to experiment with the Google Cloud Platform (GCP) with the free trial or through a shadow IT. It was really an eye-popping experience when people saw all the Google innovations, and quickly we built quite a bit of resources within the first project, just as we were doing the same way in the other cloud providers. There is nothing wrong with it until we start to plan our POC into a real product to run within our enterprises. Quickly people realized there is a resource called organization within GCP (don’t get confused with AWS’s organizations, which is not a resource but a service to manage billings). GCP organization is like the root of your DNS domain or Active Directory domain, it really is the starting point of the GCP journey for any company.
GCP isn’t the same as any other public cloud, it has a structure like any company. From the top with an organization node, and branched off with folder structure, then the projects. The philosophy of GCP is to have fewer resources within the project but to group only similar IAM resources within the project while managing multiple projects using folders. This design really makes clean IAM with the least privilege access.
A few concepts of the GCP folder structure you need to know:
- IAM is inherited from top-down in an additive way. This means the permission you have on the project or resources are all the permissions you add together from the Org level coming down with all the folders' path to the project. This means the most generous permission wins.
- Folders are only a maximum of ten levels deep. I have not encountered any customer reaching this limit. (It was only four levels deep before)
There are many ways to design folder structures, it is truly a work of art for a cloud architect to leverage GCP folder structures to design the entire GCP infrastructure. Here are a few considerations from my experience:
- First top-level folders considerations: Typically a company’s culture and political landscape have to be considered first. Do you have multiple business units, how autonomous are business units, or are any merge/spinoffs in the near future? These answers will drive the top folders’ design for those business units. If your company is product-driven, environmentally separated top folder structure such as Production/Staging/QA/Dev will be adopted. In any case, a dedicated IT top folder is always used as most of the companies have IT organizations to provide centralized IT services of Network, Security, Governance, etc. This is perfect mirroring to GCP’s centralized cloud foundation principles. And last but not least, don’t forget your IAC (Infrastructure as Code) folder, this is your bootstrap top folder when you plan to manage your entire GCP infrastructure programmatically.
- Folder Structure could represent the organization team chart. However, it is not necessary to mirror it, but use it as a guideline to separate team resource access. A typical sample is the data lake access, which can be separated by regions, countries within the region, and productions within the country.
- IAM is also one of the main reasons to leverage folders. This involves the concept of how each cloud architect designs GCP projects in the least privileged methodology. In a nutshell, how to isolate resources from each other by dividing them into more projects, but balance out manageability of multiple projects using folder structure. I often manage IAM at folder/sub-folder level, project-level IAM assignment is usually the exception roles of its parent folder. This provides easier IAM assignment and simpler IAC coding.
- Use sub-folders to whitelist exceptions. To manage the GCP environment in a centralized IT world, you typically have some Organization Policy constraints restricted at org-level, such as network-related constraints, etc, then you can whitelist your IT Network sub-folder for network resource creation. Uniform folder whitelisting is always better than mixed projects and folders.
- Hierarchical Firewall Policies — This is still a beta feature as of this writing. With this new feature, Firewall rule can be applied from Org or Folders, and VPC (Virtual Private Cloud) projects underneath will inherit those firewall rules from the top. Unlike Cloud IAM inheritance by the union, the hierarchical firewall will try to match rules from top-bottom fashion. This provides a centralized firewall rule management.
- Don’t forget your IAC pipeline. You always have to divide and conquer using multiple pipelines to create your GCP environment, your folder structure is the perfect representation of your pipeline and repo.
- Last but not least, please use the unique folder names crossed the entire org/folder structure. There are two reasons behind that, one will be confused for anyone to see IAM inheritance from the folder name listed in the console; second, it will be hard for search process.
The folder structure design seems pretty simple but carries a lot of GCP design principles. Leveraging this truly unique feature from GCP can really benefit your GCP manageability.
