Explorer for Google Groups

Edges and nodes for days

Ever bumped into the Google admin-managed Groups parent limits? They can be an issue for B2B multi-tenant application authorization, particularly if you are serving Google Cloud Storage objects in a differentiated way.

Use this utility to proactively check whether you’re getting close to these limits, and to understand hotspots.

  • It provides a visualizer as well as CSVs for offline analysis
  • Batch mode enables you to run it at low load times and analyze the cached results later

You can search for an identity (user, service account, or group) whose ancestry to map, or for a group prefix for which to display summary statistics.

You can also highlight a group’s parents within the visualizer to identify potential group optimizations.

Considerations

The utility does not include group ownerships, which do count towards the quota; there isn’t a way to do this other than scanning the entire groups tree.

Each edge joining a child via multiple groups to a single parent (ie. diamond patterns) is counted; the actual Groups limits may be calculated differently for some children.

Scaling

This utility takes a number of measures to address performance and scaling in addition to the batch mode mentioned earlier; it…

  • caches prior query results.
  • supports multiple service accounts, since some aspects of the Admin SDK Directory API Groups quota metrics are per-identity; two service accounts appear to be sufficient to support reading almost 5k groups.
  • takes advantage of Go concurrency with WaitGroups.
  • implements channel-based throttling.

Note that for constant-load applications such as this utility, exponential backoff actually increases load; it is only useful for spiky load use cases.

Webserver

The Go webserver implements endpoint-specific approaches to serve static files, dynamically generated JSON, and interactive query responses.

Visualization

The graphical visualization is based on vis.js network diagrams and includes a sample data generator for simulation.

Tuning the visualization for 5k nodes required a bit of trial and error. For instance, physics-based graphs improve node spacing for large graphs, but are too slow for thousands of nodes; however, some of the physics options still have a useful effect, though it’s different than it would be with physics enabled.

Digression: Rate Limiters

To test the Explorer utility’s scaling, I needed a large Groups hierarchy with sufficient nesting, which in turn meant another utility. Creating group hierarchy is much less performant than reading it.

The Explorer utility uses channel-based throttling, but apparently this approach spends a disproportionate amount of time in the overhead once you go over tens of queries per second, and it’s then better to use the rate limiter package.

According to the rate limiter docs, most callers should use Wait; however, all the code samples I’d been able to find use Allow and Reserve. Daz Wilkin provided this great snippet demonstrating the correct use of context and the burst parameter.

What’s next

Upgrade the utility to the:

Investigate whether this comprehensive Groups, IAM policy, and projects visualization utility better meets your requirements.

  • It includes Cloud IAM policies and projects in the hierarchy.
  • It loads the data into a JanusGraph database which you can visualize with multiple tools.

Review whether you can replace some or all of your Groups for managing Google Cloud Storage access by downscoping with Credential Access Boundaries.

  • These are supported at both the bucket and object level.
  • You can create a service account that can access every Cloud Storage bucket that you own, then apply a Credential Access Boundary that only allows access to the bucket with your customer’s data.
  • You can create an IAM condition that makes permissions available for objects whose name starts with a specific prefix.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store