Implementing Multi-Tenancy in Cloud Spanner

by Christoph Bussler and Sireesha Pulipati

Christoph Bussler
Dec 1, 2020 · 23 min read

Introduction

Multi-tenancy is a software architecture pattern in which a single or few instances of an application serve multiple tenants or customers, often hundreds or thousands. This approach is fundamental to cloud computing platforms where the underlying infrastructure is shared among multiple organizations. Basically, multi-tenancy can be thought of as a form of partitioning based on shared computing resources like databases. An analogy is tenants in an apartment building: shared infrastructure, but dedicated tenant space. Multi-tenancy is also the hallmark of most, if not all, software as a service (SaaS) applications.

Categories of criteria

The different architecture approaches of how to map a tenant’s data to Cloud Spanner are as follows:

  • Database. A tenant resides in a database, and several databases are in a single Cloud Spanner instance.
  • Schema. A tenant resides in exclusive tables within a database, and several tenants can be located in the same database.
  • Table. Tenant data are rows in tables shared with other tenants.
  • Agility. The ease of onboarding and offboarding activities for a tenant wrt. instance, database or table creation.
  • Operations. The availability or complexity of implementing typical tenant-specific database operations and administration activities like regular maintenance, logging, backups, or disaster recovery operations.
  • Scale. The ability to scale seamlessly to allow for the future growth of the size and the number of tenants. In the description of each partner the number of tenants is discussed that can be supported by the pattern.
  • Performance. Ability to allocate exclusive resources to each tenant addressing the noisy neighbor phenomenon and enabling predictable read/write performance for each tenant.
  • Cost. The costs associated with whichever data management pattern selected and the ability to split the cost proportionally among tenants (if needed).
  • Regulations and compliance. Features to address requirements of highly regulated industries and countries that may require complete isolation of resources, maintenance operations. For example, data residency requirements for France require that personally identifiable information (PII) is physically stored exclusively within France.

Data management patterns

The following sections describe the four main data management patterns in context of multi-tenancy: instance, database, schema and table.

Instance — one tenant per instance

In this data management pattern, each tenant’s data is stored in its own Cloud Spanner instance and database. A Cloud Spanner instance can have one or more databases, however, in this pattern only one database is created in an instance for full and complete isolation. Following the example of the HR application above, a separate Cloud Spanner instance is created for each customer organization with one database each.

Data management pattern: one tenant per instance
Criteria     | "Instance — One tenant per instance" 
| data management pattern
-------------+------------------------------------------------------
Isolation
| Enables greatest level of isolation; no database
| resources are shared among tenants.
-------------+------------------------------------------------------Agility | Onboarding and offboarding requires considerable
| effort to set up or decommission the Cloud Spanner
| instance, instance specific security, and
| connectivity. This can be automated through
| Infrastructure as Code.
-------------+------------------------------------------------------Operations | Backups can be performed independently for each
| tenant, providing separation and flexibility in
| backup schedules. This data management pattern
| results in higher operational overhead owing to the
| large number of instances to manage and maintain with
| respect to scaling, monitoring, logging, and
| performance tuning.
-------------+------------------------------------------------------Scale | High scalability, both with respect to the number of
| customers as well as tenant specific scalability.
| Cloud Spanner, being a highly scalable database, can
| accommodate virtually unlimited growth for a tenant
| by increasing the number of nodes. This pattern can
| support an unlimited number of tenants as for each
| tenant a Cloud Spanner instance can be created.
-------------+------------------------------------------------------Performance | There is no resource contention among different
| tenants owing to separate instances. Also allows for
| tailored performance tuning and troubleshooting for a
| tenant without impacting others.
-------------+------------------------------------------------------Cost | Cloud Spanner is priced based on the number of nodes,
| amount of storage and amount of network used and not
| based on the number of instances. That means, using a
| single instance with a higher number of nodes vs
| multiple instances with a smaller number of nodes
| each totaling to the same number of nodes effectively
| amounts to the same cost. This data management
| pattern does not necessarily result in higher cost
| compared to other data management patterns in that
| aspect. However, isolated instances cannot share
| resources and thereby cannot attain resource
| efficiencies that may be possible with other data
| management patterns for applicable workload patterns.
-------------+------------------------------------------------------Regulatory | In this data management pattern it is possible to
and
| store data in specific regions, implement specific
compliance
| security, backup or auditing processes as per the
requirements
| regulatory and compliance requirements applicable for
| some industries.
  • Disadvantage: Greatest operational overhead
  • Regulatory and compliance requirements for some tenants demand greater levels of security and auditing protocols
  • Size of tenants vary significantly so that sharing resources among high volume, high traffic tenants might cause contention and mutual degradation

Database — one tenant per database

In this data management pattern, each tenant resides in a database within a single Cloud Spanner instance. If one instance is insufficient for the number of tenants, multiple instances can be created. This implies that a single Cloud Spanner instance is shared among multiple tenants as multiple databases can reside in a single instance in the context of this pattern.

Data management pattern: one tenant per database
Criteria     | "Instance — One tenant per database" 
| data management pattern
-------------+------------------------------------------------------
Isolation
| Complete logical isolation on a database level;
| underlying instance infrastructure resources are
| shared.
-------------+------------------------------------------------------Agility | Onboarding and offboarding requires some effort to
| create or delete the database and any specific
| security controls. Can rely on automation through
| Infrastructure as Code.
-------------+------------------------------------------------------Operations | Backups can be performed independently for each
| tenant providing flexibility in schedules. This data
| management pattern involves less operational overhead
| compared to the instance data management pattern, you
| have only one instance to monitor for performance and
| scale (for up to 100 databases).
-------------+------------------------------------------------------Scale | Cloud Spanner instances can be scaled to thousands of
| nodes and can accommodate any level of growth for the
| tenants.
| There is a limit of 100 databases per Cloud Spanner
| instance currently which limits the number of tenants
| on-boarded onto a single instance. For every 100
| tenants a new Cloud Spanner instance needs to be
| created and there is no limit to the number of
| instances.
-------------+------------------------------------------------------Performance | Databases are spread across Cloud Spanner instance
| nodes and thus share the infrastructure resulting in
| resource contention among multiple databases (noisy
| neighbor effect) and impacts the performance.
-------------+------------------------------------------------------Cost | Cloud Spanner is priced based on the resources
| consumed rather than on the number of databases or
| instances. So, this data management pattern doesn’t
| necessarily result in higher cost compared to others.
| However, the shared resources in terms of compute and
| storage can result in efficiencies that can reduce
| the overall cost. The converse, where severe resource
| contention requires additional node capacity than
| otherwise, is possible too.
-------------+------------------------------------------------------Regulatory | In this data management pattern, it is not possible
and
| to meet any specific data residency regulatory
compliance
| requirements, if the desired location is different
requirements
| from the instance regional configuration.
  • Disadvantage: Limited number of tenants per instance and location inflexibility
  • Tenants require system-based data separation and backup/restore, but are fine with infrastructure resource sharing.

Schema — one set of tables for each tenant

In the schema data management pattern a single database, which implements a single schema, is used for multiple tenants with a separate set of tables used for each tenant’s data. These sets of tables can be differentiated from each other by including tenant ID in the table names as either suffix or prefix.

Data management pattern: one set of tables for each tenant
Criteria     | "Schema — One set of tables for each tenant" 
| data management pattern
-------------+------------------------------------------------------
Isolation
| Low level of isolation; no table level security.
-------------+------------------------------------------------------Agility | Onboarding a customer is a fairly simple task
| involving creation of new tables and associated keys
| and indexes. Offloading a customer means deleting the
| tables, which may have a temporary negative impact on
| the performance of the other tenants within the
| database.
-------------+------------------------------------------------------Operations | Operations including backups, monitoring, logging
| cannot be performed separately for tenants by Cloud
| Spanner itself. Those have to be implemented as
| separate functionality by the application itself or
| as utility scripts.
-------------+------------------------------------------------------Scale | Cloud Spanner instances can be scaled to thousands of
| nodes and can accommodate any level of growth for the
| tenants. However, there is a limit of 5000 tables a
| single database can have.
| This pattern therefore supports only
| floor(5000/<number tables for tenant>) tenants in
| each database. If that is exhausted, a new database
| needs to be added for additional tenants.
-------------+------------------------------------------------------Performance | High level of resource contention is possible (noisy
| neighbor). In order to ensure good performance,
| indexes need to be designed separately for each set
| of tables as well.
-------------+------------------------------------------------------Cost | Cloud Spanner is priced based on the resources
| consumed rather than on the number of tables,
| databases or instances. So, this data management
| pattern does not necessarily result in higher cost
| compared to others. However, the shared resources in
| terms of compute and storage can result in
| efficiencies that can reduce the overall cost. The
| converse, where severe resource contention requires
| additional node capacity than otherwise, is possible
| too.
-------------+------------------------------------------------------Regulatory | In this data management pattern, it is not possible
and
| to meet any specific data residency regulatory
compliance
| requirements, if the desired location is different
requirements
| from the instance regional configuration. Also
| implementing specific security and auditing controls
| impacts all the tenants that reside in the same
| database.
  • Disadvantage: Higher operational overhead and lack of security controls at the table level
  • Multi-tenant applications where the data does not require strict separation based on legal or regulatory requirements.

Table — one table for several tenants

The final data management pattern is to serve multiple tenants with a common set of tables. Each table contains data for several tenants. This data management pattern represents an extreme level of multi-tenancy where everything — from infrastructure to schema to data model — is shared completely among multiple tenants. Within a table, rows are partitioned based on primary keys with tenant ID as the first element of the key. From a scalability perspective Cloud Spanner can support this pattern best since it can scale tables without limitation.

Data management pattern: one table for several tenants
Criteria     | "Table — One table for several tenants" 
| data management pattern
-------------+------------------------------------------------------
Isolation
| Lowest level of isolation; no tenant level security.
-------------+------------------------------------------------------Agility | Onboarding a customer does not require any setup on
| the database side. The application can write data
| directly into the existing tables. Offboarding simply
| means deleting the customer’s rows.
-------------+------------------------------------------------------Operations | Operations including backups, monitoring, logging
| cannot be performed separately for tenants by Cloud
| Spanner and they have to be implemented separately.
| Little to no overhead as the number of tenants
| increases.
-------------+------------------------------------------------------Scale | Cloud Spanner instances can be scaled to thousands of
| nodes and can accommodate any level of growth for the
| tenants.
| This pattern can support an unlimited number of
| tenants.
-------------+------------------------------------------------------Performance | This pattern works very well in the context of Cloud
| Spanner as its internal distributed storage and
| processing as well as load balancing can easily deal
| with this pattern.
| A high level of resource contention is possible
| (noisy neighbor) if the primary key spaces are not
| designed carefully preventing concurrency and
| distribution, though. Following best practices is
| very important. Deleting a tenant’’s data might have
| a temporary impact on the load.
-------------+------------------------------------------------------Cost | Cloud Spanner is priced based on the resources
| consumed rather than on the number of tables,
| databases or instances. The shared resources in terms
| of compute and storage can result in efficiencies
| that can reduce the overall cost. The converse, where
| severe resource contention requires additional node
| capacity than otherwise, is possible too.
-------------+------------------------------------------------------Regulatory | In this data management pattern, it is not possible
and
| to meet any specific data residency regulatory
compliance
| requirements, if the desired location is different
requirements
| from the instance regional configuration. It cannot
| provide system level partitioning either (compared to
| the instance or database pattern). Also any specific
| security and auditing controls cannot be implemented
| without affecting all the tenants.
  • Disadvantage: High resource contention, lack of security controls for each tenant
  • Maximum resource sharing for tenants using free tier application functionality when minimizing cost at the same time

Combination of data management patterns and tenant life cycle management

Overview of data management patterns

The following table compares the various data management patterns across all criteria in order to provide an overview on a very high level abstracting from the details above.

Combination of data management patterns

In many cases, a single data management pattern is sufficient to address the requirements of a multi-tenant application. The design of a multi-tenant application can then assume a single data management pattern and all interactions with Cloud Spanner are based on the implemented data management pattern.

  • Regular tier. A regular tier might be established for paying clients that have no specifically strong requirements when it comes to scaling or to isolation. For these clients the schema data management pattern or the database data management pattern might be appropriate. In both cases the tables and indexes are exclusive for the tenant. A difference is that backup is easy in case of the database data management pattern, but not supported by Cloud Spanner for the schema data management pattern — in that case a tenant-backup has to be implemented as a utility outside Cloud Spanner.
  • Enterprise tier. The enterprise tier is usually a high end tier with full autonomy in all aspects. A tenant in that tier has dedicated resources that include dedicated scaling as well as full isolation. The instance data management pattern is well-suited for this requirement.

Tenant data life cycle management

Tenants have a life cycle and corresponding management operations have to be implemented within a multi-tenant application. Beyond the basic operations of creating, updating and deleting tenants, the following additional data-related operations should be considered:

  • Backup tenant data. Similar to export, it might be necessary to back up data for individual tenants. When the data management pattern is instance or database, database functionality like export or backup can be used. However, in the case of the schema or table data management patterns,this operation has to be implemented by the multi-tenant application as the database functionality would not be able to determine which data belong to which tenant.
  • Move tenant data. A very important operation is moving a tenant from one data management pattern to another (or within the same data management pattern between instances or databases). A use case is a small tenant in a table data management pattern growing to such an extent that it has to move to a database data management pattern. This requires extracting the data from the table data management pattern and inserting that data into the database data management pattern. When application downtime is possible, an export/import can be performed (see above), however, when downtime is not possible this corresponds to a zero downtime database migration in the context of Cloud Spanner. Another reason for moving tenants is rebalancing to mitigate a noisy neighbor situation.

Application design

Multi-tenant application design is about implementing tenant-aware business logic, meaning that each execution of business logic must always be in the context of a known tenant.

Dynamic tenant connection and query configuration

A mapping configuration is typically used to dynamically map the tenant’s data to the requests from the tenant application.

  • In case of the schema data management pattern, the correct table names have to be determined.
  • In case of the table data management pattern, queries have to be executed against the database with appropriate predicates to retrieve only the data of the tenant in question.
tenant id -> (data management pattern, 
database connection string,
[table_prefix])

Query generation and attribution

A fundamental underlying principle of multi-tenant applications is the ability to share resources for tenants so that several tenants can share a single cloud resource. The data management patterns above fall into this category except for the case where a single tenant is allocated to a single Cloud Spanner instance.

select * from EMPLOYEE
-- TENANT 356
where TENANT = 'T356';
select * from T356_EMPLOYEE;
-- TENANT 356
select * from T356_EMPLOYEE;
-- {"TENANT": 356}

Tenant access life cycle operations

Depending on the design philosophy, a multi-tenant application can implement the data life cycle operations as described earlier directly, or a separate tenant administration tool can be created.

  • Starting a tenant. This life cycle operation implements the opposite constraints: application logic can access a tenant’s data while those life cycle operations are disabled that would otherwise interfere with the application logic.

Application isolation

The various data management patterns support different degrees of isolation of tenant data. From the most isolated level (instance) to the least isolated level (table) different degrees of isolation are possible.

Summary

Multi-tenancy is an important application design data management pattern, especially when resource efficiency plays a vital role. Cloud Spanner supports several data management patterns and can easily be used for implementing multi-tenant applications. With Cloud Spanner’s extreme scalability and super strict SLAs, it is an ideal database for large multi-tenant application deployments.

Disclaimer

Christoph Bussler is a Solutions Architect and Sireesha Pulipati is a Data Management Customer Engineer at Google, Inc. (Google Cloud). The opinions stated here are our own, not those of Google, Inc.

Google Cloud - Community

Google Cloud community articles and blogs

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Christoph Bussler

Written by

www.real-programmer.com | Solutions Architect at Google Cloud

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.