Multi-Tenant Application Configuration — An unpopular opinion
I have seen quite a few solutions to application configuration since I started my programming career, some 20 years ago.
Along the way, I have seen tens of different solutions to that problem, and what I ended up preferring is a solution that I have found to be the best of breed.
Explaining this rather non-conservative solution proved to be a challenge, so I thought it would be good to share this so that I can point newcomers to this post: “See — even someone on the internet said it!”
The solution I’m going to present is more applicable for B2B systems rather than B2C systems for multiple reasons
Problem Definition
So what do I mean when I say “Application Configuration”?
This means options or settings such as: Default language, Column order or titles in a table, Paid module flags, Max page size, Region-specific feature flags, etc.
To try and generalise this, what we want to solve is application configuration that has the following characteristics:
- Applicable to multi-tenant systems (e.g. multiple accounts running on the same system, dozens of them)
- Non-secret\Non-sensitive (e.g. not database passwords, hostname, etc.)
- Not applicable for multi-users data (e.g. users profile not a good fit — hundreds of them)
- Configurations are updated by multiple teams, not just the Development team, but also Technical Support and non-tech staff members
Example
We want to set a default language for a certain enterprise account. We want all users of that account, when they login, to start off with a certain language.
In a classic RDBMS solution it might look like this:
A piece of code can then be written to query the table:
This will return the default for “*” since there is no specific value for “CORPORATE_C”, if we’d query for CORPORATE_B, we would get “FR” as the default language.
This is the most common solution I have seen, sometimes the default is defined in the code to reduce the dependency on the database, but otherwise, the configuration storage is a database, which is the main point.
So What’s the problem?
There are multiple challenges I have noticed with this solution:
- Logging Changes/Audit Trail — It’s impossible to track changes without developing a complete audit solution around the changes, so updating directly on the database, becomes a risk.
- Rollback — Performing a rollback is more difficult as it is harder to know what exactly was changed.
- Visibility Into Changes — When a developer performs a configuration change, it would be hard to see who made the change and what was changed.
- Support Hierarchies — Consider a Department inside a specific Account that can have a setup that is different for the main account default. So “Department” becomes another configuration parameter. RDBMS does not make it as easy to represent this hierarchy.
- Share across environments — When you consider Local, Staging, Production, QA, etc., it means tooling needs to be developed such that any change to the production database needs to be aligned to the other environments. Otherwise, recreating a defect becomes a huge headache if it is configuration related.
The alternative solution
So we want a Hierarchical solution that has full audit/log built into it, we want to be able to see all changes, and we want it to be shared across all environments?
Wait. Isn’t that problem already solved? Isn’t that what source control is all about? isn’t that exactly what we do with our production code? We push it to a source control, then we publish this to relevant servers and we are done.
We can branch, we can send for peer reviews, we can treat configuration as a first class citizen!
Reading this using python:
Of course this is just a basic and simple example.
Yes, But..
What about deployment/distribution of the change?
It is much easier for people to consider configuration in a database than to consider physical files as a form of configuration distribution.
Personally, I find that this part is very easy to overcome, anywhere from Ansible to Rsync, another straightforward solution is using an NFS/EFS, in addition to the simple fact that we can just solved this with whatever solution we use to distribute our code.
Another reason for concern might be in creating the back-office UI, creating a CRUD (Create-Read-Update-Delete) user interface where the storage is a database is way more natural and straight forward vs doing the same against a set of files.
Lastly, distribution time in a database is in the milliseconds scale vs a source control push and pull on the server end, which, at best, will take couple of seconds to complete.
To Conclude
We are talking about two classes of solution for application configuration.
Database Configuration
Pros
- Easy to update using standard database tools & sql clients
- Very fast to distribute
- Easier for back office development
Cons
- Requires developing dedicated audit trail layer
- Harder to notify team about changes
- Easy to miss changes if done directly on the database
- Requires developing a solution to sync config between environments
- Hard to debug locally (too easy to miss changes that are only present in environment X)
- Harder to manage hierarchies in and RDBMS
Files backed by Source Control
Pros
- Built in audit trail
- Built in rollback and roll-forward
- Easy to share across environments
- Easy deployment system (same as code)
- Lends itself to complex hierarchies
- Easy to notify about changes & review change by multiple team members
Cons
- Harder to distribute across servers.
- For non developers - Requires building an application that will have source control pull/push permission
- Harder for back office development
So which of these do you use ?
