and how bounded contexts can help to avoid it
It doesn’t take long to learn plenty of commonly-held software principles with catchy acronyms. YAGNI, SOLID, KISS, and perhaps one of the most attractive of all, DRY, or Don’t Repeat Yourself. Extracting anything that’s used in multiple places across a codebase to a single place is one of our strongest instincts as developers. Not only does it mean that there’s a single source of truth and a single place to change if necessary, it just feels good. Identical or very similar code existing in multiple places seems untidy and perhaps even verging on unprofessional, particularly if it comes with reams of boilerplate required to massage data into different forms. But as with most good things, it can be taken too far. I did that during the architectural design of SQL Clone, and it ended up causing problems that we’re still dealing with resolving now.
SQL Clone consists of an HTTP(S) server which exposes resource-based APIs, and several components which talk to them. One of these is a web UI provided by the same server. There are also PowerShell cmdlets and an agent service, both of which can be installed on any number of machines. As with any application, it has domain concepts which thread through everything and which each component needs to be able to handle.
At its core, SQL Clone needs to create point-in-time copies of databases (images) and provision virtualized databases (clones) based on them, and so there are
Clone classes corresponding to these concepts. They were designed to be largely immutable, while a set of
Operation classes describes state transitions like creating and deleting images and clones. These were all also intended to be “complete”, such that you could only construct an instance of an
Clone which is fully-formed and valid, so that none of the properties could be unexpectedly null because they hadn’t been set.
These objects would be the single source of truth for what the whole system understands as an image, clone or any other domain concept. This meant that as they evolved, we could just change this one class and everything else would be kept in sync by the compiler.
Well, almost. Although this worked for the SQL Clone PowerShell cmdlets, which are written in C# and compiled with the rest of the product, the web UI is written in TypeScript, which couldn’t directly use the C# classes. But we could generate TypeScript typings from the C# objects, and the TypeScript compiler would use those, and everything should follow nicely.
Oh, and it didn’t really work for the database layer, either. Having recently been involved in a migration from LINQ to SQL to Entity Framework, I was keen to avoid coupling too closely to an ORM, and there were parts of the model which didn’t directly translate to SQL Server tables. So a new collection of objects with names like
CloneEf were created and all the relevant mapping code written.
That’s all, right? Ah, not quite. Some operations, notably creating an image, could take hours, and in a distributed system we didn’t want the agent having to be able to reach the server at any arbitrary time in order to continue — it’s not unreasonable for there to be a temporary network interruption or for the server machine to be down for an upgrade or reboot. To support this, we wanted to give the agent all the information required to perform an operation in the initial request rather than relying on it querying the server for it in an ad-hoc manner. We considered a few different approaches, but in the end, this birthed a set of objects with names like
CloneExpanded which contain copies of all their dependencies in a denormalized manner. And these needed mapping code, too.
This isn’t exactly as DRY as we were hoping. But isolating ourselves from the specifics of the database table structure seems reasonable. And we did have some particular design objectives around how information is shared with the agent. But having taken care of those exceptions, and dealt with the TypeScript code, we’re there, right? One model class can be used in most of the code, and it still mostly provides a single source of truth, given that the variants that exist are kept in sync by the compiler through the mapping code.
In order to share these domain objects between the server, the UI and the PowerShell cmdlets, they are serialized as JSON and returned as responses from the API. The web UI is hosted by the same server that’s providing the API, so it moves in lockstep, and there are no versioning issues. But the domain objects aren’t necessarily the ideal shape for consumption by clients. The object model is heavily normalized (remember, DRY), but clients don’t necessarily want to have to make multiple HTTP requests (and maybe receive a lot of unnecessary information), then post-process the results in order to obtain a representation of a resource that’s appropriate for end users.
Sometimes, there are properties which the application needs to store but are actively bad to send back to clients, such as encrypted passwords. These had to be handled with a further level of indirection. And despite our best efforts to avoid over-sharing, we recently found out that we were sending the full text of SQL scripts back to clients unnecessarily, which was both a significant waste of bandwidth and memory and a potential security concern.
As we’ve found more cases where some level of denormalization makes sense, the domain objects have drifted away from their original DRYness, becoming, perhaps, unpleasantly moist. They have properties which are only valid when they’re “hydrated” by going through the database layer, and are only used by clients, such as information on the most recent operation associated with an image or clone. This doesn’t cause a huge number of actual problems, beyond some unnecessary joins, but it’s not a particularly clean design pattern. We also handle IDs in different ways in the front end and back end, meaning that the web UI has to perform a transformation to fold the ID into every object it receives.
But although there are some problems with communicating with the web UI, our real issues are with PowerShell (and potentially any other clients of our API). Installed cmdlets do not move in lockstep with the API, but instead expect to receive responses of a particular shape. And since our cmdlets rely on a .NET library which deserializes these responses into the original domain objects, they throw exceptions if there are any incompatibilities. This applies even if they’re within properties that the cmdlets never use.
This is quite insidious, and we’ve been caught out by it a couple of times. We renamed a property three levels deep in one of the domain objects and thereby accidentally introduced a breaking change for older cmdlets, merely because this sub-object could no longer be deserialized by old cmdlets. This was despite the fact that the cmdlets have never accessed that object. We’ve also been unable to add new values to enums, since they also can’t be deserialized by the old code without exceptions being thrown. We’ve now put a number of tests in place to verify backwards compatibility, and some workarounds to allow us to more safely add new enum values, but it’s a lot of work to end up with something that you’re only fairly confident in.
And ultimately we don’t have an API that we’re proud to document and share. It’s not designed well as an API, as it’s just a serialization of an object model that makes sense inside the server. And because there’s always a risk that we’ll change some sub-property of a sub-property without realizing, we’re not totally confident that it’s stable.
Resolving this situation has required understanding that just because there are certain concepts which need to be represented in many parts of the system, these different parts might not need (or want) the same representation as others. The server, the agent and the clients (web and PowerShell) represent different bounded contexts in which the concept of an image or clone should be allowed to differ according to the different requirements of these components. The concept of bounded contexts is one of the core ideas of domain-driven design (DDD), first proposed by Eric Evans.
For example, to an agent, which performs file system operations on a particular machine, a clone consists of a mounted virtualized disk and an associated database, so it needs to have specific information about e.g. where the mount point is. This information is not useful for the web UI, but unlike agents, it needs to be able to represent clones which previously existed but have now been deleted, along with metadata such as the original creation time. While many properties are useful in both contexts (such as the database name), some are extraneous, and some are actively bad to expose in the wrong context (as we ran into previously with SQL scripts).
While being provided with unnecessary information doesn’t immediately seem harmful (except for being a waste of bandwidth), once something has been added to the API it can be difficult to remove it again, especially if you’re not in control of your consumers, and you risk leaking your abstractions in a way that can make the whole system less flexible. And sticking rigidly to a single representation of the data forces consumers to do more work than necessary — you’re potentially introducing an impedance mismatch which will slow down future development indefinitely.
We’re currently in the process of moving away from serializing these domain objects and instead using DTOs (data transfer objects) across this boundary. We were already using this pattern over two other boundaries (to the database and to the agents), but not realizing that the clients also represent a different bounded context that needs a different representation has been fairly costly.
There is more boilerplate associated with using DTOs. We need to create new objects (initially, ones which exactly copy the existing API) and write more code to copy data into them. But by formalizing our API in this way, we can be confident that its shape will be stable even if the server’s domain objects change, provided that the DTOs themselves don’t change and we guard the conversion process with tests.
Could we create different APIs for the web UI and for PowerShell, customized for their own specific use cases? We could. But as both are used by end users and are used to accomplish the same sorts of tasks, for us, their contexts seem sufficiently similar. It’s also relatively easy for us to revisit this if their needs diverged by introducing a new API for the web UI if necessary, since it moves in lockstep with the server.
Once the whole API is using DTOs, we can also start to version our API effectively. The first DTOs we’re creating represent the domain objects as they are today, but in the future we’ll be able to create a new v2 API which can be designed for the convenience of clients rather than the server. In DDD terms, this relationship is referred to as an open host service with a published language. We can also serve endpoints for both the old and the new API concurrently to allow migration. This will allow us to make other breaking API changes in the future, too, without immediately requiring that everyone updates their existing cmdlets.
We’re still on this journey at the moment, and moving all our controllers across to use DTOs will take some time. Designing and implementing a new API will take some more on top of that. But once we get there, we’ll finally have decoupled these different bounded contexts, and our problems with being too DRY will be water under the bridge.