Maximising Tool Reuse Across Projects
How REWIND promotes the sharing of tools and tech
Many many moons ago at REWIND, we had a problem. We often needed to move fast on new projects and we didn’t want to reinvent the wheel each time, so we needed a bag of tools available and waiting for us from day one in each of our sparkly new Unreal projects. We needed a means of guaranteeing all of the tools that we wanted to share between projects were simply there by default, with no developer effort.
Package Management
One approach to solving this problem would be to adopt a package management approach, where each package could be an Unreal Engine plugin that could be selected for use and integrated into the project.
An immediate concern for us with this approach (other than that package management in C++ is still not great) was that, at REWIND, we try as much as possible to follow the mentality that we all own our internal tools. Good ideas for tools can come from anywhere, at any time, in any project, and the gating of tools behind a package management solution tends to implicitly enforce a mentality where if an issue is encountered with a package, then it’s the responsibility of whoever the maintainer is, which means we would have introduced a new dependency chain, slowing everything down.
Instead, we wanted to encourage all members of the team to make additions, improvements and fixes to our shared tech and for the process to be easy. Whilst it’s not impossible to get a team on board with doing this in a package-managed world, it does introduce quite a few additional steps to the development cycle.
If a developer identifies a bug in a piece of shared tech for example, in order for the fix to be transferable to all future projects, they would need to first iterate on the fix in the project, then integrate it into the repo hosting their shared tech that gets distributed via package management, then integrate it back into their project to finally validate.
There’s also the inescapable problem of versioning with this package management. How many engine versions do we support with our shared tech? How do we roll changes and improvements for one project on one engine version into another? Again, not insurmountable problems, but the process seemed pretty at odds with our desire to move fast.
Enter Streams
What we really wanted was to enable REWIND developers to be able to make changes to their version of the shared tech for a given project directly in their project depot, and then to provide a workflow that easily allowed those changes to be pushed back up for use in future projects.
Like most real time studios working with Unreal, REWIND uses Perforce for source control, and in Perforce, there is the concept of streams.
Streams in Perforce are kind of analogous to branches in Git. The motivation behind Perforce streams, like branches, is to facilitate iterating on features in child streams before pushing back to the mainline. They support the notion of one codebase having a relationship with another, a rigid workflow defining how changes in one can be propagated to another, and offer some pretty nifty features like a visualisation of all streams via the Stream Graph, their relationship with one another, and whether one stream has changes that can be propagated to another.
With all this in mind, many people are at first surprised by our approach to hosting projects as child streams of our ‘mainline’ stream hosting all shared tech and tools. It is however a pretty powerful one.
With this model, we can still consider each project unique, conceptually its own repo. Both shared tech and project code sit alongside one another in the stream. Developers are free to make changes, additions and bug fixes directly in the stream and specialise the tools specifically for the project if needed. If not, then we have a well-defined integration path between the project’s stream and other projects, allowing us to take our improvements with us as we move onto future projects. Creating new streams would mean that all tech defined in the parent stream would be immediately present from the get-go, and we get to neatly sidestep the problem of versioning since if needed, each project effectively can become its own version of the tools specific to that version of the engine.
Supporting Unreal Customisations
The one major problem we had left to solve is that, depending on the scale of a project, we wanted to either use our modified version of Unreal built from source (affectionately dubbed ‘REWIND UE4’ internally) or instead opt for the vanilla version of Unreal Engine distributed via the Epic Games Store.
In order to allow any project to make use of our custom version of Unreal, we defined the parent stream of all projects to be the REWIND UE4 stream. All projects then, by nature of inheritance, are able to merge down changes from REWIND UE4, make changes local to the project, and copy back up anything that is worth retaining for the future.
By maintaining a distinct version of Unreal, we also need to keep pace with Epic as they release new versions and updates to the engine. Again, we solved this via streams. We maintain an epic_ue4 stream, pure and untainted straight from the source. The parent of the epic_ue4 stream is epic_ue4_integration, and the parent of the integration stream is rewind_ue4.
The epic_ue4_integration stream then in effect becomes our staging ground. When a new version of the engine comes out, we first drop the codebase in epic_ue4. We then propagate all changes from both epic_ue4 and rewind_ue4 into the integration stream, reconcile all changes, fix all merge conflicts, run all tests until we’re happy and confident our tools are all behaving as expected.
Finally, we copy our new and improved version of the engine back up to rewind_ue4, ready for all future projects to use. Projects can then opt to adopt the new version of the engine if desirable by choosing to merge everything down from their rewind_ue4 parent stream.
Vanilla Unreal Engine Projects
Sometimes at REWIND, the needs of a project dictate that we don’t use our built-from source version of the engine, yet in these cases, we still want to maximise the number of internal tools we have at our disposal.
We have handled this case by preferring to write all shared technology/tools as UE4 plugins and maintaining them in a discrete REWIND subdirectory of the UE4 Plugins directory. A child stream of rewind_ue4 named plugin_only effectively then filters out all of the UE4 codebase, except for our REWIND plugins directory, and remaps the directory to the root level of the stream so that it sits in the location that a UE4 project expects.
We then, for each project targeting vanilla UE4, create a child stream of the plugin_only directory. This means that we then have access to all shared technology exposed via plugins, and still maintain the tool propagation workflow between projects/streams.
For added fun, we have also started subdividing those projects’ streams that hang off of plugin_only into engine versions, organised by virtual streams that sit between them and the plugin only stream, as a neat way of knowing versions of UE4 that a project targeted at a glance, as well as giving us a protection mechanism for preserving past projects if we choose to alter/update our Perforce stream directives in the future!
It’s been several years since we put this stream architecture in place, and so far it has scaled really well with us as we have grown. It has provided countless opportunities to level up our shared toolset; from large new features grown during R&D periods, to tiny modifications and bug fixes discovered during development. It has been a great means of empowering our whole team to contribute to our shared tech, reinforcing the notion that we all own our code.
XRLO: eXtended Reality Lowdown is brought to you by REWIND, an immersive design and innovation company. If you want to talk tech, ideas, and the future, get in touch here.
Your claps and follows help us understand what our readers like. If you liked our articles, show them some love! ❤️
We’d also love to hear from you. If you’re passionate about all things XR, you can apply to contribute to XRLO here. ✍️
