Patchworks or: Why the “Remnant: From the Ashes” update was 42 times larger on the Epic Games Store
Much to the chagrin of players of Remnant: From the Ashes who had installed the game via the Epic Games Store, the latest update came in at a whopping 37 Gigabytes. In other words; the update was as big as the entire game. But on Steam the same update only weighed in at around 923 Megabytes. How come?
People were quick to blame Epic, but through my experience with their platform from building Legendary — an open source replacement for the Epic Games Launcher — I did not believe they were entirely at fault. So, I investigated…
Disclaimer: I do not have access to the Epic Games Store Publishing documentation, all information presented herein is based on reverse-engineering and inference e.g. from packaging tools. None of this is to be taken as an attack on Gunfire Games, Epic, or any other party.
An introduction to game distribution, the Epic/Valve way
Game distribution over the internet is not as simple as one might think. While you could simply have people download the game’s files this is only efficient if there’s never any update released. Any updates would require either redownloading the changed files in full, or need to have delta patches available for a variety of older versions to the new one. While possible, the latter would cost a lot of storage and processing power to create and provide. So several game distribution platforms — including Epic and Steam — use a different approach; the game data is split into “chunks” that are some predefined size (1 MiB in both cases). This means that when an update is released your client only needs to download the chunks that have changed while the ones that haven’t can be reused from the existing installation. This approach of splitting the game into small parts also has other advantages e.g. when it comes to caching on a CDN.
Now while the basic idea is the same for Epic and Steam the approach taken is slightly different: Whereas Steam creates chunks for each file, Epic effectively concatenates all the files and then splits that giant stream of data into the separate chunks.
Of course this is a very simplified explanation. The underlying process is fairly sophisticated and will employ techniques such as deduplication, so that two files containing identical data can “share” chunks to avoid having to redownload the same data twice. Chunks are also compressed (and in some cases encrypted) before they’re uploaded to the platform.
Obviously your client needs to know how to reassemble these chunks back into the game. This is done through a “manifest”, which tells the client what chunks and files are part of the game, how the chunks make up which file(s), etc. The manifests for Epic and Steam are again implemented slightly differently, but they do serve the same purpose.
If you want to learn more about the low level details of Valve’s and Epic’s data formats and how files are downloaded I recommend checking out SteamKit + DepotDownloader and my very own Legendary project.
Into the (manifest) breach
Now that we know the basics, we can start figuring out what happened. Thanks to the aforementioned Legendary project we have a full implementation of Epic’s manifest format for python which we can use to interactively explore the manifest’s contents in ipython:
As the screenshot shows only four files appear to have changed. Those being the main executable and two .pak files which contain pretty much all of the game’s data. While the two Paks together are roughly 37 GB, only a small part of them had actually changed so under normal circumstances only the changed parts would have to be downloaded. But as another set of commands will confirm the files, and indeed manifests, share no chunks at all.
Normally this would only happen if the entire file changed, which could also be the result of the compression method or encryption key changing. But neither of those things are the case. So why did this happen?
Building patches
To see whether this was an issue with Epic’s platform or not I decided to simply do what the game developer would do and run Epic’s “BuildPatchTool” myself. This will ultimately show me whether this was a problem with the packaging on Epic’s end, or the developer’s.
Running the tool without the documentation is thankfully quite simple thanks to the help text it provides. In my case I just needed to run the “PatchGeneration” command with both builds of the game and a shared output directory to get the chunks and manifests created under normal circumstances (with reuse of existing data). Using the tools “DiffManifests” mode we can now see what the download size for updating from the old to the new version would be:
2.2 GB, that’s already a reduction by nearly 17 times! But still larger than Steam’s 923 MB. But this is not the most efficient that patches can be. See, since the manifest created here is supposed to be generic and to be used for fresh installs as well as upgrades from arbitrary versions it cannot reuse existing data as effectively as possible. For instance, when the reusable data makes up only part of a chunk and the other part is now junk, then the download process would incur a potentially significant overhead from having to download old data just to throw away parts or even most of it. For that reason Epic’s tool will prefer to create entirely new chunks when an old chunk cannot be wholly reused.
But as I mentioned, this is not the most efficient it can be, because there’s another mode: ChunkDeltaOptimise. Let’s run it and see what the diff says afterwards:
Success! The download size is now down to 923 Megabytes, the exact same as the update on Steam. In this mode the tool will create a manifest that is specifically optimized for updating from the old version we used to the new one. Thus it can reuse data more aggressively (overhead doesn’t matter, the data is already there) and create a manifest that only downloads what is absolutely necessary. The sad part here is that we would have likely never seen this actually being used. From all of the games in my library (100+) only one had a delta manifest available for its last update, unsurprisingly, it was Fortnite.
Since this is not done by default this actually is a way in which Epic’s platform performs worse than Steam (I don’t know how Valve addressed this issue specifically, but whatever they do it is done by default). But under normal circumstances it would only be worse by a factor of ~2.4 rather than ~42, which probably would have caused far less complaints from users. So clearly something went wrong elsewhere for the newer build to be so out of whack.
Note about processing time and storage requirements: Creating the initial chunks and manifest took around 10 minutes, adding the update took another 4–5 minutes, and the optimisation process took around 3–4 minutes. The final “CloudDir” had a size of around 38 GiB. Everything was done on an i7-2600K running at 4.2 GHz and a 2 TB MX500 SSD.
Mistakes were (probably) made
So this is the point where we go from looking at Epic’s platform to theorising what actually happened here. From the above we can conclude that the patch was created without reusing existing data, but how does that happen?
Based on playing around with the BuildPatchTool and the published manifests I came up with a few things that could have contributed to this problem:
- Epic’s tool requires the chunk old data to be present locally on the machine building the patch, and does not appear to be able to download existing data in the “PatchGeneration” mode
- The build environment might not take into account that the data has to be preserved to build efficient updates. Steam’s patch builder does not require this so it is possible an automated process involving cleaning the output directory was incorrectly reused. It is also possible that Epic’s documentation does not make this clear enough to the developer
- The artifact published had a different “AppName”/”ArtifactId” and might have been a staging build that was then pushed to “Live” instead of rebuilding the patch against existing data
While this is again speculation based on what we’ve learned about Epic’s tools and their platform so far, it gives us an idea that the problem probably lies on the developers end, or more specifically, with their deployment process.
Fixing the situation
So could you somehow fix this and possibly prevent people who are still on the old patch from downloading a massive update, while also not having the people who already updated or freshly installed the game download another patch? Why yes it is, thanks to delta manifests!
As explained earlier one of the biggest features of delta manifests is that they can more aggressively reuse existing data, but what’s more is they can do it even when the patching between two manifests that don’t share any chunks (but do have a lot of the same data in those chunks). And when we run ChunkDeltaOptimise on the manifests in such a case we get this:
Again, a patch just around 923 MB! So if the developer had actually used delta manifests in this case, this whole issue could have been averted. And unlike the normal patch generation this mode appears to work even with already uploaded builds where the game data is not still available offline. Though again, nobody except Epic themselves is actually using it.
Going forward the developer should consider not deleting the directory, and for future patches reuse the data when possible. And when not possible run the optimisation mode provided by the tool to prevent another full-game-download situation.
If the output of the last upload has already been deleted it may be possible to download the data again, even if Epic’s tools do not offer this. Using the manifest parser from Legendary and the manifest of the latest builds one could relatively easily write a python script to download all the Chunks so they’re available for offline use again.
Conclusion
So now that we’ve gotten through all of that, what can we conclude?
First of all this was very likely not a problem with Epic’s platform. While a normal patch still would have been bigger than on Steam, it would have been only about 2 Gigabytes, which likely wouldn’t have caused any complaints. And if the optimized manifests had been generated that number could be brought down to the same 923 MB as the Steam update.
Based on what we now know the likely explanation is that the deployment process on the developer’s end did not retain the output after the build has been uploaded, preventing the tool from building efficient patches and causing users to have to re-download the entirety of the changed files.
Finally if you’ve read this so far I want to thank you! This was a few hours of tinkering + writing but I had a bunch of fun. And if you like you can help me recoup my spending on caffeine on ko-fi :)
