FSR 2.2: AMD’s Upscaler matures
FidelityFX Super Resolution (FSR) is among the most important tech to come out of AMD’s Radeon group in recent years. Following NVidia’s highly successful DLSS, FSR is interesting because it’s not simple imitation: no use of Machine Learning, no need of ML acceleration. FSR is one case where “oldschool” is a novelty, bucking the trend and the perception that ML is an essential tool for this task. In this essay I discuss the evolution of FSR so far and attempt to answer one question: are we there yet?
From that low-res goal however, an 8K-size discussion of game upscaling emerged including DLSS and XeSS, benchmarking, and a meta-analysis of other upscaler reviews. Just skip any parts you don't care about :-)
Brief History of FSR
FSR 1.0 (June 2021) was a stopgap, with value in the right place. It can run anywhere and it beats all previous spatial upscalers used in game engines or display hardware. It starts with Lanczos and improves on it.
Image quality is OK in games that have easy rendering. By “easy” I don’t mean Tetris; I mean avoiding most elements that are hard for upscalers like particle systems, fast animation, or moiré- and aliasing-inducing geometry like wire fences or dense foliage. A good example in this category, but still demanding enough to need upscaling on the RX 580 I had at the time, is Terminator: Resistance. Even after I moved to a 6800 XT I found FSR 1.0 necessary and satisfactory in Resident Evil: Village.
But FSR 1.0 could already lose to good TAAU implementations in some games that avoid the same hard cases. Example: Quake II RTX, a classic with very simple geometry and animation for today’s standards, with a new path-tracing engine that makes rendering slower but not harder for upscaling.
Even in a simple/easy scenario like Quake II, spatial upscalers can have some trouble. The picture above is a zoom into distant walls that have a pattern of horizontal stripes with visible aliasing problems; both FSR and TAAU also add shimmering/instability there even without movement.
FSR 2.0 (May 2022) is a temporal upscaler that combines information from previous frames, motion vectors, and other data like disocclusion masks to support games with fast action or complex visual effects. The initial version was clearly a step above all preexisting temporal upscalers — except DLSS 2. And it still has good cross-platform compatibility; the more sophisticated algorithm makes it less profitable for very old GPUs, but it still supports a lot of hardware including current-gen consoles and handhelds.
FSR 2.0 made good progress to neutralize DLSS as a big competitive moat. With my 6800 XT, I could now play games like Deathloop at 3440x1440 max settings with only small compromises in quality; certainly smaller than turning down enough rendering options to get the same framerate.
FSR 2.1 (September 2022) brought improvements in several areas, with two minor updates quickly following. FSR 2.1.2 (October 2022) is right now the best version you can use in many games — you can generally mod the latest 2.1.2 DLLs in games that ship 2.1.0 or 2.1.1.
FSR 2.2 (Mar 2023) is another big update with a sizable list of changes. In my own judgment, based mostly on a single game that I just completed (The Last of Us Part I), FSR 2 now ties with DLSS 2 in Quality mode — but see discussion about image quality later.
Meanwhile, NVidia is not one to rest on its laurels. Competition for DLSS was obviously coming; it now includes FSR 2, Intel’s XeSS and maybe even Unreal 5’s TSR. We now have DLSS 3’s Frame Generation, exclusive of the RTX 4000 series. FSR v3.0 should soon add FG as well. Other DLSS exclusives include DLAA and video playback upscaling, less popular and useful but certainly nice-to-haves.
Is Machine Learning Essential?
I should admit I'm partial to FSR since v2.0. This is an engineer’s bias: FSR started delivering >80% of the quality of DLSS, and now approaching 100%, at 10% of the computational cost; with wide hardware compatibility including older GPUs that need upscaling the most; it’s portable and open source, delivering a consistent cross-platform presentation for developers.
The ML approach has an advantage for higher scaling factors, but it’s still early to tell if it’s the ML or it’s just the maturity of each system. The case for ML was made stronger by Intel’s XeSS, more recent than FSR 2 but matching or beating it on Arc GPUs that can run the full ML model.
“AI magic” doesn’t exist. The best possible upscaler will have limitations. This boils down to basic information theory: just like a lossless compressor cannot achieve arbitrary compression rates, an upscaler cannot recover missing pixels. But it can cheat! For example, an upscaler could have an ML model that knows how a patch of grass should look like, so when it’s trying to upscale a region of a frame recognized as grass it can do a good job filling detail — but it’s invented detail, sourced from its model.
Think of upscaling as the Star Trek transporter. In today’s episode, you put a can of Coke in one end but subspace interference caused only 25% of the bits to reach the other end. The decoder has to fill in molecular detail but its training data didn’t include every drink and a can of Pepsi materializes. It's close enough, nobody can tell the difference except for Vulcans.
Benchmarking Upscalers: Image Quality
Image quality is hard to benchmark because human vision is complicated. Technologies like VRS and foveated rendering can drop quality in big chunks of the frame with no perceptual impact. Game & GPU reviewers that discuss upscalers always get called out for “pixel peeping” stuff that can't be noticed in normal gameplay, only with zooms or slow motion.
There are tools to measure perceptual quality such as VMAF, others that compare two videos and produce a visual diff. I find it odd that no game / GPU reviewer uses those tools to have a more objective standard than their own “lying eyeballs”. Perhaps the cost/difficulty of customizing tools for this niche use-case is high. Still, gaming tech reviewers performed detailed upscaler comparisons. I like the methodology developed by Alex Battaglia at Digital Foundry, best demonstrated in their deep dive of XeSS.
The review from Hardware Unboxed is the most recent, aimed at FSR v2.2 and testing 26 games that feature both DLSS and FSR 2. I found their results mostly good but disagreed on a few points. In their 4-level relative scoring from Tie to Significantly Better (+++), levels are separated by criteria like "easily noticed in gameplay", with Tie split from Slightly Better by differences that are by definition hard to notice. There's no scoring in non-relative terms i.e. frequency and severity of artifacts compared to native rendering, although that's implied from their relative scoring we can't see that data. Finally, I personally also couldn't see a few of the differences in quality that the YouTube video supposedly shows even after many attempts pausing and re-watching the relevant sections.
Reviewers that make this level of discrimination always offer a disclaimer that "YouTube compression is crap". Not defending YouTube, but in a video in 4K resolution where 95% of the footage has slow action, any artifact or differential quality that can be hidden by compression is also too subtle to notice in normal gameplay. Remarkably for 1440p tests, where encoding the video at 4K should offset most compression losses. I trust HUB, I'm sure they saw the quality differences; this is about scoring criteria.
HUB's review concludes that the gap between DLSS and FSR did not change from earlier versions, in contradiction to their own scores. Let's annotate those scores with the FSR 2 version used by each game:
Every game that scores any Tie uses a least FSR v2.1; FSR v2.0 is never a match for DLSS at any game, quality level, or resolution. And every game with FSR 2.2 scores a Tie or DLSS+ in 4K/Quality. These correlations are visible even over the noise of issues in the tested games:
- The single game in this table that really uses FSR v2.2 is TLOUp1; all others use months-older prereleases. None of those games is patched to the final v2.2 yet, and you/reviewers can't do that update because non-official FSR builds are typically linked to the executables, not shipped as standard DLLs (I confirmed this for F1 2022). Similarly, God of War uses a pseudo-FSR v2.0. This is a DirectX 11 game and FSR 2 only supports DirectX 12 and Vulkan, so AMD put together a one-off hack for GoW; that's why GoW will never have an official update to a newer FSR.
(Sadly GoW is the game that DF picked for their FSR v2.0 vs DLSS face-off. But it's hard to blame DF or HUB for testing games that AMD promoted in their marketing for new FSR releases, because those are "big" games.)
- Some games with a newer FSR like Forspoken or TLOUp1 are widely considered buggy launches. (Update: a TLOUp1 patch released after HUB's review fixed bugs in both FSR and DLSS integration.) Meanwhile RDR2, stuck with FSR v2.0, is a paragon of rendering. Implementation quality can also affect other upscalers, for example, Dead Space is another botched release and HUB points out its poor job with DLSS so the close scores are given not because FSR v2.1 is very good but because DLSS is almost as bad.
Despite all the points above, HUB's review illustrates well a very objective disadvantage of FSR 2: its maturity is much more recent; even if we find that v2.2 is the real thing that competes with DLSS, only one game in HUB's table includes the final release from March 2023. Some games also don't have the latest DLSS, but its maturity is two years ahead of all competition. A game that ships an one year old version of DLSS 2 has pretty good DLSS, only missing incremental refinements. But a game stuck to the 10 months old FSR v2.0 is way behind. I hope reviewers take a new look at this once we have more than a single game featuring FSR at v2.2 or better. I would also pick XeSS v1.1 as the first release of Intel’s upscaler that should be used in any “showdown” with DLSS and FSR 2.
Benchmarking Upscalers: Performance
Performance is much easier! We can measure frame rates and times very precisely so one would expect an easy method for benching upscalers. Well, we’re halfway there. You need benchmarks (synthetic or real games) that support multiple upscalers. With a high-quality integration. Preferably using the latest major update of every upscaler. For games you need a built-in benchmark, to avoid bias from scene selection or reproducibility.
I was hopeful when I found Tellusim’s benchmarks. Tellusim is a real-time rendering engine supporting multiple APIs and platforms, raster & RT; but it's not used for games (for one thing, its RT denoising is poor). They make GravityMark, a rendering benchmark that’s interesting if limited: the focus is on a scene with tons of moving objects, pretty cool with RT. Recently they announced an upscale benchmark that does it all: DLSS 2, XeSS v1.1, and FSR v2.2. It’s work in progress, you can download a development build. There are many scenes with separate launchers. In my system some of them crash and others have bad rendering artifacts. The ones that work have upscaling problems like ghosting in slow-moving objects, a telltale of poor integration e.g. missing motion vectors or disocclusion masks. These issues can impact both quality and performance, so the benchmark is not yet in a usable state even just for performance testing.
The 3DMark Upscaling Tests
In March 2023, UL updated the popular and excellent 3DMark with the AMD-sponsored FSR 2 feature test. This completes a suite that includes the NVIDIA DLSS and Intel XeSS tests. I find it unfortunate that these are separate rendering scenes so the framerates are not directly comparable across upscalers. The tests are easier to use for measuring each upscaler across different hardware and scaling factors.
All three tests seem designed around the strengths of the sponsor’s GPU or upscaler. The DLSS and XeSS tests have a lot of glass and shiny metal, showing off their focus on Ray Tracing. Meanwhile, the FSR 2 test adds particle systems (soldering sparkles, smoke with volumetric lights and shadows cutting through). It also uses RT but less aggressively, well except for a hilarious disco ball that looks out of place in a futuristic setting.
I wish UL would just add support for all three upscalers to the SpeedWay test, which I presume was designed without engineers of the DLSS, XeSS, or FSR teams picking and choosing elements that make them look good. Meanwhile, we’ll have to make do with the existing upscaling tests.
All 3DMark tests are typically never updated after their first release, except for minor bugfixes, to not break the sequence of comparable scores. But tests can be extended with new variants: for example, the DLSS test allows you to pick DLSS 1, DLSS 2, or DLSS 3. But once a “DLSS N” option is there, it’s frozen. The DLSS 2 test uses DLSS v2.1.29.0. The more recent DLSS 3 test uses v2.4.0.0, but it always enables Frame Generation. You can also mod the benchmark for updates that are compatible with the shipped DLLs.
Benchmarking the Benchmarks
The CPU load of most game engines scales poorly with resolution; on the GPU, performance scales better with pixel count. All 3DMark upscaling tests use Ray Tracing, making the GPU work harder. RT games can be heavy on the CPU as well, but all 3DMark graphics tests avoid factors that cause that — big open worlds, asset streaming, NPC/enemy AI, etc.
The theoretical limits for upscaling performance are 2.25X for Quality mode and 4X for Performance. It’s no surprise that the game that gets closest to those limits is Quake II RTX, where “Quality” (resolution scale 65%) hits the full 2.25X and “Performance” (50%) scores 3.46X of native. With its heavyweight path-tracer on the GPU paired with an otherwise ancient game engine, the game’s GPU:CPU imbalance makes it an almost ideal upscaling benchmark. Unfortunately, the only modern upscaler supported is DLSS 2. Quake II even with RTX is also not a good proxy for mainstream, modern game rendering.
Since each upscaler uses a different benchmark, let’s start by comparing them at native, full-resolution rendering (no upscaling). My primary GPU is a 7900 XTX that can’t do DLSS, so I’m using an RTX 3080ti for all charts / numbers that include all three upscalers.
All tests run 61–63% as fast in 1440p compared to 1080p, 47–49% as fast in 4K compared to 1440, and an identical 30% as fast in 4K compared to 1080p. This is very good, probably an effect of all tests using the same rendering engine, APIs, features, and even shared assets. If we normalize upscaling scores relative to native rendering at the same resolution, they should be very fair to compare across upscalers.
The 3DMark FSR 2 test
The FSR 2 test had been announced for some time and its release one month after the FSR 2.2 release is probably not a coincidence. We didn't have a 3DMark FSR test for FSR v1, v2.0, or v2.1; and we'll never have those, the new benchmark starts with v2.2. In contrast, DLSS and XeSS tests shipped quickly after the first releases of those upscalers. Perhaps FSR v2.2 is the first version AMD feels good enough to show off.
Cross-benchmarking upscalers
FSR v2.2 is the clear winner in performance. DLSS always comes in second place and XeSS is always last. These results are very consistent and FSR’s lead is larger in both higher scaling factors and in higher output resolutions, so Performance level at 4K is the largest win.
These results are on an RTX 3080ti, biased for DLSS with the Tensor Cores. XeSS is the most disadvantaged system since, unlike FSR, it's also designed to use ML inference that wants specialized accelerators but only optimized for Intel's XMX cores in Arc GPUs, not for RTX Tensor Cores.
We can argue that you get what you pay for: DLSS has better image quality. And the performance gap is more significant at the extremes. If we start with a game that does native 30fps at 4K, DLSS/Performance would get you 76fps but FSR/Performance gets you 86fps, on an RTX GPU. But this gap is smaller in the more popular Quality mode and in most real games.
I made a final chart only for FSR v2.2, comparing its performance scaling in the RTX 3080ti versus the 7900 XTX. Surprisingly, FSR’s performance scales better on NVidia's GPU than on AMD’s home turf, except for the 4K/Quality test. This might just be different scaling of the renderer between the two GPUs; I wish I had a closer match of Radeon/RTX GPUs (the 7900 XTX is significantly faster than the 3080ti even with RT). In any case we can certainly state that FSR v2.2 is not biased for Radeon, it runs really well on the competing NVidia GPUs.
Intel should optimize their XeSS for Tensor Cores, running the full-quality ML model on RTX GPUs. XeSS today is not competitive in any GPU other than Intel’s Arc, still struggling for any adoption. XeSS's image quality is difficult to compare to FSR 2 considering the dearth of titles with the latest FSR at this time, but the latest XeSS (v1.1) seems to beat previous FSR (v2.1) at higher scaling factors even in non-Arc GPUs. That's still too little when combined with a significant deficit in performance. Full support for RTX GPUs would make XeSS runs well on the leading hardware, making it more credible as a standard, cross-platform upscaler — a category where FSR 2 now rules completely unchallenged.
Conclusions: Room for growth
My performance tests show that FSR 2 is very efficient. It scales better than all competing upscalers. Both DLSS and XeSS use ML inference that's more computationally intensive, but I'm a bit surprised that at least for DLSS in the Ampere architecture this acceleration is not sufficient to offset the overhead of ML-based upscaling. In any case, it looks like AMD has more headway to add enhancements than the competition; they could probably increase their frame-time overhead by 50%, at least on RDNA 3 or other higher-end targets like Navi 21 or GA102, and still compete in performance with other upscalers. I suspect we will eventually see FSR improvements that will only apply to some GPUs, not locked by special execution units but only limited by sufficient shader performance.
As I write this AMD is preparing FSR v3.0, adding Frame Generation and perhaps other features as well. We don't know at this time how FSR v3.0 is implemented; RDNA 3 supports ML acceleration but not at the same level as NVidia's or Intel's architectures, and it also doesn't have an equivalent of NVidia's Optical Flow hardware for FG. The shaders will have to do the hard work again and it's likely that AMD will stick with handcrafted algorithms. The good news is that there’s a good chance that FSR v3.0 FG will support more than just RDNA 3. But image quality will be again the interesting battleground given that DLSS 3’s FG doubles down in reliance on specialized accelerators, locking out even high-end Ampere.
