The reality of today’s world is that display resolution is growing at a pace with which
GPUs are finding it very hard to keep up. Add to that the advent of Real-Time Ray
Tracing, PC GPUs need a way to free up rendering budgets to provide desired
performance levels. This is where image reconstruction and upscaler techniques like NVIDIA DLSS,
AMD FSR and Intel XeSS come in.
Image Upscaling is a pretty old technique in computer graphics, but only recently has
it been integrated into game rendering. 8th Generation Pro consoles have used
Checkerboarding to solve a similar problem, however, it’s not an upscaling technique
per se.
NVIDIA Deep Learning Super Sampling
The first image upscaler we will be looking at is NVIDIA’s bespoke AI-Driven Deep
Learning Super Sampling / DLSS. Even though the technique has ‘Super Sampling’ in its
name, it’s not a super sampler at all. DLSS uses deep learning algorithms to upscale
lower resolution images into higher resolution ones. It does it the same way Temporal
Anti Aliasing provides anti-aliasing. TAA uses previous frame data and motion vectors
to clean up the present frame, DLSS does that too. However, TAA renders at the final
target resolution and then combines frames whereas DLSS allows faster rendering at a
lower input sample count, and then infers a result that is a similar quality to the TAA
result but with half the shading work.
DLSS launched with these principals back in 2018. DLSS 1.0, as most people call it, had
some big issues. 1.0 had to train the Deep Neural Network(DNN) on a per game and
per resolution basis. This was a very inefficient way to do Upscaling as it meant
significant extra effort from game devs and Nvidia Engineers to add the feature in a
game. 1.0 also considered games deterministic, which is not always true. This
approach made it unsuitable for games that had variability in its coding and it fell of
hard trying to upscale such games.DLSS 1.0 also did not use motion vectors.
Thankfully, these issues were solved by the release of DLSS 2.0. This version of DLSS got
rid of the per game per resolution training regime and moved towards a much more
generalized approach. It uses a single DNN for every game and incorporates Temporal
Vectors to increase accuracy. It also has a tweakable sharpening component to it.
Nvidia trains the network on thousands of 64xSS images( Ground truth). This allows DLSS
2.0 to reconstruct images with highly detailed textures and images (mirroring native
quality) with the temporal data acting as a reference. This also cuts the huge
deployment and maintenance costs associated with integrating DLSS into a game and
provides much more seamless integration for game devs.
DLSS 2. x has been a huge success for Nvidia with an evergrowing list of games that
support the feature. However, it is proprietary. DLSS needs Tensor cores (First launched
with the Volta Architecture) to run and as such is locked to just the GeForce RTX and
Quadro RTX lineup.
AMD FidelityFX Super Resolution
AMD’s answer to DLSS is Fidelity FX Super Resolution(FSR). In practice, it’s a highly
modified Lanczos upscaler present in most GPU driver Control Panels already. Now you
might be wondering what Lanczos is, fret not I have the answer. In plain words, it’s a
mathematical formula, created by Cornelius Lanczos which can be used as a low pass
filter to smoothly resolve a digital signal(in this case our lower resolution image).
FSR, unlike DLSS, is fully spatial technology. It means that unlike DLSS it does not rely on
Temporal Vectors in its reconstruction, which in theory will mean fewer motion
artefacts. It resolves its images by looking for high contrast edges and then resolving
them smartly into a higher-resolution grid. This technique gets rid of the shimmering and
jaggies that most upscalers have. FSR also inherits the AA present in the game by
default and does not replace it (unlike DLSS). This means that imperfections caused by
Temporal AA cant be resolved by FSR, unlike DLSS. The Lack of motion vectors also
means in surface details resolve worse than DLSS does. This limits FSR’s usefulness to only
images 1440p and above as under 1440p the algorithm simply does not have enough
data to upscale effectively compared to DLSS.
Even with its drawbacks, it is still a solid upscaler at higher resolutions. It is also miles
ahead of DLSS 1.0 and it is open source. Game integration is pretty easy and since it
does not rely on any DNN, the upscaler requires no upkeep from AMD. FSR is also vendor agnostic.
It can run on either FP 16 compute shaders ( RX 6000/5000/ Vega / RTX 30/RTX 20/ Intel
UHD ) or run on FP 32 ( RX 500/400/GTX 10 and 900).
Game support is less than DLSS 2. x but FSR is in its infancy. By DLSS 1.0 standards it is
already doing much better. It also gives a lease of life to older GPUs like 10 series and
Vega to be able to drive games at a higher resolution, which it may not have been
able to do natively. FSR also has different scaling options from 1.3x to 2.0x.
Intel Xe Super Sampling
Intel is a relatively minor player in the GPU space. However, since 2017, Intel has been
trying to get into the Graphics side of PC computing with the acquisition of Raja
Koduri(AMD), Tom Petersen(Nvidia) and others. So when Intel announced Arc
Alchemist, their brand new dGPU solution, it was no surprise that an upscaler like XeSS would
also, come up.
XeSS, like DLSS, uses a DNN to upscale its images. Moreover, it uses both spatial and temporal
motion data to drive its upscaling. Its implementation is very similar to DLSS 2. X, which is
a huge positive over AMD FSR. However, since the GPUs are not released
yet(launching 2022)every detail about the technology is not known at this point. Intel in
their presentation pointed out Ghosting issues ( present to a certain degree with DLSS )
and shimmering (Present in FSR at lower quality settings ) as potential drawbacks of
their competitor’s solutions and are promising a much better resolve of such issues
using their solution. But we still don’t know how they will achieve it. With that said, what
Intel is promising isn’t something that’s out of their reach if they’ve done their
homework.
Like Nvidia, Intel has also built bespoke hardware into their Xe HPG GPUs. These units
are called XMX matrix maths units and they function in a similar way to the Tensor cores
inside RTX GPUs. However, unlike Nvidia, Intel’s solution will work on competing
hardware, but won’t be as performant. Non-Intel Hardware will run XeSS with the DP 4a
instruction set. This means Radeon VII and above AMD cards and Nvidia 10 Series and
above GPUs will run XeSS.
Intel also plans to open up its XeSS SDK. At this point, we don’t know when this will
happen but it’s a good sign. Proprietary is always a hindrance for progress and Intel
committing to open standards is a good thing. In the meantime, game developers will
be able to get their first look at the technology later this month, when Intel releases the
initial, XMX-only version of the XeSS SDK. This will be followed by the DP4a version, which
will be released later this year. Presumably, it will have the same or similar upscaling
multipliers like FSR and DLSS.
Upscalers : The Future of Graphics
The age of raw pixel counts is dead. Moreover, GPU manufacturers are no longer looking to waste resources trying to ground the highest possible pixel clarity. Rather, the focus now is on techniques to get more out of a feasible amount of pixels. As such, upscalers are the unsung heroes of this effort. All of them have their strength and weakness and each one offers a unique way to tackle the problem put in front of it. DLSS, in conclusion, offers the best option on the market right now, but that may very well change in the coming days.
This concludes our brief look at the different Upscaling options available in the market
right now. However, I would like to point out that these are not the only available
upscalers present in the PC space. Most games these days come with TAA and as such
have an upscaler built into the engine. Unreal‘s TAAU is perhaps the best in engine
Upscaling algorithm out there. There is also a Direct ML Upscaler shown way back in
2019. Microsoft has been developing it for quite a while now and presumably will soon
launch on Xbox Series consoles and PC (hopefully with Windows 11).
