Windows DXVA2 (via Direct3D 11) Support in GStreamer 1.17

DXVA2 based hardware accelerated decoding is now supported on Windows, as of GStreamer 1.17.

This is a list of supported codecs for now

• H.264 (d3d11h264dec)
• HEVC (d3d11h265dec)
• VP9 (d3d11vp9dec)
• VP8 (d3d11vp8dec)

What should I do to use them?

No special steps or dependencies are required to build this new element indeed.

The above listed new decoder elements are part of the d3d11 plugin in GStreamer. It doesn’t require any special build time dependencies and/or libraries as everything is already provided by the Windows SDK. Once it has been built, the only requirement is whether your hardware (i.e., GPU) is able to support hardware decoding or not.

NOTE: This is a hardware decoding feature, so if the VM does not provide a way to pass-through the GPU, it will not work inside the VM.

When you run gst-inspect-1.0 it will show a list of available decoder elements. This is an example what you might see with gst-inspect-1.0:

[gst-master] PS C:\Work\gst-build> gst-inspect-1.0.exe d3d11
Plugin Details:
  Name                     d3d11
  Description              Direct3D11 plugin
  Filename                 C:\Work\GST-BU~1\build\SUBPRO~1\GST-PL~3\sys\d3d11\gst
d3d11.dll
  Version                  1.17.0.1
  License                  LGPL
  Source module            gst-plugins-bad
  Binary package           GStreamer Bad Plug-ins git
  Origin URL               Unknown package origin

  d3d11vp8dec: Direct3D11 VP8 Intel(R) Iris(R) Plus Graphics Decoder
  d3d11vp9dec: Direct3D11 VP9 Intel(R) Iris(R) Plus Graphics Decoder
  d3d11h265dec: Direct3D11 H.265 Intel(R) Iris(R) Plus Graphics Decoder
  d3d11h264dec: Direct3D11 H.264 Intel(R) Iris(R) Plus Graphics Decoder
  d3d11videosink: Direct3D11 video sink bin
  d3d11videosinkelement: Direct3D11 video sink
  d3d11colorconvert: Direct3D11 Colorspace converter
  d3d11download: Direct3D11 downloader
  d3d11upload: Direct3D11 uploader

The output might be slightly different in your case. For instance, the device name might be something different than “Intel(R) Iris(R) Plus Graphics”. That’s expected :) It will vary based on your hardware vendor and device naming.

Also, if the list doesn’t contain elements for some codecs (for instance, d3d11vp8dec), it’s very likely that your hardware doesn’t support decoding the codec (for example, some Nvidia GPUs don’t support VP8 decoding).

Moreover, if you have multiple GPUs on your device, you will see separate per-GPU decoder elements, with longer names for instance d3d11h264device1dec or so.

Why do I need new D3D11 decoders?

GStreamer already ships with two vendor-specific decoder implementations: one is the Nvidia (aka NVCODEC) plugin and the other is the Intel MSDK plugin. So what’s the benefit of this new implementation?

The main advantage of vendor-specific APIs is supposed to be that they are expected to perform better than generic APIs like DXVA2. In most cases this might be true but sometimes it is not true. The performance and reliability can vary depending on how well it’s integrated in a framework. Moreover, not just the decoder implementation itself, but “how media pipeline was configured in an application” is a very important factor for performance and reliability.

In summary, the strengths of the new d3d11 decoders are:

• Zero-copy playback with d3d11videosink
• Vendor-independent implementation
• UWP support

Zero-copy playback

On Windows, both NVCODEC and MSDK plugins will copy decoded data into a new memory space (cuda, gl, or sysmem depending on the details), which consumes more memory and will make applications slower.

However, when d3d11 decoder elements are configured and at the same time d3d11videosink element is selected for rendering, the decoded data will be passed to d3d11videosink without any copy operation.

The only memcpy-like operation will be color space conversion (YUV to RGB format), but it’s often an unavoidable operation, because YUV is not supported as the render format for most rendererers (Windows DirectComposition seems to support it, but that’s a special case).

Vendor-independent implementation

This is a very useful aspect of this new implementation. Due to the fact that DXVA2 and D3D11 are standard APIs on Windows and provided by the OS, in theory NO vendor-specific consideration is needed (in reality, app-specific workarounds would likely be needed due to buggy vendor-specific driver behavior).

So there would be no reason for the application to write hardware-specific code in the general case. Moreover, these new elements also work on AMD GPUs since we previously only supported Intel and Nvidia on Windows. They should also work with other GPUs supported by Windows 10 such as Qualcomm, but I have not tested that yet.

UPDATE: I’ve tested this on Surface Pro X (Qualcomm) and it worked very nicely

UWP support

When running on UWP (Universal Windows Platform), most (possibly all?) hardware-specific operations are required to be handled via the native Windows graphics layer such as Direct3D11/12. Due to this, these new d3d11 decoder elements are a requirement for hardware decoding of video on UWP. When I tested this with a UWP application on my laptop, it worked quite well (as I expected)!

Note that UWP is not officially supported by GStreamer yet, but I am expecting it will be possible soon thanks to the efforts of Nirbheek Chauhan who is an active GStreamer maintainer and also maintains Cerbero, the build system of GStreamer. Related to UWP, a very interesting talk from him is available here: https://gstconf.ubicast.tv/videos/gstreamer-windows-uwp-and-firefox-on-the-hololens-2/

We’re excited to see new Windows specific features coming to GStreamer more and more. Stay tuned for more news! :)