How does Shadow adapt its stream to all devices?
Shadow is a cloud computing service giving access to a full Windows PC on various devices like Android, iOS, Linux, Mac, and PC. The teams strive to make the user feel like he is playing on his local device buffed with a gaming hardware setup, and display management is a key point for that.
Shadow infrastructure is based on data centers spread all around the globe. These data centers provide virtual machines connected to one or multiple hard drives, a CPU, a GPU, but no physical display. Here is the point, a display (either physical or virtual) connected to a video interface is required to acquire the frames generated by Windows with high-performance capture APIs.
1. How to get a display?
A couple of solutions were considered bringing pros and cons.
1.1 Using a physical dongle
The first idea was to plug a physical dongle on every board in data centers. This had the advantage of letting the graphic adapter think that there is a physical display accessible by the virtual machine at all times but it brings several disadvantages too.
Counting hundreds of thousands of graphic cards, and adding a physical dongle to each one of them would induce a cost not negligible and would make the infrastructure hardware-dependent. Indeed, manual intervention would be necessary directly into the data center to add or change a dongle. Not to mention the fact that a faulty dongle would make the graphics card unusable for end-users.
Technically speaking, the type of dongle determines the performance. HDMI cables provide lower bandwidth than DisplayPort for instance. Let’s take the latest versions of these standards for comparison.
HDMI 2.1
- Supports 3840x2160 (4K UHD) with a 120Hz refresh rate
- Supports 7680x4320 (8K) with a 60Hz refresh rate
- 48Gbps bandwidth
DisplayPort 2.0
- Supports resolutions higher than 7680x4320 (8K) and 8K with a 120Hz refresh rate
- 80Gbps bandwidth.
Even if a DisplayPort 2.0 would do the job for a good streaming experience now, Shadow is dedicated to improving the service provided. The day Shadow will need to have a higher bandwidth for a better streaming experience, all the dongles will have to be changed.
The scalability of this solution was a big issue and thus, was not adopted by Shadow.
1.2 Creating a virtual display
In simple words, a virtual display is a software simulation of a physical one. Graphics card manufacturers offer the possibility to create virtual displays using proprietary API (Shadow is using Nvidia and AMD official APIs).
This solution brings a lot of flexibility and overcomes the problems caused by the use of a physical dongle. Virtual displays can be added or removed dynamically on video interfaces exposed by the GPU which offers the possibility to create virtual displays with different video interfaces depending on GPU capabilities. Moreover, having more than one virtual display becomes conceivable.
The main drawback is that there is no certainty that a virtual display is available in case of error during creation. Shadow also becomes dependent on the manufacturer’s API. However, scalability is the game-changer, and that’s why we’re going to dive into this path.
2. How to configure a virtual display?
The virtual display configuration, basically its capabilities, goes through another concept: the EDID.
2.1 EDID
EDID, or Extended Display Identification Data, is a standardized metadata structure that is communicated by the display to a source defining its capabilities. These metadata include information such as available resolutions, manufacturer’s name, serial number, and several other insightful information (see figure 1).
EDID also specifies the video interfaces supported (see figure 2) and audio configuration: channels, sampling frequencies, and sampling bit rates supported which is also important for the Shadow audio stack.
The challenge is to know which EDID has to be used to configure the virtual display.
2.2 Using a unique EDID
At first, we considered using a unique EDID for configuring all the virtual displays created, meaning they all would have had the same capabilities.
On the one hand, this EDID would have been tested and refined by removing all unsupported capabilities and adding all the ones supported (eg. resolutions, refresh rates, HDR, etc) by Shadow. Furthermore, it would have been applied only once per virtual display (for creation), reducing the risks of display errors.
On the other hand, one of the goals of Shadow is to provide an experience as close as a local gaming experience. Imagine a user setting a 5K resolution in the virtual machine when its local hardware does not support decoding such a stream. Proposing display or audio capabilities on the virtual machine that would deteriorate the streaming experience is unacceptable.
How to be sure that the EDID used to create the virtual display will have capabilities supported by the end-user local hardware?
2.3 Forwarding the EDID of the end-user
The solution that naturally comes to mind is to simply forward the end-users EDID. The end-users would retrieve all the local display capabilities of their local computer.
Unfortunately, some platforms like Android or iOS do not allow access to the local EDID at all. Shadow is a streaming service available on different platforms, relying on the end-users EDID does not allow us to cover all of these.
This can cause a compatibility problem in the other direction. The local EDID may have some capabilities not supported by Shadow (eg. audio configurations or very high resolutions) and, because the Windows display stack is sensitive, it can produce BSOD when applied to the virtual machine.
The best solution appears to let the end-users have their local display capabilities but restrict them to the ones supported by Shadow.
2.4 Building a custom EDID
Shadow decided to base its display management on a controlled and tested EDID on the virtual machine. It contains basic capabilities such as resolutions restricted to FHD with a 60Hz framerate and lower, support of HDMI and DisplayPort adapters to be supported by most graphic cards for instance.
To let the end-users feel like they are on their local computer, the capabilities of their local display supported on Shadow virtual machines are inserted into the custom EDID. The end-users win on both counts and Shadow is limiting the side effects of unhandled display and audio settings.
Note that the basic EDID is used as a fallback if the custom creation of a virtual display fails using the custom EDID.
Conclusion
There have been several iterations before finding the solution described in this article. The ultimate goal was to provide the end-users with a seamless experience to make them forget they are on a cloud streaming service. We found a way to get the best performances by overlapping both the virtual machine and the end-user local computer display capabilities.
We named that feature “fit to screen”!
