Screen Share Test In Unity

Source: Shutterstock

Video conferencing apps like zoom, skype, google meet etc. have become immensely popular in past three years due to the pandemic that shook the world to its core. These applications come with so many features like whiteboard sharing, screen sharing, chatting, remote access, annotations and what not. Now think of a similar application but in 3D. Experiencing such features in a 3D application would be more realistic. Isn’t it?

With XR becoming a trend, there comes a need to test such features in our 3D applications as well. So in this blog we will share an insight on our screen share testing journey in unity.

Development specifications:

We’ve considered an application where screen share is implemented using Agora Engine. The test strategy shared in this blog can be applied to applications with similar implementation. The approach might help in other applications as well.

In our 3D application, we have a gameobject called “DisplayCanvas” under which we have “DisplayPlane” gameobject. In the DisplayPlane, there is a component called RawImage. When the user clicks on share-screen button in main scene, user’s active screen is shared and all the other avatars present in that room can view that screen. So when screen is shared, we observe that a new texture is generated in the “RawImage” component of DisplayPlane. When the screen is not shared, that texture value is set to Null.

DisplayCanvas >> RawImage>> Texture

Test:

Validate that when user clicks on share-screen button, screen appearing on DisplayCanvas is same as user’s active screen.

Test Steps:

1. Click on the screen share button
2. Take the screenshot of entire screen
3. Get the Raw Image component of DisplayPlane.
4. Convert the texture image as well as the screen shot to Texture2D
5. Make both the Texture2D as readable.
6. Compare both the textures pixel by pixel.

Click on the screen share button:

Using Arium, we implemented UnityPointerClick on screen-share button.

GameObject screenShareBtn = _arium.FindGameObject(“IconCanvas/ShareDisplayButton”);
_arium.PerformAction(new UnityPointerClick(), screenShareBtn);

Now the screen is shared.

Taking screenshot of the entire screen using Unity:

Taking screenshot of complete screen (not just game view) in Unity is quite a tedious task. The “CaptureScreenshot” method of “ScreenCapture” class is used just to take the screenshot of game view. We thought of multiple approaches before reaching the final one.

At first, we thought we will use the “CopyFromScreen” method of System.Drawing.Imaging namespace of C# to capture screenshot. But we figured out that this approach works fine only with older versions of .NET framework.

Later we came across a library called Pranas for taking screenshot. But we discovered that this library is not compatible with Mac, so again we could not use it.

Next we found that we can take screenshot of entire screen directly through terminal. We already have below mentioned command for this.

screencapture test.jpg

where test.jpg will be the name of the screenshot.

So now we wanted to execute this terminal command through our script. For this, we used the Process class. The FileName argument specifies the terminal command to be fired. Here, the command is screencapture . The Arguments define at which location and with what name the screenshot will be saved. We then used Start() method to fire the command.

For better code reusability, we created a separate method for implementing this. We called the method CaptureScreenshotThroughTerminal()

public static void CaptureScreenshotThroughTerminal()
{
 var p = new Process
    {
    StartInfo =
    {
       FileName = “screencapture”,
       Arguments = “path_to_save_screenshot/name_of_screenshot.png”
    }
 }.Start();
 }

This way we capture entire screen screen-shot in unity.

Get the Raw Image component of DisplayPlane:

After capturing the screenshot, we need to get the RawImage component of the DisplayPlane, we did that by using GetComponent method of GameObject class.

RawImage displayPlaneTexture = _arium.FindGameObject(“Canvas/DisplayPlane”).GetComponent<RawImage>();

Convert the texture image as well as the screen shot to Texture2D:

We need to convert images to Texture2D so that we can compare both our images using GetPixels method which is available in Texture2D class.

So, for converting screenshot to Texture2D, we will first load the screenshot from it’s path and then change it to Texture2D by explicit type casting. Converting Texture to Texture2D is not complex. We just need to use the command displayPlaneTexture.texture as Texture2D

Texture2D screenshot = (Texture2D)AssetDatabase.LoadAssetAtPath("Assets/Images/Textures/screenshot_134.png", typeof(Texture2D)); 
Texture2D displayPlaneTexture2D = displayPlaneTexture.texture as Texture2D;

Make both the texture 2D as readable:

You must be wondering why are we even doing this step? Readability is required when we need to read, write, and manipulate the pixel data of any texture. In the next step, we will be comparing screenshot Texture2D and displayPlaneTexture2D pixel by pixel.Hence, we essentially require this step.

We have to use the “SetTextureImporter” function of “SetTextureImporterFormat” class for this purpose. The SetTextureImporter takes the Texture2D as first argument and a boolean as second argument. The boolean argument specifies if we want it to convert it to readable or not.

SetTextureImporterFormat.SetTextureImporter(displayPlaneTexture2D,true);             SetTextureImporterFormat.SetTextureImporter(screenshot,true);

Compare both the textures pixel by pixel:

Now that we have two readable Texture2Ds , next task aligned is to compare both of them using their pixels. For this we created a class called CompareTexture taking two Texture2D type arguments. Next, it is creating a Color array of pixels by fetching pixels using GetPixels method. After that, both the arrays are now being compared through their length. If length of both the arrays is not same , it will immediately return false. If it is same then it will compare each element of the array throughout it’s length. If any pixel is not matched in between , it will return false and our test would fail. Otherwise, if all the pixels match then this CompareTexture would return True.

public class CompareTexture{
public bool CompareTextureFunc(Texture2D first, Texture2D second)
{
Color[] firstPix = first.GetPixels();
Color[] secondPix = second.GetPixels();
if (firstPix.Length != secondPix.Length)
{
   return false;
}
for (int i = 0; i < firstPix.Length; i++)
{
   if (firstPix[i] != secondPix[i])
   {
      return false;
   }
}
return true;
}
}

Through test script we can easily call this CompareTextureFunction by creating instance (cmpTexture) of this class.

cmpTexture.CompareTextureFunction(screenshot,displayPlaneTexture2D);

Validate using Assertion whether the image getting displayed in DisplayCanvas is same as the one being shared.

This completes our screen-share test. :)

Thank You for reading!!!!!

Authors:

Medium Profile link :
https://medium.com/@alisha.raizada