Tailor-Made Tools We Use to Test Video Editing Apps
Hi, my name is Victor and I am the head of the QA Department at Rosberry. Over the last 12 years our team has implemented more than 170 projects, of which 28 happen to belong to the photo and video category. In this article I would like to give a short overview of several custom-made tools and libraries we came up with to test Filmm, a new effects and filter app developed by us for both iOS and Android. Its initial release took place on May 08, 2019 and since then it’s already been featured by Apple 15 times in 25 countries, whereas Rosberry team has kept on enhancing and updating the app on a regular basis.
Library of Formats
Users in real life are not actually confined to using only one format of photo and video content they upload for editing. This content might be:
- captured on the Filmm app camera;
- shot with a standard camera;
- made in a third-party mobile or desktop application;
- downloaded from the web.
Having all this in mind, at the very start of the Filmm project, we did a serious preparatory work to get together an impressive collection of test videos and images. It included all known and widely used formats as well as some edge cases (for example videos with the negative start time or those not starting with a keyframe). Moreover, for this gallery we also generated videos of different durations from less than 1 second, made on a smartphone camera, to 65536 seconds (about 18 hours). Such testing footage helps to make sure the app operates in a correct way and would endure different loads. As of now the whole collection is regularly updated and complemented to be used by our QA-engineers within other projects.
Orientation Check Tool
Filmm is not only about video though. Oftentimes users also add photos and static images on their videos: to create slide-shows or as an intro to a clip.
Given that, we had to pay special attention to metadata. Broadly speaking, metadata is data that provides information about other data, but not the content of the data, such as the image or video itself. For example, such kind of metadata as EXIF allows information to be transported within a file, in a way that can be understood by other software and human users. In our case one of the things we should always be aware of is the image orientation data which might influence the way the image will be displayed.
Images with such metadata can be shot with a selfie camera with the smartphone flipped upside down. To make it easier to figure out how an initial image should look like and whether it is displayed correctly, we have created special test images which contain all necessary orientation data. In fact, the images have not been rotated, it is the letter F which is positioned differently on each image. So the image viewer should rotate or reflect them according to the parameters embedded. Actually, similar orientation checks are also applicable to videos.
You can download the examples of test images here and here
A Test Card
App testing doesn’t stop with the image orientation check. For the Filmm project we managed to customize another useful tool. Some might probably remember that in the past, late at night when all TV programs were over, a TV-set was showing some checkerboard and color bar pattern. It had several names (a startup/closedown pattern or a test card) and was used to enable television cameras and receivers to be adjusted to show the picture correctly. So we thought, why not try and tailor a test card to our own needs?
With this tool applied, we made testing of many tasks easier as we were able to:
- Check if the Cropper works as we thought it should.
The Filmm app allows cropping videos using the built-in aspect ratio presets: 9:16, 4:5, 1:1, 3:2, 16:9. A Test Card grid consisting of squares allows you to select the cropping area with certain accuracy and compare it with the final result. If we see that the result does not match with what we’ve expected, then there is something wrong and we should tackle down the issue. - Ensure that the horizontal and vertical mirroring functionality behaves as specified.
The numbers at the corners of the image help to facilitate the tests related to image transformations. We know exactly in which corner each digit should be after mirroring is applied. Moreover, if mirroring does not work properly, we will immediately see that the numbers are not symmetrical. - See if there are any geometric distortions.
The circles around the numbers also allow you to pay attention to the cases when proportions in the resultant video are distorted. It is visually easier to notice if a circle becomes an ellipse. - Notice the distortion of colors.
Since some video testing tools are similar to those we use to test static images we also use the Test Card as a part of the Calibration Video. It can serve as a testing footage to help us check that the effect we implement works properly, for example blends with a video as anticipated. This video tool can also be used within other testing procedures applied for images.
A Color Table
Now that the Filmm app is live, you can make your video have a highly professional look and feel, as it offers a whole set of adjustment tools: Curves, Brightness, Contrast, Saturation, Vibrance, Temperature, Tint, Hue, Shadows, Highlights, Exposure, etc. All this is pretty much about tone and color and in actual practice requires thorough checks of how color adjustment works on a variety of devices and with different operational systems. To make testing a bit more organized and smarter we came to use a customized Color Table.
It is actually a color swatch with commonly used hues which have letter identification across and numerical designation down. If we know what color is used as an input, we can easily check if it is the same at the output. Here’s a typical test scenario:
Set Temperature = -50
Description
Given A new image is added to the project
And Adjustments screen is open
And Temperature effect is selected
When A user sets Temperature intensity = -50
Then the color of cell 9L is (151,163,2) (#97A302)
It describes which RGB/HEX code should be there after assigning a certain value to a specific cell, for example 9L. We take the value specified in the scenario, for example “Temperature -50”, and export the result.
We select the color dropper and grab cell 9L (this tool is available throughout many image editors, or you can use a standalone utility for these tasks on a smartphone) and compare it with the RGB code of the expected result. If the color code matches the expected code that is specified in the scenario, then the setting being checked is working correctly. Such scenarios are prepared for each color correction tool.
Dynamic Overlays Testing Tool
The Filmm application also offers a good variety of dynamic overlays: professional filters with authentic film textures, light effects, and subtle distortions. Some of them might be videos laid over a user’s video, while others would be animations consisting of separate PNG frames.
Accordingly, the main task of any tester here would be to ensure the correct playback sequence of dynamic overlay frames, as well as their specified frequency. To make the life of our QA engineers easier, we made our own drawing of animation frames to help us check the test scenarios we wrote.
You can download the frames here for free
As you see, there are 20 frames. Each frame has a border of empty smaller squares. When a ready-made video with overlays is played, the squares are one by one filled with color as the frames change. Thus, playing an animation, a tester can notice with the naked eye that the frames are played in the right order. But there are digits in the center of each frame, why don’t you use them to check the sequence and order, you might ask? You are right, there are digits which also change, but the human eye can hardly see that they are going one by one in the right order — they are changing too fast. The digits serve a different purpose. With their help we can check how the mirroring feature is doing the job. If a user flips a video vertically, horizontally or any other way, the digits will also be reflected in a certain way (turned upside down, or mirrored, or else).
You can also notice a circle in the center of the animation frames. It allows us to visually check that the proportions of the animation frames are not distorted. Also in the Filmm app with certain aspect ratio chosen, some dynamic filters should have only the upper, lower or central part of the original dynamic filter visible, and this can be checked using such a set of frames too.
Bottomline
Of course we understand that in one little article it’s pretty difficult to tell about all the testing tools and libraries we use within our projects. It’s even more difficult to give a detailed overview of how everything works as the reader is usually devoid of context we have as a development team. But even that said, we believe it’s worth sharing our experience and materials which can come in handy for anyone working with similar projects. Over the years, we have accumulated, tried and tested a lot of useful things that we will keep on telling about. Our QA department is very fond of non-standard tasks and tools and we will be glad if you could also share your experience in the comments, or tell us about the tasks that you managed or failed to cope with while testing photo and video editors.
By the way, you can download most of the tools we’ve told you about here.
