Quality Assurance in Videogames
INTRODUCTION
I am a QA Engineer also interested in games. So I investigated how games do QA.
I discussed the topic with 15 senior game devs from a total of 11 companies. Small sample size. The following article is based on their input and my personal past experience in games
ENTERPRISE
In the world of enterprise software we have a number of techniques for quality — automated tests that are distributed on unit, integration and e2e level in the testing pyramid, regular code reviews, linting and code coverage tools, continuous integration and delivery, git workflow, etc
INDIES
Having more engineers results in rapid escalation in code complexity to the point where everyone knows just a small piece of the tech context. The techniques above aim to manage that complexity.
When you narrow your scope the problem narrows as well. A small project of 1–2 senior devs can cope perfectly fine without any of the techniques above. This article focuses on companies of more than 20 people, not small teams.
COLLECTED DATA
From the 11 researched companies
- Only 1 company is doing any form of automated testing, but admitted it is complementary and not the core part of the testing effort
- Only 1 company is doing QA during the whole lifecycle of the project and not at the end in waterfall way
- Most (NOT all) companies do code reviews, but all people shared that is occasional and not regular practice
- The 2 biggest companies on the list rely on brute force of numerous manual testers on minimal wage (NOT engineers)
- No-one mentioned linting, code coverage tools, CI and CD
- Git is not used and is frowned upon
- Most people admitted they do not see value in the techniques mentioned above
TECH CONTEXT
People justified not doing automated testing because games
- change too much
- are too visual
- are too complex
All of these are true and they make test maintenance expensive. But not a single one of these arguments is unique to the games industry and all apply to enterprise software with the same strength or more
MANAGEMENT
I could not discuss the QA topic with a game CEO / CTO, so I will fall back to my personal experience in games. I worked for a company of 150 people and reached a management position that was high enough to warrant my presence on important management meetings which gave me a glimpse of the attitude of the CEO
- He was very resistant to spending resources on QA, hired primarily junior manual testers
- CEO called the testers “clickers”
- When deadlines approached, CEO insisted managers “step on the necks of developers”
This leads us to the overtime problem — game companies are notorious with its insane overtime. My personal experience shows that people do their poorest cognitive work when pressured, stressed, angry or just tired.
This seems to be recognized problem, as some people shared their company does not do overtime.
ATTITUDE
Example 1)
In games the code processes a lot of assets — audio, models, levels. Bugs originate not only from the code, but from poorly made levels. Example — characters can fall out of the level, because of gaps in the geometry, or their path could be blocked by something invisible. Example — sometimes level designers leave unused assets buried in non-visible parts of the level, these redundant assets negatively impact performance.
I discussed this issue with people. The general attitude is that assets should look good but their technical implementation is largely irrelevant. Some people responded that this is just “not their job”
Example 2)
I asked a senior engineer with 10 years of experience why git is not used in games. I expected engineering arguments. He told me that the guru John Carmack struggles with it and that game artists are not competent enough to learn it
Example 3)
I asked a colleague on a lead position how developers manage to deliver without any QA. His said that game developers are just better engineers than enterprise engineers.
Example 4)
While most people did not see particular value in the techniques mentioned above, some even went to the length to call such approaches “bullshits”
Example 5)
A senior professional on a lead position with 10+ years of industry experience shared that he would not do much work unless put under pressure by his managers
CONCLUSION
My goal is NOT to offend anyone. I consider these guys friends and we share a passionate love for video games. But I also strongly disagree with such attitude. I have the impression that there is a lot of ego and tribalism in the culture of making games
TARGET AUDIENCE
When we have bugs in enterprise software, we lose customers and risk lawsuits. But games are not critical applications. Gamers are forgiving of bugs, they will rant on forums and use hostile language, but buy into the product, including preorders. Some gamers are tribalized and will defend their favourite franchise when its next instalment is broken. So some game companies do minimal testing and use clients as testers.
UPDATE
I was recommended an article that is relevant to the topic — https://grumpygamer.com/unit_testing_games
