The Myth Of Testing Tools In Games

I'm a fan of test automation. I keep up on all of the latest and greatest automated testing tips and techniques from people like The Braidy Tester. I try to tie as much automated testing as possible into the applications that I write on a regular basis. When it comes to application testing, you won't find many supporters as die-hard as I am. However, I do recognize that test automation has its limits. In games testing, automation testing is for the most part only useful for verification purposes.

What does that mean? It means that you can use automated testing tools in games to verify that content is formatted as described and to a lesser extent verify that the content is "well formed," but unless your regular testers find a repeatable type of content failure and are able to train a tool to identify that particular type of content failure, you won't be able to find what is wrong with your content.

You can use automated tools to automate game UI testing and level load testing, but very little can be done to automate gameplay testing for 99% of the games on the market. You can use automated QA to generate the massive amounts of combinations for combination testing, but you still need a human to evaluate the results in most cases.

You can automate harnesses against backend servers to ensure that the proper errors are thrown and that the proper data is passed back and forth, but you still need to be testing the game itself against the server caused by humans.

While most applications can gain a real benefit from test automation and can even reduce their test headcount needs via automation, video game testing is almost the last place where flesh and blood cannot be replaced effectively at this time.

Unfortunately, many people are under the impression that automation testing for games is a lot further along than it really is. Look at Dave Perry's take on it. (I've already called him a God-damned idiot, what else can I do?) Back in March, I dug in a bit deeper against his assertions.

Long story short, investing in testing tools in games will help release a better product, but it will not replace the need for an effective tester to wield the tool.