Automating Games QA (part 3)

In this installment, we're going to be talking about combination testing.

Most games nowadays have some sort of customization system, be it your character, your "crib," your vehicle, etc. Testing the entire gamut of combinations by hand can actually get to the point where it is impossible to test within the time available for testing.

For example, let's say that you have a standard human avatar with a customizable shirt and pants. There are 10 different shirts available and 10 different pants available. That is 100 combinations right there. Let's add 10 hairstyles. That bumps it up to 1,000 combinations. Add 10 different fleshtones...10,000 combinations. Add a second gender...20,000 combinations. Add 5 different faces per gender...100,000 combinations. It adds up quickly.

Combination testing is designed to hit the two simplest types of bugs: single-value and two-value settings.

Now, if you look at the example above, while there are 100,000 combinations, there only a few individual settings: 10 shirts, 10 pants, 10 hairstyles and 5 faces per gender. That can mean either 40 settings if you assume everything else equal between the genders beside the faces, or 70 settings if everything is seperate between the genders. An automation script to individually cycle each of these single-value settings can quickly help eliminate bad items, and if the script screencaps each item, manual verification of item appearance will go fairly quickly.

One last thing: while handling the single-item tests, check the amount of memory that each item uses. A good additional test is to set all of your settings to their most memory-intensive setting and play the game that way to check for borderline out-of-memory conditions.

The second most-common type of bug is when two values are interdependent on each other. For example, you may have a hairstyle that clips through some geometry on a certain shirt in this example. Now even with automation, you're still going to have to manually verify the screenshots, so you want to minimize the number of shots you are looking at. This is where all-pairs testing comes in. This type of testing is also called pairwise testing, and there is a very in-depth example here.

This gets very easy if your combination lists are data-driven. Feed your lists into a tool like ALLPAIRS from James Bach and pass those lists into the game for testing. Pass the created list into your framework and have at it.

Automation testing is all about getting the grunt work done by the machine so you can focus on the non-automatable tasks. Always be on the lookout for tasks like this that you can automate.

The next games QA automation column will be on automating content testing. It's probably the hardest type of automation, and it will not be a good fit for most studios, which is why I left it for the end. Stay tuned.