[Testing] Automating Games QA (Part 1)

In most software houses, automation has made major inroads over the last several years. Vista has over 700,000 lines of automation code that is run against every build. Companies like WorkSoft try to make creation of automated test cases easier for average test departments. Even unit tests run by developers are, at least in some way, considered automated tests.

However, automation testing has had a difficult time infiltrating game development houses for a couple of fairly hefty reasons. The first reason is that the number of bugs contained in the code are generally dwarfed by the number of bugs contained in the content. The second reason is that it is difficult to impossible to automate most games because most games have an element of randomness to them, and as such, it makes it difficult to determine success or failure on a test case in an automated fashion. Finally, there is rarely, if ever, a standardized way of querying a game about the state that it is in, or even passing input to a game to trigger a response.

This isn't to say that automation is impossible in a game scenario, but automation does require an additional level of developer interaction and even imagination that other scenarios simply do not have. There are generally four areas where automation testing can be efficiently used in game development: User Interface, Game Flow, Combination and Content.

User Interface automation testing is where you are going to see the biggest initial gain from a QA standpoint, and is an excellent place to push for an automation start in any company. The goal should be that anyone in your QA department should be able to write automation test cases without much training. I'm going to describe a simple framework that you can share with your development team as a starting point.

A game UI automation framework generally consists of four seperate components. The game hookup, the communication component, the use case library and the test cases themselves.

The "game hookup" is a piece of code inside the game itself that listens for commands and queries from the communication component. For example, on the Xbox, your game hookup may just be a background thread that sits and listens on the debug channel. On a PC game, it may listen on a named pipe or an IP address or some other similar item.

The "communication component" is generally going to be a COM component that sits on your PC, and is responsible for brokering communication between the use case library and the game hookup.

The "use case library" is a set of user actions in user-oriented-named subs written in VBScript. Each use case sends the appropriate commands to the communication component to execute a certain action, and requests information from the communication component to verify that an action executed correctly. This is usually jointly maintained by the developers and more technically oriented testers.

The "test cases" are the actual test cases that call the subs in the use case library to handle each individual test case. Let's hook all of these up and see how these would work and evolve over time.

You are working on an Xbox title. As a test to verify that UI automation is useful, the development team agrees to create the game hookup and communication component. The game hookup will only recognize a limited command set, "getscreen," "getcontrol," "getvalue," "nextcontrol," "prevcontrol," "cancel," and "activate." "Getscreen" returns the name of the currently active screen. "Getcontrol" returns the name of the currently active control. "Getvalue" returns the value of the current control, but can later be extended to let you query more data. "Nextcontrol" and "Prevcontrol" are simply tab-order style items, and act by sending the "up" or "down" input for the main controller. "Cancel" acts as the B button. "Activate" acts as the A button or START button. The communication component takes one argument when it is created: the name of the Xbox on the network that you want to control via automation. It has a sub for passing a command, a function for passing a query, a sub for sleeping for one tenth of a second, and a sub for restarting the console as a cold boot and launching into your game.

You decide that all of your test cases are going to be designed to start from power-up. As a result, your first use case will be get to main menu. The flow at this point in development is no videos, just the "press START" screen and then the main menu. You write your use case similar to this pseudocode...

Sub GetToMainMenu(Xbox1 As TestControls.XboxControl)
  Xbox1.Reboot
  For X = 1 to MaxTimeOut
    If Xbox1.Getscreen = "press_start" Then
      LogSuccess "At Start Screen"
      Xbox1.Activate
      For Y = 1 to MaxTimeOut
        If Xbox1.Getscreen = "main_menu" Then
          LogSuccess "At Main Menu"
        End If
      Next Y
      LogFailure "Could Not Get To Main Menu"
      Exit Sub
    Else
      Xbox1.Sleep
    End If
  Next X
  LogFailure "Could Not Get To Press START Screen"
End Sub

A sample test case that called this would be:

Sub TestSystemLinkMatchmaking
  Dim Xbox1 As New TestControls.XboxControl("TestXbox1")
  Dim Xbox2 As New TestControls.XboxControl("TestXbox2")
  GetToMainMenu Xbox1
  GetToMainMenu Xbox2
  ...

As part of your nightly build process, you then hook up a couple of these scripts to be executed after the build is done and deployed to some Xbox test kits. When you come in the next morning, check the logs.

So, let's say that TestSystemLinkMatchmaking reports a failure. It's easy to open the test case and manually try the steps inside to verify the failure. If it does fail, you can even tell the developer to just run the automated test case to trigger the failure. It saves the developer time to repro and debug the problem.

If your use cases are prolific enough, you can even write individually coded cases for regression testing and repros of manually found bugs.

It requires some effort to create a system like this, and changes to the UI require adjustments to the use case library. Major changes may even require changes to the test cases themselves. However, the time savings in comparison can be fairly hefty.

Obviously, this is an extremely simplified example, but hopefully it got you thinking.

In the next installment, we'll go over game flow testing.