Test Plans #1: The Smoke Test

The first item I have in my test plans here at Ritual is the smoke test.

A smoke test is a high-level set of test cases that can be run in about 15-30 minutes that will show you whether or not the build is stable enough to be tested.

A smoke test can have three possible results.

• A pass means that the build passed the smoke test without any major issues popping up.
  
• A pass with notes means that the build passed the majority of the smoke test, but issues popped up that wouldn't severely affect testing. For example, if during your test you find a bug that only manifests itself when you die and restart the level, that would warrant a note, but the build is still testable.
  
• A fail means that the build is so unstable or major functionality is so completely blocked that testing the build would be futile. A good example would be if you were testing a first person shooter, and your weapons wouldn't fire.

It is important to document your smoke test for the team. It does two things. First off, it lets the team know what your minimum expectations of each build are. Second, your more ambitious colleagues can run the smoke test prior to doing any sort of check-in as a verification step.

Here is a modified version of a smoke test that I currently run (with NDA stuff removed):

1. Tester shall launch the game.
2. Tester shall drop down the console and go to [removed].
3. Tester shall use the “[removed]” cheat to give all weapons, and change the current weapon to the [removed].
4. Tester shall fire one [removed].
5. Tester shall save the game.
6. Tester shall exit.
7. Tester shall launch the game again.
8. Tester shall load the saved game, and verify that the ammo count for the [removed] is the same.
9. Tester shall fire the [removed] and [removed] to ensure that those weapons work.
10. Tester shall then go to [removed].
11. Tester shall verify that the [removed] can move, collide with [removed], and can fire.
12. Tester shall then go to [removed].
13. Tester shall again use the “[removed]” cheat.
14. Tester shall select the [removed].
15. Tester shall fire the [removed] at a wall and verify that [removed].
16. Tester shall fire the [removed] at a [removed] and verify [removed].
17. Tester shall finally attempt to load all remaining maps.

Notice that it's very straightforward. It's not an in-depth test by any stretch of the imagination, but this test can be completed in 20 minutes and in those 20 minutes, I'm exercising the entire game. If any step doesn't work, I determine if that warrants failing the build or just a note on the pass.

Now your smoke tests will not be static...they will change. It is important to update the test at least once per milestone to exercise anything that was delivered in previous milestones, or to remove tests for features that were removed, etc.

In our next installment, we'll go over the risks section.