Test Plan #2: Milestone Acceptance Criteria

Okay, so it's been 11 days since my last installment...I've been busy.

After my smoke tests, I usually include common criteria that will be appended to each milestone acceptance test. Since these usually include overarching goals, it's best to make sure that they're established and signed off on up front.

In addition, "Acceptance Criteria" may be a bit of a misnomer. You specifically want to list anything that will cause you to fail a milestone, because that's what the team will care about most.

For example, this is a section from my "Milestone Acceptance" section on [deleted]:

A milestone shall be considered to have failed if any of the following are true:

1. A reproducible crash is found during the first sixty (60) minutes of testing the milestone.
2. Three (3) reproducible crashes are found during the first eight (8) hours of testing the milestone.
3. One (1) major milestone deliverable is found to be not implemented or non-functional during the testing of the milestone.
4. Three (3) minor milestone deliverables are found to be not implemented or non-functional during the testing of the milestone.
5. At least one (1) level transition does not work.
6. At least one (1) level cannot be completed without using the “noclip” cheat.
7. At least one (1) level does not load.
8. At least one (1) weapon does not work.
9. At least one (1) multiplayer mode does not work.
10. A multiplayer game cannot be hosted or joined.
11. If there are more than ten (10) active severity 1 bugs in the [deleted] bug database.

A milestone shall not be considered passed until the milestone has passed a milestone acceptance test.

The above may seem very harsh, but it's also very common in video gaming...especially from a publisher standpoint. More and more, publishers are including stability bars in their milestone acceptance criteria. So let's walk through the list really quickly.

Numbers 1 and 2 are there simply to act as a deterrent against submitting unstable builds. If a build can't last an hour without crashing, why am I looking at it as a progress milestone? The three-crash bar acts as an additional stabilizer.

Numbers 3 and 4 are pretty basic as well. Milestones are supposed to have a certain set of deliverables. Those deliverables may be shuffled between milestones, but what is delivered in a milestone is expected to 1)be there; and 2)work. While there may occasionally be dickering regarding what is a major or minor feature, that's a small price to pay for making sure that each milestone has what it says it will have.

Numbers 5 through 10 are very specific to the title. In [deleted], the goal is to have the entire game capable of being played through very early, so as part of my acceptance test, I'd go through and play the entire game from beginning to end.

Finally, we have number 11. This one is the one that people usually get the most grief over. Every project has a bug database. It may be just E-mail going back and forth, it may be an Excel spreadsheet, it may be a modified version of IssueTracker, it may be a full-featured bug client like Seapine TestTrack Pro, but you always track your bugs. The goal of this is to make sure that major bugs don't carry over from milestone to milestone.

Bugs are always rated by the test team on their severity. Severity levels are fairly well defined, and severity 1 is the worst. For a bug to be severity 1, it must be either a crash bug, cause data loss, cause the title to fail certification on a console platform, or be a bug that would definitely cause the product not to ship.

Every product has these sev 1 bugs. The more you have active, the less stable your product really is. The goal is to keep no more than ten of them on our plate from milestone to milestone.

The biggest issue you are going to see if you implement a severity bar in your MAT is a mass-resolve of bugs just prior to a milestone. Easiest solution: tell them that reactivations of sev-1 bugs count towards the 10. It may seem cheap, but so is trying to get around the submission guidelines.

Milestones are there for one reason and one reason only: to act as a constant heartbeat for a project. Publishers pay when they feel the pulse. If your product doesn't have a decent pulse, then it's crisis time.