Test's Most Powerful Weapon

Lots of people think that they know the perfect silver bullet for testing. Some believe that automation is the savior that will reduce testing time to nothing and reduce or eliminate human error. Others believe that in-depth test plans and suites will ensure that no stone is left unturned. Still others believe that only white-box testing will allow the product to advance beyond the Stone Age.

Well, all of those are useful tools, but all are forgetting the most powerful weapon to a test department: a live tester.

True, your automation tests can glide through your product and help ensure nothing has broken, test suites ensure that nothing has been missed in the product and white-box testing helps uncover bugs that would hide for years. But the bugs that you find with these tools cannot compare to the bugs you will find with one intelligent tester just sitting back for 5 minutes and saying, "I wonder what would happen if..."

If you don't have at least a small period of ad-hoc or directed ad-hoc testing on your product per day, you're missing all sorts of wonderful bugs. It could be something as simple as a timing based bug. It could be something as major as an issue requiring a complete architectural overhaul.

In video games, ad-hoc is especially important, because we need our testers to be as abusive as regular gamers are going to be. We need to think, "I wonder what would happen if I abused the weapon physics system to get around massive chunks of the level." We need to think, "I wonder what would happen if I shot that grenade the guy tossed at me in mid-air." We need to think, "I wonder what would happen if I drop this unbreakable crate into this tram that has to move to the end of this railway, and orient it in such a way that when the tram moves forward, the crate collides with the ceiling."

In "Halo 2," you could toss a grenade on the ground, bounce yourself high into the air, and land on top of the buildings so you could run past 30% of the level. The other two are things that came from our ad-hoc testing on "SiN: Episodes."

Fact of the matter is that tools are merely that...tools. And the end result of using a tool is only as good as the craftsman using the tool.

Let your testers ad-hoc. It refines their skills so when they do more automation testing, they write deeper tests. It hones their instincts so when they run their test suites, they instinctively test into the holes in those suites. It sharpens their intellect so when they do white-box testing, they do more mental checks for invalid inputs while walking through the source.