November 17, 2004

Classifications of Testers II: The Sequel

Here are a few more tester classifications that I thought I'd share.

STRESS TESTERS. These guys define abuse. If your program is expecting only one input per second, expect them to find a way to cram over a million inputs per second through your pipeline. If your program relies on a database server, these guys will find ways of stressing your database server that you would never believe. Frankly, the Steam guys could have used one or two of these guys. The great thing about a stress tester is that they'll be able to tell you at exactly what point your system fails. The downside is that they may not be able to tell you why.

PERFORMANCE TESTERS. These guys define grace. As testers, performance testers supplement the stress testers by helping you find the performance problems under normal usage. UI hangs for a second? It's a bug. Your frames-per-second drop below 30fps on the minimum system requirements? It's a bug. Occasionally, you have to make your performance testers upset in order to make your stress testers happy. Also, expect your developers to have to answer for any decrease in performance. If you lose even one frame or request a second, you'll have to account for it.

INTERNATIONALIZATION/GLOBALIZATION TESTERS. These guys help you sell overseas and keep you out of jail. Internationalization ensures that all of your UI screens and shortcuts lay out correctly, work with right-to-left scripts, handle IME inputs, etc. Globalization testers ensure that you aren't making any politically risky moves, like mishandling the border between Pakistan and India, or getting involved in this whole "Taiwan/Republic of China" deal, or popping up a "V" sign with your hand the wrong way. In some countries like India and China, making mistakes of this magnitude can not only get your product pulled from shelves, but if you have local representatives in those countries, it can get them arrested.

THE BUDDY. This is a fairly new type of tester, but an invaluable one. If you have a tester who is having problems finding bugs on his own, put him with a capable tester and have him watch the other tester test. The two of them together will find more bugs than they would seperately. On "Amped 2," we would all buddy-test for about an hour every week. The sheer amount of bugs that we would find during those periods was insane. We would have one tester "drive," and the other tester observe. By splitting the testing and observing duties, you allow one person to just be a user and you allow the other person to put 100% of their mental prowess into finding the bugs. It works.

THE CONFIG TESTER. These guys come in two different flavors: hardware and software. Generally, these guys help you track down OS-specific or hardware-specific issues. Want to see how your app works on Windows Me with Long File Names disabled? How about how your app works on an ASUS A7Pro motherboard? Need to reproduce a graphics glitch that only happens when you're running multimon with a Matrox DualHead card and an ATI Rage? This is your guy. However, for him to be effective, you need to give him a script that he can run on every configuration that exercises all of your core functionality. If you don't, he'll just be spinning his wheels.

THE NUMBERS MAN. This guy would be a nightmare to Las Vegas. He'll look at the bug counts for each area, place a bet as to where a bug will be, and be right 99% of the time. Only problem with him is that he's useless until someone else has done some testing first.

Anyone have more?

No comments: