Dave Perry Blogs About Testing

Dave Perry elaborated a bit in a recent blog entry about the "dark future for testers." I figured I'd address his rant point by point.

(1) To say that paid testers are better than everyone in the community would be dumb. (Considering we hire most of them from the community.)

Nobody is saying the community is dumb, and yes, most testers do get hired from the community, but then again, we're cherry picking from the community. Those with the talent rise to the top, get hired, and progress.

(2) There's no degree course in testing that I'm aware of? Meaning a professional tester is no more qualified (on paper) than a community tester? (Other than they get paid.)

There are computer science degree programs with focuses towards quality assurance. The best one in the country that I'm aware of is Dr. James A. Whittaker's program at the Florida Institute of Technology. That said, a tester with a college degree can make over $60,000 a year in the Dallas area working on productivity software, while that same tester will be lucky to make $30,000 a year around here working on games.

(3) Never just assume you are the best at anything.

Testers don't assume. We hypothesize, experiment and report our findings.

(4) 150,000 people testing, kills 20 people testing, I don't care how you slice it. Interestingly testing needs really bad players too, yet I've never seen a test team hire some? That's the guy that jumps at a locked door 100 times because he has no idea what he's doing, and uncovers some great bugs testers don't find.

Here we're going to start getting into the numbers debates.

I get 150,000 people on an open beta. Of those, I know that only 1% on average are going to respond, so that leaves me with 1,500 responsive players. Previous data on usage for people who use an open beta shows that 95% don't spend more than 1 hour testing...period. That's not one hour per build, that's one hour total. That drops us down to 150 people who are going to be looking beyond the first hour of gameplay.

Now, out of this 150, how many are going to actually be decent testers? Most people who join open betas join to play, not to test. Play testing is a subset of testing, not a replacement for testing.

(5) Self-testing will also be a major future for games. Where the game plays itself, trying everything everywhere (to the extent feasible by the endless combinations generated) again every point on every surface will get tested for penetrations etc. A job testers can’t do accurately and it would be mind numbing to do this all day every day for months.

Self-testing is the holy grail for testing. Some companies, like Microsoft, hire testing engineers to create testing tools and automation harnesses for their games so that they can try to automate as much testing as possible, but they are approaching it from a testing point of view. Besides, your programmers are already crunching to just get their code done...are you going to have them work without sleep to get the test tools done as well?

Full content automation is still many years away. I should know, I've written content automation frameworks and used them on projects. The frameworks take time to write, the testing tools take time to write, verifying the tools work correctly takes time, etc. That was all time that I spent on my own, and they paid off, but only on products that were using stable, licensed engines. The cost/benefit just isn't there for engines that aren't stable.

But even if you can self-test, you're still missing the one big thing: how can you detect that your test failed? Failure detection is the hardest part of automated testing, especially for content automation testing, because you have to try to capture a thought process. Not all failures can be detected by code.

(6) Community testers contain hackers (rare on professional teams), these guys try every tool, every exploit on your engine and look for vulnerabilities. Again you don’t just get the one that might be on a professional team, you get dozens of them.

Most game companies that work on single-player games don't hire hackers. Most game companies that work on multi-player games have software developers in test that do white-box analysis of the code to check for issues, as well as write test tools to abuse the network connection. I'm not aware of a single MMO company out there at the moment (besides yours, it seems) that doesn't employ people who do abuse their MMO's.

The catch is that if an internal person hacks the MMO, the hack gets reported and fixed. The MMO is paying this internal person's wages, so he has a financial imperitave to report this issue. If an external person hacks your MMO, given how financially lucrative real-money trade has become, what is his imperitave to report the issue?

Dave, I understand that you're trying to cut down the cost of making your games. That said, you're forgetting the one critical role that QA plays, and the real reason your plan won't work. Developers are loyal to the code, producers are loyal to the company, but testers are loyal to the user. Their job is to try to keep the game from getting out broken...to protect the user. Most "showstopping bugs" that ship out were most likely found by your internal test teams and reported, but waived in order to make a ship date or because nobody outside of QA agreed about the severity of the bug. Out of all the titles I've shipped, the number of showstoppers I've missed is in the low teens, while the number of showstoppers I've caught but were waived number in the mid-hundreds.

If your development team won't listen to testers that you hire, pay and see every day, what makes you think that they'll listen to volunteer testers that they never see?

If you want to cut costs, approach it like the movie industry. Do more pre-production. Don't hesitate to cancel projects early on. Look for ways to reuse code and content. Create a common set of engines and tools so that we can properly harness them and automate them.

Now that said, I am going to agree with Dave on one point...the days when a tester could have little to no technical know-how are nearing their end. Testers nowadays need to know how to write automation scripts, read code, analyze stack dumps, trace through and find a flaw in a debugger, etc. If you can't do that, you won't be of much use to most modern QA departments.

But beyond that point, Dave is merely showing his ignorance about current industry practices with QA, and it's a shame that industry QA doesn't have a more public face to counter this schmuck. Until industry QA does get a "celebrity," the discipline is going to continue to lose support and have misinformation like Dave's missive spread like wildfire, because unfortunately CEO's do read him.