November 29, 2006

Open Letter To QA

I've been watching the news today, and I've been seeing my comments regarding my brush with Sony's FPQA department taken with various levels of sensationalism and intrigue, so I figured I'd take a brief moment to address several of the comments I've been seeing and then drop this sordid matter before it goes much further.

To all developers: QA should be an integral part of any development process, not an afterthought. This doesn't just mean developer testing (unit tests, integration tests, etc.). This doesn't mean the certification process by the platform holder. This means real testing with real testers. Playtesting should not be how you find bugs. Shipping the product should not be how you find bugs. There are people out there who excel at finding these types of problems before they pound your review scores to dust...get them, keep them happy, and put them to work. Most importantly, listen to them. Testing without action is masturbation.

To Sony QA: I realize that your staffing structure is a direct result of cost cutting measures. However, several people in your in-house development houses joke about the bugs they receive. A big part of the reason they get laughable bugs is that when you're bringing that many people on for such a short period of time, the quality of training the testers receives suffers, as does the quality of bugs. Test leads do what they can to filter bad bugs from getting through, but there is a finite number of hours in the day and the longer the hours are that your testers work, the more items that are going to slip through their fingers.

If you want to adjust the perception, bite the bullet. Hire great testers, bring them on full-time, work them a reasonable number of hours, pay the benefits. It takes time to change a culture, but change has to start somewhere. A defeatist attitude like "the 5% rule" I was told about only proves the culture's point. (And if testing is going on from day 1, 5% should never happen.)

To the Press: A lot of people place blame on any bugs in a shipping product solely at the feet of quality assurance. Some people believe that bugs making it out are the result of QA sloppiness, or QA "not fighting hard enough" for the customer. To be honest, there are times when that is the case. However, knee-jerk accusations towards QA don't help anyone. In fact, it is reactions like that which have led many publishers to believe that since the highly-paid testers "missed this issue," they may as well employ "controller monkeys" instead. After all, they're cheaper, work longer hours, and are disposable.

And when you get an article like this, don't just take my word for it! While I stand by everything that I said, nowhere have I seen any attempts to contact Sony for a statement. Nowhere have I seen a response from Sony. The only response I've seen have been from former Sony QA members who said, "Yep, sounds right." Please try to present a balanced viewpoint.

To my regular readers: Sorry for the distraction. I didn't think sharing my experiences would lead to such a hubbub.

1 comment:

Russ said...

You know, ROM, I'm impressed that you've had the stamina and fortitude to continue to try and do what is right for QA. I am sad to see you taking hits for being honest and telling the truth about an industry that is important but is treated like garbage. When I came to that realization, I found that I had no desire to fight it and walked away. I really do wish you the best of luck and hope that you do make a difference in the industry. Talk is cheap, I know. But, I thought I'd let you know that I do appreciate your efforts and am rooting for you.