[Testing] Purpose of Third-Party QA

I had a very interesting E-mail last week, and I thought I'd share an edited response on this blog. The paraphrased question was, "What is the purpose of third-party QA? Don't the publishers have QA departments of their own?"

It's actually a very valid question, and the answer is fairly basic, but is going to require a fairly hefty explanation. The primary purpose of a third-party developer's QA department is to save the third-party developer money over time. In-house QA saves money for third-party developers in two primary ways: it reduces publisher chargebacks and can reduce development time.

You may remember when I was going on about how much money it took for a game to break even. A big part of why it takes $40 million in retail sales before a third-party developer sees dime one above their royalty advance is that publishers tend to charge back every possible expense towards the developer. Two big expenses that get shuffled back are QA and support, and in-house QA can help reduce both of those expenses.

With in-house QA, your testers can find significantly lower-level bugs than publisher QA earlier. Testers can help ensure the stability and completeness of milestone builds prior to their submission to the publisher. In addition, if your QA department has a good relationship with their publisher counterpart, they can even help ease in some of those "borderline" milestones...especially since it's the decision of the publisher test lead usually as to whether or not your milestone is going to be accepted. Both of these result in lower amounts of downtime for publisher QA, as well as reduced bug counts from publisher QA...which tends to result in a decrease in QA costs being charged back to the developer. Also, fewer bugs in the final product tend to result in a decrease in support costs and an increase in good will from the community.

Of course, the ability of testers to find these lower-level bugs directly corrolates to the ability levels of the testers you have, their knowledge of the game assets and codebase, and the degree of communication between your test department and the rest of the product team. If your team has a "throw it over the wall" mentality, the benefit you will reap from in-house QA will be greatly diminished. There is no reason to keep in-house QA in the dark about any part of your project.

The second way that in-house QA saves you money is by potentially reducing development time. Some major publishers do not put any appreciable amount of QA staff on a project until eight weeks prior to ship (*cough*Sony*cough*), and even then, they're generally low-paid contractors. However, under the infinite monkey theorem, these are invariably going to lead to a large amount of bugs coming in during the time when change management is the order of the day. By having your internal QA department finding bugs from day 1, as long as you keep fixing the bugs as they are found rather than leaving them to the end of a milestone or the end of the project, you can reduce the number of man-hours spent during the end of a project doing major refactoring and just focus on the simpler bugs that the infinite number of monkeys hurl your way.

Again, this assumes that you are in fact fixing the bugs as they are found. A good way to enforce this is via bug count metrics being included inside internal milestone requirements. (No more than X total bugs open, no more than Y sev1/2 bugs, etc.)

If you have a good in-house QA department and a good in-house QA process, the expense you incur via QA will be offset by the faster turnaround in milestones (and their related payments), the reduced amount of chargeback towards your royalties, and improved name recognition as a company that releases stable, fun products.