April 18, 2006

[Testing] The Hidden Costs Of Testing At The End

There's been a disturbing trend in the gaming industry lately. More and more publishers are scaling back their test organizations to only having a small quantity of test leads, and then hiring on tons of contingent testers in the final eight weeks of a project. This is a bad idea.

On paper, it seems to make sense. You can get a contingent tester for about $320/week on average, while an experienced tester with three years experience is going to run you $600/week, and your test lead is going to run you just over $900 a week. (For the sake of this argument, we'll measure coverage by contingent manweeks, and consider a test lead and a regular tester as two contingents for purposes of measuring productivity. These figures also assume that nobody will ever be working overtime, which as we all know, is completely unrealistic, but the numbers would get a lot scarier if we factored in the OT.)

The way that some companies work is that they have a test lead on the project from the beginning, add three regular testers to the project about six months from ship, and then bring on about six contingents during the last four months. On a one-year project, that works out to about $115,000 for 324 contingent manweeks of testing. Under the "new" approach, a test lead is assigned six months prior to ship, and about 34 contingents are brought on eight weeks before ship. This amounts to about $111,000 for the same number of contingent manweeks, or a savings of about $4,000 on one project.

The "savings" scale the longer the project is going on. Assuming a two year project, given the above rules, the contingent spending wins $144,000 to $163,000 if you bring in 47 contingents to match the manweeks.

So on paper, these sorts of decisions make sense from an accounting standpoint, until you factor in two items: facilities costs and software development.

First off, it's significantly cheaper to come up with space and equipment for ten people than it is for 35. Space, power, computers, consoles, TV's, monitors, snacks, etc., it all adds up.

Second, let's say that after 324 manweeks on a project, your test team has found a total of 5,000 bugs. Under the initial system, those bugs would have been found and fixed throughout the development cycle. Under the "cheaper" system, all of those bugs would hit the development team at the very end.

Massive amounts of showstopper bugs at the end of a project lead to slips, and that's where the savings erode. For every week slip under the initial system, testing costs increase by $4,600 just for manpower. Under the contingent-heavy system, you're paying them all for an extra week, so your 34 contingents and your test lead just added an extra $11,800 to your project's cost. Larger teams, like the 47 CSG team mentioned earlier, add an extra $16,000 a week.

On a one year project, a one-month delay eats away all of your savings. On a two-year project, six weeks eats away your savings. Of course, some companies decide to ship and patch rather than slip, but that costs reputation and support costs as well.

So by scaling back your test organizations, what are you really saving?

(Edit: Added tag, fixed typo.)

9 comments:

Ranter said...

Your numbers are impressive etc. blah blah blah.
But, you're forgetting probably the biggest factor, NOBODY CARES ABOUT TESTING BUT TESTING...

Zman said...

$320 per week - thats $8 an hour where are you outsourcing to?

Michael Russell said...

You can get CSG's cheap through almost any temporary agency, but the cheaper you go, the lower they get paid ($8 is the cheapest I've found, we paid $15 back at MS with a $10k finders fee per).

So you really get what you pay for with most CSG staffing places.

Zman said...

Wow, I am more out of touch than I thought. I had no idea testers were that cheap - are they really any good at that price? I used to think $45 (agency price) was cheap for a developer and at that price they usually could barely turn the PC on.

Michael Russell said...

Not really...hence the reason why I count an experienced tester or test lead as at least two contingents.

It's rather unfortunate that most of the places that are going the primarily-CSG route seem to be hiring the cheapest CSG's they can find.

I had one person tell me of some CSG's that were brought in from California who spent over a week entering bugs against their Xbox 360 title...based on the PS2 TRC checklist.

Yet another hidden cost of the CSG route...

Duri said...

Yes. I've spent 14 years in the QA industry, almost half in games. The other half included building outsource labs, large client/server labs, teaching methodology, consulting, etc. I've written articles on the subject and currently work in the financial industry as a consultant. I started in games.

There are two main problems here;
1) Senior management in the game industry is used to being able to get kids to test games for min-wage and pizza, because everyone wants to play games for a living. Since this is what they're used to, they don't understand why you'd pay more or what you'd get for it.

2) Almost everyone in games QA starts out in games. This means there is very little knowledge transfer from the other areas of software QA to games. Considering sheer numbers of man-years of experience, the non-games world is going to be way ahead. But people who leave games rarely come back, and people who go into games from the non-games QA industry are usually not the most skilled. Would you go from $65,000 to $18,000 a year to test games?

I've built four games test labs. In all of them I used well paid people, and I trained them in modern skills. They all excelled in cost per project. In other words; they found more bugs faster, cheaper, than the teams of 35 random kids.

Unfortunately, they all failed. They failed because the game industry managers would look at the higher per-man-hour rates and run, without looking at the total number of man-hours vs. cost. It was cheaper to hire fewer, more skilled and motivated people than it was to hire a truckload of goobers.

Unfortunately, the rule #2 is no less true of game industry managers than it is of game industry testers.

DonWan said...

"NOBODY CARES ABOUT TESTING BUT TESTING..."

What an ignorant statement. Good luck with your projects buddy. I can only hope this poster was just trying to get someone's goat and doesn't carry this creedo for something really important like testing NASA materials or hospital equipment. You must be from financing. I mean that's like saying "My company doesn't care about quality, our customers or even the longevity of our business for that matter."

Look, the point is the Industry can actually save money by trying to develop quality into their product - instead of trying to test quality into their product. That involves testing and feedback involvement during the whole project cycle *IF* you put the time, effort and money necessary into employing a quality test staff who knows what they're talking about and integrating theminto the dev process. Yes quality costs money, but needs to be seen as a worthwhile expense like marketing, HR or even team building exercises. Guess what, you could get rid of testing all together if your project DIDN'T HAVE SO MANY BUGS - which would otherwise cost you money in lost sales, bad company rep, customer services issues, etc. So stop kidding yourselves into thinking QA is only a loss leader.

As for cost cutting, some of these comments I feel miss the mark completely. Going back to building quality into your product instead of trying to test quality into the product, in many cases, you're seeing grandfathered (aka "broken") code infrastructures, inefficient (dare I say wasteful) production/development expenses and a lack of investment into automation and data gathering/understanding/tracking. Instead of outsourcing or trying to find local grads to test for $10/hr., break out of that tired model and hire about half of those as "QA-centric Engineers" at $20/hr, get them inserted into the Dev cycle for the express purpose of assessing risks to quality, and then have them help test the code base AND the product as it fleshes out. Then at the end of the cycle, you hire temps to focus test your product from a clean end user perspective for that beta layer of feedback.

In my opinion, not many people in the game Industry especially are doing this correctly and that's going to have to stop if they want to remain competitive in this noisy, over-saturated market of increasing costs, legal woes, ESRB demands, confusing format models, etc...

Narissa said...

Somehow I stumbled upon your site as I was doing a job search - your comments about interviewing at a certain company are spot on. As one of the "contingent monkeys" or "kids" referenced, I can attest to the validity of your analysis of cost vs quality. Several additional factors not mentioned - with contract workers, you have a longer chain of command, which leads to greater inefficiency, less satisfied employed (which leads to lower productivity), lower accountability (with 1-2 leads and a shiny new game, how many of the kids do you think are just running around shooting stuff for fun? Answer, a lot. People slide under the radar). The low pay, no benefits, and atmosphere described also ensure that the job has high turnaround, as competent and/or ambitious people want to advance, and the low hire rate for contract staff only furthers this.

Narissa said...

Got cut off, but consider my several year's delayed response a nod in agreement. Then again, those of us who want to work in the industry take what we can get to build the resume. :)