Seattle Weekly Weakly Tests Testing

Sam Kalman gave a recent article from the Seattle Weekly on testing a once-over and found it lacking.

I'm in the fortunate position to have never been on the contract testing side of things, but I've had contractors working for me both at Microsoft and at Ritual.

While the article may be a partially accurate picture of how contract testing is done off-site (or on-site with Sony), contract testing on-site used to be markedly different. Note, I said "used to be."

Inside Microsoft's Redmond campuses, space is always at a premium. Offices that used to only hold a single person get doubled or tripled up nowadays...and that's for the FTE's. Cramped space doesn't make it any easier to get your work done.

In addition, Microsoft has been shifting their staffing allocations for testing. Back with "Halo 2" for Xbox, there were three test leads, three SDET's, three Bungie FTE testers and five Microsoft FTE testers for a total of fourteen testers. There were also eighteen contract testers on the game...almost but not quite a one-to-one ratio of FTE's to contract testers. Thirty-two credited testers...not bad.

Compare that to "Halo 2" for Vista. Two test leads, seven FTE testers and two credited contractors. They must not have thought it would be very hard to test...after all, it's a port. Of course, then there's the shared Tools & Technology group that is split between every single MGS release, but since they're a shared team, you really can't count them towards test.

So at this point, we have eleven credited testers, or about 33% of the number of testers that "Halo 2" Xbox had. What do you get for that? The poor performance is only the start of the issues with "Halo 2" for Windows Vista. Hell, they didn't even spell "Windows" right in the manual. (See page 31.)

There may have been more testers, but most likely they came on at the end and how I feel about that is well documented.

I don't know how I'm supposed to feel right now...I'm just afraid that things are going to get much much worse before they get better.

(Update: 8/3, fixed typos.)