November 27, 2006

Quality Assurance at Sony

READ ME FIRST: Given that so many news outlets have been taking portions of this out of context, I need to say this.

1) I have nothing against Sony's QA department, contrary to what some reporters have said. I was commenting on the impression I got of how QA was perceived within Sony, not QA in Sony.

2) I talk about the impressions that I got seven months ago. Things may have changed, I don't know.

3) The department in question is the "last line of defense" inside Sony. From what I have been told, individual internal developers may have their own QA staffs on top of these.

4) These were my impressions, and are not necessarily the opinions of my past, present or future employers.

Sam Kalman made a post on November 22nd about a bug in Genji found by Chris Kohler, and it begs for the following story to be told.

Back in April, I was interviewed for a FPQA Manager position at Sony Computer Entertainment America's San Diego office. Sony was extremely nice. They flew me down and back first-class, took me out to lunch, etc.

Everyone I met there was a consumate professional, but there was a lot of underlying tension. I signed an NDA so I can't go into specifics, but there was talk about issues that only came up on production UMD's for PSP games, major friction between test and development teams with little to no management backing for test, little to no shared technology, extremely lax "user effect" bug metrics for determining whether or not to fix something, and a variety of other fairly hefty issues, not just from a process standpoint, but a overall culture standpoint. Microsoft is known for giving QA a bit too much say in the products that are developed, but the feeling I got inside Sony was that QA was seen as nothing but a bunch of monkeys with controllers.

The straw that broke the camel's back came in the last hour of my interview. I was told that the way that Sony tests their games is that there are one or two test leads on a project starting at about six months out. At T-8 weeks, between 80 and 100 temporary testers are brought on to test the game for those eight weeks. That's it. This was done for financial reasons, and as a QA Manager, I would be expected to run test the same way. Obviously, I didn't feel that was a valid way of handling QA.

The following morning, I sent an E-mail to Sony removing myself from consideration for the position because I didn't feel that I could run test the way that they wanted me to.

At Microsoft, the stringent QA processes often strangle creativity. At Sony, the lax QA process allows creativity to squash quality. It's hard to walk a middle ground where QA and creativity work hand in hand, but it is a tightrope that this industry is going to have to learn to walk if it is going to succeed in the 21st century and beyond.

(Update: Welcome, visitors from Sony/Psygnosis and readers of the Escapist. Please don't take this as criticism of Sony, just of the practices as they were described to me. No company has QA perfected, and Sony has released some wonderful titles over the years. However, past success is not a guarantee of future success as this incident proves. Trust in Sony's ability to deliver is already shaken, not only from a consumer standpoint, but a developer standpoint as well. [Hell, I still haven't received my taxi fare reimbursement...]

First-party games are supposed to push the envelope with killer gameplay, crystal-clear graphics and first-rate quality. First-party games are supposed to sell not only the abilities of the console, but the promise of the platform.

Consider this a prod towards delivering the true promise of the platform: next-generation gaming for the masses. The masses don't like patching.)

(Edit: 10/24/2007, 9:30am: Added sponsored links.)


Jason said...

I used to work qa at sony. That description is dead on. I'm at a thrid party company now, and the interaction between qa and devside is much more open and helpful to the overall process of getting a clean game out. Bottom line, qa is as necessary and important to a succesful title as anything else. More focus needs to be put on finding the fine line you speak of.

Danger said...

I wonder how nintendo does their QA. Zelda twilight princess also contains game destroying bugs. And if this 8 week thing is something that has been going on since the beginning, I never ever came upon a bug in any of the old ps1 games from sony. Didn't own a ps2 so can't speak for that platform.

KØZ said...

QA suffers in every software project. I am a SQE and I see this in every industry I have had contact with. The only way that an industry or company starts to see the value in QA is when a defect causes a major financial disaster. Even after that major event QA eventually loses its luster and the old ways return.

The real issue is that companies won't let QA own a project. QA is just a test team, never the owners of analysis, design, and test. There are too many other department s with bigger egos that make sure QA is relegated to QC.

Vincent said...

I've worked at sony. The only thing worse than the way they handle QA is their customer service policys.

Brit said...

I've worked in the QA field for over 3 years now.

There is a naive part of me that envisions a future where QA is finally acknowledged as the important part of game development that it genuinely is and is (gasp) PAID accordingly. You still want to use QA as the "foot in the door" to the gaming industry? Fine - bring people in from the street andi try them out. But not everyone can do it or do it well. It requires a specific mindset as well as skill set. So people with the experience and track record should be compensated to the appropriate degree. We're talking the last line of defense before the product is in the hands of consumers. Obviously the designers and programmers and producers do a lot of hard work and should be considered the top dogs. But a buggy product sends a message to consumers that the company really doesn't take enough pride in the product to give it the polish it deserves. Isn't that worth something in the long run - to put your name on something that looks like you cared about it?

The company I do QA for now is not in the game industry so I make a decent wage and have adequate benefits. As much fun as I had working in game QA, there is no way I could return to it until the industry decides to grow up and adopt business practices that reflect the high-profile it is in.