Starting in September, I started having issues getting my mail reliably. My wife hasn't...just me. She even noticed my name removed from the little slip of paper on the inside of our mailbox. I guess one of our three mailpeople doesn't like me. It wouldn't be so bad, really, it if wasn't what I wasn't getting.
I haven't been getting several of my bills, but all of my junk mail has been arriving. As a result, I've had to pay several late fees because I'm one of those guys who sends in the bill when it arrives, and if I don't get a bill, I don't think about it.
Of course, that's a bit difficult to do now because I still haven't received my first paycheck, even though it was mailed from inside Dallas several days ago. Now the earliest it can arrive is Wednesday because the Post Office won't be delivering any mail on Monday because of New Year's Day, and no deliveries on Tuesday because of the state funeral for Gerald Ford.
Now watch my first Google AdSense check get "misdirected" as well when it ships out in February...
December 31, 2006
December 29, 2006
The Pirate's Plank
Okay, let's talk about the number one issue that has come across on the tech support forums and E-mails for "SiN Episodes: Emergence," the Radek office trap.
Issue: Player trapped after Radek leaves office
Severity: 1
Repro Steps: There are two ways that this could happen with the initial release of the game.
We didn't see that behavior at all during testing. Even with less than .1% of our customers at the time experiencing it, we still went through and got it fixed in the first update over Steam.
After that update, we were still getting reports of this behavior. Obviously it shouldn't happen, because the trigger was grown to encompass Radek's final pathnode, so there was absolutely no way that the Elexis trigger brush wouldn't be tripped.
We managed to track down a copy of the warez version of the game, and used that to isolate out the problem. In this case, the same behavior as above was still happening, but the cause was changing because of how the warez morons decided to crack the game.
The last four months of testing on the game were spent using an internal Steam server. The reason that we were testing over Steam was that Source and Steam act differently when shown loose files as opposed to files inside GCF's. When a file is inside a GCF, it is considered an official release. When it's a loose file, it is considered a mod. The warez morons unpacked the files from the GCF's and modified a DLL file. As a result, it triggered the behavior change in the engine.
Here's what happened. "SiN Episodes" loaded up SE1_U4Lab02 and tried to load the pathnode file. The pathnode file was missing some file system metadata, so the engine thought that the pathnode file was out-of-date. The engine loaded the pathnode file, but then created a new "quick" pathnode file. The "quick" pathnode file is there so that map authors can quickly test their maps before building the full pathnode file. Because of how the map was constructed (dynamic bridge, sealed doors, etc.), the "quick" pathnode file had no link between Radek's start point and his end point.
As a result, if someone using the warez version plays the level and does not die, Radek uses the "out-of-date" pathnode file and completes his path, but is still subject to the previously mentioned potential causes. However, if the player exits the game after loading the level or restarts, or dies on the level and reloads, or starts a new game after getting to U4Lab02 once, Radek will never trigger the Elexis portion of the cutscene, and the player will be trapped.
In this case, a bug was introduced by the pirates because it was causing a codepath to be executed with the production data that would never be executed with production data.
On the upside, because of how the savegame files work, we could tell by looking at the savegame whether or not the player was using a legitimate copy of the game. On the downside, the vast majority of the people I spoke with who were asking for support for this issue knew that they were using a pirated version of the game...they just didn't care.
And with that, I weep for the future of the PC games industry. People felt that they were entitled to support, even for a copy of the game that they didn't buy. Once a sense of entitlement falls on a crowd, no amount of education will change their ways.
Issue: Player trapped after Radek leaves office
Severity: 1
Repro Steps: There are two ways that this could happen with the initial release of the game.
- The player is on a very slow machine, and does not watch Radek go through the door. (28 reports)
- The player made a massive mess in the crane control room, and Radek has to teleport past some debris. (11 reports)
We didn't see that behavior at all during testing. Even with less than .1% of our customers at the time experiencing it, we still went through and got it fixed in the first update over Steam.
After that update, we were still getting reports of this behavior. Obviously it shouldn't happen, because the trigger was grown to encompass Radek's final pathnode, so there was absolutely no way that the Elexis trigger brush wouldn't be tripped.
We managed to track down a copy of the warez version of the game, and used that to isolate out the problem. In this case, the same behavior as above was still happening, but the cause was changing because of how the warez morons decided to crack the game.
The last four months of testing on the game were spent using an internal Steam server. The reason that we were testing over Steam was that Source and Steam act differently when shown loose files as opposed to files inside GCF's. When a file is inside a GCF, it is considered an official release. When it's a loose file, it is considered a mod. The warez morons unpacked the files from the GCF's and modified a DLL file. As a result, it triggered the behavior change in the engine.
Here's what happened. "SiN Episodes" loaded up SE1_U4Lab02 and tried to load the pathnode file. The pathnode file was missing some file system metadata, so the engine thought that the pathnode file was out-of-date. The engine loaded the pathnode file, but then created a new "quick" pathnode file. The "quick" pathnode file is there so that map authors can quickly test their maps before building the full pathnode file. Because of how the map was constructed (dynamic bridge, sealed doors, etc.), the "quick" pathnode file had no link between Radek's start point and his end point.
As a result, if someone using the warez version plays the level and does not die, Radek uses the "out-of-date" pathnode file and completes his path, but is still subject to the previously mentioned potential causes. However, if the player exits the game after loading the level or restarts, or dies on the level and reloads, or starts a new game after getting to U4Lab02 once, Radek will never trigger the Elexis portion of the cutscene, and the player will be trapped.
In this case, a bug was introduced by the pirates because it was causing a codepath to be executed with the production data that would never be executed with production data.
On the upside, because of how the savegame files work, we could tell by looking at the savegame whether or not the player was using a legitimate copy of the game. On the downside, the vast majority of the people I spoke with who were asking for support for this issue knew that they were using a pirated version of the game...they just didn't care.
And with that, I weep for the future of the PC games industry. People felt that they were entitled to support, even for a copy of the game that they didn't buy. Once a sense of entitlement falls on a crowd, no amount of education will change their ways.
Tags:
Piracy,
SiN Episodes
December 28, 2006
Pirate Wall of Shame?
Here's a bit of an idea that's been bouncing around my head for a little bit...
How would you guys feel if I set up a "Piracy Wall of Shame" for these guys who decide to warez our products and still contact support?
The rules would be:
1) Once you have verified that someone stole your game and contacted support, you'd forward the entire support thread (minus any confidential information) to the Wall of Shame E-mail address.
2) Once the WoS verifies the claim, the E-mail thread is posted on the site.
3) Each individual game would have its own WoS page that contained all the threads.
4) Each WoS page would have a link to where they can buy the game legally via Amazon.com. (The link would have a commission link back to cover the costs of running the site.)
5) If someone who was posted decides to purchase the game after being posted on the site, they can be removed by E-mailing a digital picture of the game package, receipt and their name/E-mail address written on the receipt to the WoS site.
6) Once the WoS verifies the purchase, the E-mail is removed, and a notice is put up that a legitimate sale was created through use of the site.
Any thoughts?
How would you guys feel if I set up a "Piracy Wall of Shame" for these guys who decide to warez our products and still contact support?
The rules would be:
1) Once you have verified that someone stole your game and contacted support, you'd forward the entire support thread (minus any confidential information) to the Wall of Shame E-mail address.
2) Once the WoS verifies the claim, the E-mail thread is posted on the site.
3) Each individual game would have its own WoS page that contained all the threads.
4) Each WoS page would have a link to where they can buy the game legally via Amazon.com. (The link would have a commission link back to cover the costs of running the site.)
5) If someone who was posted decides to purchase the game after being posted on the site, they can be removed by E-mailing a digital picture of the game package, receipt and their name/E-mail address written on the receipt to the WoS site.
6) Once the WoS verifies the purchase, the E-mail is removed, and a notice is put up that a legitimate sale was created through use of the site.
Any thoughts?
Tags:
Piracy
The Nerve...
Okay...it's been almost a month since I was laid off from Ritual. Almost a month of not having to deal with idiots asking for support for the pirated version of the game. Aside from still not having my personal belongings back, my time at Ritual was starting to fade from my memory...until 15 minutes ago.
What do I receive in my E-mail box?
And people thought I was making shit like this up...
What do I receive in my E-mail box?
From: wilton wilfred [address removed]My reply:
how can i get pass through the office of viktor radek? i've been standing for half an hour after he spoke and i realise that something wrong here. after searching for an answer, somebody said that i have to install steam. is this required? heck,i just want to finish this game.
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
I'm glad that you want to finish the game.I'd been trying to decide whether or not to post my post-mortem of this particular issue or not. Guess I should finish writing that fucker up so I'll stop getting E-mails like this one.
However, the issue you are encountering was introduced by the pirate group Provision when they broke the Steam copy protection on the game.
If you want to finish the game, please purchase a legitimate copy. If you purchase over Steam, it is available for $14.99, or if you want a copy at retail, it's $19.99.
Perhaps if more people had bought the game instead of pirating it, I wouldn't have been laid off from Ritual.
Sincerely,
Michael
And people thought I was making shit like this up...
Tags:
Piracy,
SiN Episodes
December 27, 2006
Code From Hell
I'm still wading through some of the worst code ever written. I keep coming across more nuggets that make me feel like I'm walking through a nightmare, like this gem which shows up nineteen times on one page:
...or broken JavaScript client validation with no follow-up server-side validation...
...or hand-coded time parsing routines that think 18:00 AM is a valid time...
...or repeated code to translate boolean
...or Hungarian prefixes that could never have applied in a sane world (
...if I ever think about outsourcing code to the company that created this abomination against maintenance coders, I may just have to shoot someone.
grdDeleted.AllowPaging = dsDeleted.Tables[0].Rows.Count <= 10 ? false : true;...or nine different ways of pulling from the same
DataReader in the same function......or broken JavaScript client validation with no follow-up server-side validation...
...or hand-coded time parsing routines that think 18:00 AM is a valid time...
...or repeated code to translate boolean
true and false values into the words "Yes" and "No" for storage as strings in the database......or Hungarian prefixes that could never have applied in a sane world (
cbxStuff is the name of a floating point variable, for example)......if I ever think about outsourcing code to the company that created this abomination against maintenance coders, I may just have to shoot someone.
December 24, 2006
Language and Human Interaction
Blue's News linked to a quiz today asking "Which American Accent Do You Have?"
The sad thing is that all it really did was make me think of how much progress I've lost over the last two years with my speech.
You see, when I was a child, I had quite a few speech impediments. It took over three years of working with a speech therapist to get to the point where I sounded "normal." A big reason why it took so long was that I never really had a lot of "face-to-face" interaction with people back when I was a child. I was pretty withdrawn, and spent more time with books than anything else.
Over the last two years, my interaction with people has been mostly through E-mail or over forums. My previous job didn't leave me a lot of time for a social life of any sort so I never really had a chance to get out and interact with people, and unfortunately, some minor impediments have started creeping back into my speech patterns unless I really concentrate on my speech.
Oh well...it happens. At least I have time now to get out and interact with people again. Last night, I even went to a Shackmeet over in Fort Worth. It was fun geeking out with the holiday spirit. After all, how often can you make a "Hand of Vecna" joke during the Christmas season and have it be appropriate?
| What American accent do you have? Your Result: The Midland "You have a Midland accent" is just another way of saying "you don't have an accent." You probably are from the Midland (Pennsylvania, southern Ohio, southern Indiana, southern Illinois, and Missouri) but then for all we know you could be from Florida or Charleston or one of those big southern cities like Atlanta or Dallas. You have a good voice for TV and radio. | |
| The Inland North | |
| The Northeast | |
| Philadelphia | |
| Boston | |
| The West | |
| The South | |
| North Central | |
| What American accent do you have? Quiz Created on GoToQuiz | |
The sad thing is that all it really did was make me think of how much progress I've lost over the last two years with my speech.
You see, when I was a child, I had quite a few speech impediments. It took over three years of working with a speech therapist to get to the point where I sounded "normal." A big reason why it took so long was that I never really had a lot of "face-to-face" interaction with people back when I was a child. I was pretty withdrawn, and spent more time with books than anything else.
Over the last two years, my interaction with people has been mostly through E-mail or over forums. My previous job didn't leave me a lot of time for a social life of any sort so I never really had a chance to get out and interact with people, and unfortunately, some minor impediments have started creeping back into my speech patterns unless I really concentrate on my speech.
Oh well...it happens. At least I have time now to get out and interact with people again. Last night, I even went to a Shackmeet over in Fort Worth. It was fun geeking out with the holiday spirit. After all, how often can you make a "Hand of Vecna" joke during the Christmas season and have it be appropriate?
December 21, 2006
Only in a virtual world... (Link may not be work safe)
Since John Wayne Bobbitt, the media has had a bit of an obsession with male genitals. Clinton didn't help any.
That said, I think that this story takes the cake. Only in a virtual world could someone protest what is going on by launching a "flying penis attack."
That said, I think that this story takes the cake. Only in a virtual world could someone protest what is going on by launching a "flying penis attack."
December 20, 2006
Dev/Test Fight
Today was my third day back in a development role, and it's a bit disturbing how easy it is to slip back into the development mindset, and even more disturbing how hard I have to fight it.
The codebase that I'm working on updating was outsourced to a coding sweatshop overseas. While I've seen some wonderful code come back from outsourcing partners, this code is closer to the code I'd write on a Dvorak keyboard while high on NyQuil and NoDoz.
My current task is to add two pieces of additional functionality without breaking any of the old functionality. Unfortunately, the codebase is so fragile that looking at it funny causes the server to dump core and fling it at the sysadmins.
The developer in me is thinking that a trained chimp could do better than this monstrosity, and that I'd be best served in the long run by rewriting the mess of code in a clean three-tier setup rather than the partial-one-tier abomination I'm working in. The tester in me is trying to minimize change as much as possible. After all, while the code is fragile, it does work. The code is in production. The tester in me wants to restrict change as much as possible to limit the chances of my code breaking anything else.
I've ended up compromising with myself. The new code that is going in is being written as close to a clean three-tier architecture as possible. If I can cleanly plug into the current system with a one or two line change, I'm doing that. If not, I'm planning out how to refactor that section of code to allow the clean add, refactoring the code bit by bit and testing each step of the way, and finally integrating my new code. I'm also adding in comments as I delve into the old codebase so that later, if a full refactor/rewrite is done, there are some landmarks that can be used as guideposts.
It's slower, and the old code isn't improving very much, but I'm not affecting the stability of the product and I'm getting the functionality in.
The codebase that I'm working on updating was outsourced to a coding sweatshop overseas. While I've seen some wonderful code come back from outsourcing partners, this code is closer to the code I'd write on a Dvorak keyboard while high on NyQuil and NoDoz.
My current task is to add two pieces of additional functionality without breaking any of the old functionality. Unfortunately, the codebase is so fragile that looking at it funny causes the server to dump core and fling it at the sysadmins.
The developer in me is thinking that a trained chimp could do better than this monstrosity, and that I'd be best served in the long run by rewriting the mess of code in a clean three-tier setup rather than the partial-one-tier abomination I'm working in. The tester in me is trying to minimize change as much as possible. After all, while the code is fragile, it does work. The code is in production. The tester in me wants to restrict change as much as possible to limit the chances of my code breaking anything else.
I've ended up compromising with myself. The new code that is going in is being written as close to a clean three-tier architecture as possible. If I can cleanly plug into the current system with a one or two line change, I'm doing that. If not, I'm planning out how to refactor that section of code to allow the clean add, refactoring the code bit by bit and testing each step of the way, and finally integrating my new code. I'm also adding in comments as I delve into the old codebase so that later, if a full refactor/rewrite is done, there are some landmarks that can be used as guideposts.
It's slower, and the old code isn't improving very much, but I'm not affecting the stability of the product and I'm getting the functionality in.
First Gift
While I was at work today, I received an instant message from my wife saying that a package had arrived for me. When she told me who it was from, I knew that it was time for the old "online photo gallery" treatment.
Hrm...never heard of "The Popcorn Factory"...
Thanks for the good vibes...I've been feeling significantly more relaxed since I found something to tide me over.
How can you tell you live in a Web 2.0 world? Everything is packaged in spam.
A handtied bow that my wife wouldn't let me untie. I had to slide it off. (After twelve years of marriage, you'd think she'd let me unwrap my own present my own way...but nooooo....)
Cookies and English toffee. For me, this is comfort food. My grandmother used to buy packages similar to this all the time, so I grew up with this.
Thanks!
Hrm...never heard of "The Popcorn Factory"...
Thanks for the good vibes...I've been feeling significantly more relaxed since I found something to tide me over.
How can you tell you live in a Web 2.0 world? Everything is packaged in spam.
A handtied bow that my wife wouldn't let me untie. I had to slide it off. (After twelve years of marriage, you'd think she'd let me unwrap my own present my own way...but nooooo....)
Cookies and English toffee. For me, this is comfort food. My grandmother used to buy packages similar to this all the time, so I grew up with this.
Thanks!
December 19, 2006
Think Testing Is Easy?
Many people believe that video game testing is the easiest job in the world. After all, all you have to do is sit down and play video games for eight hours a day...how hard can it be?
Try it.
My challenge to you is to take one day between now and the end of the month, and try to find as many bugs in this game as possible. Find ways to reproduce those bugs, and report them.
Bug reports go here, or get E-mailed to Sam. Follow this template when submitting bugs for maximum bug damage.
Consider this a way for you to find out if game testing really is the dream job that you think it is.
One request for Sam, though...publish the spec. It's hard enough to test a game with a spec...without one, testing is mostly guesswork...
...but then again, I haven't worked on a game with a spec yet...why start now?
Try it.
My challenge to you is to take one day between now and the end of the month, and try to find as many bugs in this game as possible. Find ways to reproduce those bugs, and report them.
Bug reports go here, or get E-mailed to Sam. Follow this template when submitting bugs for maximum bug damage.
Consider this a way for you to find out if game testing really is the dream job that you think it is.
One request for Sam, though...publish the spec. It's hard enough to test a game with a spec...without one, testing is mostly guesswork...
...but then again, I haven't worked on a game with a spec yet...why start now?
December 18, 2006
Back From First Day...
...and note to self for future development openings:
When the programmers on the interview are warning you about the quality (or lack thereof) of their codebase, I should probably place additional weight on their warning.
When the programmers on the interview are warning you about the quality (or lack thereof) of their codebase, I should probably place additional weight on their warning.
December 17, 2006
The Cause
A tester at one of the major publishers commented regarding my "QA Bill of Rights," saying he agreed with it about 80% and that I had "messianic fantasies" regarding my position in the industry.
I wouldn't say that I have messianic fantasies. I will say that I've been trying to fill a perceived hole.
Over the last decade, I've seen quality assurance get crammed lower and lower on the priority list. The tester week-to-production week ratio at has moved from a semi-reasonable 1:10 average to 1:35 and above at some studios. During that time, complexity has increased exponentially, and the industry has answered the challenge of testing this by throwing inexperienced bodies at the problem rather than approach it the way they approach every other discipline: with experienced, knowledgable people who are passionate about their work.
Designers, developers, artists, all disciplines have had someone to champion their cause...but not testing. The end result is that testing has turned into a faceless department filled with temps whose sole purpose is to pound on their controllers until something breaks and then hope that the development team doesn't ridicule their bugs and waive them outright.
Long story short (too late), quality assurance in this industry is broken. Nobody disputes that. I hoped to drive some positive change, and I've seen some change happen. It's been at the local level mostly, but it's a start.
Now I'm not going to be able to push for change from the inside for awhile. To be honest, I'm okay with that. But now people in games QA have to make a decision. Is the cause of reform worth pushing for or not? If it is, are you going to champion change in your own organization? Are you going to champion change in your area?
The industry can change for the better, but you have to make it happen. That's what I tried to do.
What will you do?
I wouldn't say that I have messianic fantasies. I will say that I've been trying to fill a perceived hole.
Over the last decade, I've seen quality assurance get crammed lower and lower on the priority list. The tester week-to-production week ratio at has moved from a semi-reasonable 1:10 average to 1:35 and above at some studios. During that time, complexity has increased exponentially, and the industry has answered the challenge of testing this by throwing inexperienced bodies at the problem rather than approach it the way they approach every other discipline: with experienced, knowledgable people who are passionate about their work.
Designers, developers, artists, all disciplines have had someone to champion their cause...but not testing. The end result is that testing has turned into a faceless department filled with temps whose sole purpose is to pound on their controllers until something breaks and then hope that the development team doesn't ridicule their bugs and waive them outright.
Long story short (too late), quality assurance in this industry is broken. Nobody disputes that. I hoped to drive some positive change, and I've seen some change happen. It's been at the local level mostly, but it's a start.
Now I'm not going to be able to push for change from the inside for awhile. To be honest, I'm okay with that. But now people in games QA have to make a decision. Is the cause of reform worth pushing for or not? If it is, are you going to champion change in your own organization? Are you going to champion change in your area?
The industry can change for the better, but you have to make it happen. That's what I tried to do.
What will you do?
Game QA Bill of Rights
Tomorrow, I step back into "the real world" for a time to rebuild my confidence in the games industry, restore some sanity, and refill my savings. However, I cannot leave my people hanging. I've dedicated the majority of my adult life to quality assurance. I've worked with teams that I would consider among the best in the industry. I've worked with others that I wouldn't trust to test if sponges could pick up water. But regardless of the department, there were many things in play that reduced not only the effectiveness of test, but the way that test interacted with the remainder of the department.
If quality is to improve in the products that the games industry creates, the way that we treat those responsible for maintaining the quality bar must improve as well. With that in mind, I present for your consideration "The Games Quality Assurance Bill of Rights."
1. You have the right to work no more than ten hours per day.
As a tester, your primary tools of the trade are your eyes and ears. Your powers of observation are crucial to your success as a tester. Studies have shown that after six hours, a person's ability to be observant begins to decline. In practice, any tester starts being more of a hindrance than a help after about ten hours of testing. Experience has shown that each hour spent by a tester after ten hours adds an additional 2.5 hours to the overall testing load to account for double-checking questionable and bad bugs and having to retest the area that was tested.
2. You have the right to at least one full day off per week.
Everyone needs at least one day off per week, be it to rest, attend religious services, do your laundry, pay bills, etc. Just because you're testing doesn't mean that your life outside of work stops. Your family misses you, your friends miss you...you need "you" time. Take it. It will help you recharge so you'll be more effective when you come back in.
3. You have the right to advance notice of expected overtime.
Testers need to have a life outside of work. You work to live, not live to work. However, there are times when extra hours are needed, be it for milestone testing or the final push to get a product out the door. I always tried to provide at least 48 hours notice of any extra hours coming up, and if I wasn't able to give that notice and the hours were required, I excused the extra hours for testers who had plans already.
4. You have the right to information about the game you are testing.
If we don't have information about what we're testing, if we're expected to test based on "feel" or "gut," we may as well just be monkeys with controllers. Testing isn't just throwing random input at a game until something breaks...it's a directed effort into the innards of the game trying to find the flaws that our customers are going to encounter so that they can be fixed. In order to have that directed effort, we need information. We need maps, behavior descriptions, system descriptions for systems that affect the user experience but are not exposed via UI elements, interaction charts...we need to know how the game is supposed to work so that when we are in the game, we'll know if something isn't working right.
5. You have the right to access your development team.
In this industry, we have a horrible habit of segregating test from the rest of the company. Builds aren't delivered, they're "thrown over the wall." There are times when text is not the ideal way to describe a bug. There are times when a tester should be able to go to a developer and say, "I've got this odd behavior...does this seem like a bug to you?" There are times when a tester should be able to go to a developer and ask, "I'm not sure how this feature is supposed to work...can you describe it for me?" Some people think that interaction like this is a waste of the programmer's time, but ask yourself this...would you rather spend two minutes talking to the person testing your feature, or a week handling "bad" bugs? Would you rather spend an hour talking to the tester who is going to be working on breaking your system and work out most of the major kinks before the system is even written, or four months rewriting your system after a major flaw is discovered by test?
6. You have the right to be evaluated on more than just your bug count.
A lot of QA managers evaluate their testers on bug count alone, but that doesn't paint an accurate picture of your performance as a tester, and can actually lead to an increase in counter-productive activities like bug padding, bug repetition, and skipping verifying repro steps in an effort to get the bug in faster. Activities such as test casing, bug triages, meetings with developers, and buddy testing reduce the amount of time that testers can use to find bugs, but still positively affect the quality of the project. In addition, not all bugs are created equal. Is one crash bug the same as a missing period in a dialog box? Also, not all areas will be as buggy as others. I used bug counts as an indicator of a potential problem, but only as an indicator that I need to dig deeper to see if there really is a problem.
7. You have the right to know why it was decided that your bugs would not be fixed.
In most companies, there is a triage process during the endgame where bugs "live or die." There may be times when a bug is just deemed too small to fix. Other times, fixing a bug may be seen as too risky because of the likelihood of other bugs being introduced by the fix. Other times, bugs may be offered up as trade in an effort to get other bugs fixed. Regardless of the decision, testers deserve the right to know why their bugs were waived. It's demoralizing to have a bug come back "won't fix," but having an understanding as to the process that led to that decision often helps.
8. You have the right to appeal a decision made about your bugs.
There have been many times when a bug was waived because it wasn't seen to meet a certain level of severity, even though the actual bug itself was significantly worse than people thought. Either the subject line of the bug didn't make it seem harsh enough, or the description came across as soft, or just the bug wasn't properly understood. There have been many times when a tester's appeal has caused a bug to be seen in a new light, and led to the bug being fixed, resulting in a higher-quality product.
9. You have the right to have your department decide how a feature should be tested.
Many people have a low opinion of testers. They believe that people with limited to no training and/or extremely low wages in comparison to themselves couldn't possibly understand the complications and/or ramifications of a system. What happens on occasion is that the means of how something should be tested will be passed down to the test manager or test lead as an "edict from on high." The main problem with this scenario is that the person deciding how the testing should be done has no experience with testing. It could be a coder who wants "the correct codepath" tested. It could be a middle-manager who wants to feel important. It could be someone who is trying to reduce the cost of testing a feature. Regardless of who, it is the right of the test department to decide how the feature should be tested. There is nothing wrong with taking the suggestion under advisement and use it as a guide towards one way of testing a feature, but if you are going to be held responsible for successfully testing a feature, you should at least be able to determine your own plan.
10. You have the right to be treated like a human being, both inside and outside your department.
Testers have to deal with the worst of all worlds. Their jobs are often temporary, low-paying and offer no benefits. As part of their jobs, they are forced to be the bearer of bad news. Their department is often the first to be blamed for any defect with any product, even if the issue was discovered and reported during testing. Starting from that, the job is already on a bit of a down note. It gets worse when you receive condescending comments from other departments, like "You're just a bunch of keyboard monkeys," or, "This system is so complex, I don't think you'd be able to understand it." There have been times when meals were provided for all employees working twelve or more hours except for the temp testers. (All of these examples are unfortunately real.) It can be just as bad between full-time testers and temporary testers. You are not your career and your worth is not defined by your profession. You are a human being, and you should be treated like one, whether you're full-time, contract on location, or outsourced in Shanghai.
Remember, we are all test, as bugs affect us all.
If quality is to improve in the products that the games industry creates, the way that we treat those responsible for maintaining the quality bar must improve as well. With that in mind, I present for your consideration "The Games Quality Assurance Bill of Rights."
1. You have the right to work no more than ten hours per day.
As a tester, your primary tools of the trade are your eyes and ears. Your powers of observation are crucial to your success as a tester. Studies have shown that after six hours, a person's ability to be observant begins to decline. In practice, any tester starts being more of a hindrance than a help after about ten hours of testing. Experience has shown that each hour spent by a tester after ten hours adds an additional 2.5 hours to the overall testing load to account for double-checking questionable and bad bugs and having to retest the area that was tested.
2. You have the right to at least one full day off per week.
Everyone needs at least one day off per week, be it to rest, attend religious services, do your laundry, pay bills, etc. Just because you're testing doesn't mean that your life outside of work stops. Your family misses you, your friends miss you...you need "you" time. Take it. It will help you recharge so you'll be more effective when you come back in.
3. You have the right to advance notice of expected overtime.
Testers need to have a life outside of work. You work to live, not live to work. However, there are times when extra hours are needed, be it for milestone testing or the final push to get a product out the door. I always tried to provide at least 48 hours notice of any extra hours coming up, and if I wasn't able to give that notice and the hours were required, I excused the extra hours for testers who had plans already.
4. You have the right to information about the game you are testing.
If we don't have information about what we're testing, if we're expected to test based on "feel" or "gut," we may as well just be monkeys with controllers. Testing isn't just throwing random input at a game until something breaks...it's a directed effort into the innards of the game trying to find the flaws that our customers are going to encounter so that they can be fixed. In order to have that directed effort, we need information. We need maps, behavior descriptions, system descriptions for systems that affect the user experience but are not exposed via UI elements, interaction charts...we need to know how the game is supposed to work so that when we are in the game, we'll know if something isn't working right.
5. You have the right to access your development team.
In this industry, we have a horrible habit of segregating test from the rest of the company. Builds aren't delivered, they're "thrown over the wall." There are times when text is not the ideal way to describe a bug. There are times when a tester should be able to go to a developer and say, "I've got this odd behavior...does this seem like a bug to you?" There are times when a tester should be able to go to a developer and ask, "I'm not sure how this feature is supposed to work...can you describe it for me?" Some people think that interaction like this is a waste of the programmer's time, but ask yourself this...would you rather spend two minutes talking to the person testing your feature, or a week handling "bad" bugs? Would you rather spend an hour talking to the tester who is going to be working on breaking your system and work out most of the major kinks before the system is even written, or four months rewriting your system after a major flaw is discovered by test?
6. You have the right to be evaluated on more than just your bug count.
A lot of QA managers evaluate their testers on bug count alone, but that doesn't paint an accurate picture of your performance as a tester, and can actually lead to an increase in counter-productive activities like bug padding, bug repetition, and skipping verifying repro steps in an effort to get the bug in faster. Activities such as test casing, bug triages, meetings with developers, and buddy testing reduce the amount of time that testers can use to find bugs, but still positively affect the quality of the project. In addition, not all bugs are created equal. Is one crash bug the same as a missing period in a dialog box? Also, not all areas will be as buggy as others. I used bug counts as an indicator of a potential problem, but only as an indicator that I need to dig deeper to see if there really is a problem.
7. You have the right to know why it was decided that your bugs would not be fixed.
In most companies, there is a triage process during the endgame where bugs "live or die." There may be times when a bug is just deemed too small to fix. Other times, fixing a bug may be seen as too risky because of the likelihood of other bugs being introduced by the fix. Other times, bugs may be offered up as trade in an effort to get other bugs fixed. Regardless of the decision, testers deserve the right to know why their bugs were waived. It's demoralizing to have a bug come back "won't fix," but having an understanding as to the process that led to that decision often helps.
8. You have the right to appeal a decision made about your bugs.
There have been many times when a bug was waived because it wasn't seen to meet a certain level of severity, even though the actual bug itself was significantly worse than people thought. Either the subject line of the bug didn't make it seem harsh enough, or the description came across as soft, or just the bug wasn't properly understood. There have been many times when a tester's appeal has caused a bug to be seen in a new light, and led to the bug being fixed, resulting in a higher-quality product.
9. You have the right to have your department decide how a feature should be tested.
Many people have a low opinion of testers. They believe that people with limited to no training and/or extremely low wages in comparison to themselves couldn't possibly understand the complications and/or ramifications of a system. What happens on occasion is that the means of how something should be tested will be passed down to the test manager or test lead as an "edict from on high." The main problem with this scenario is that the person deciding how the testing should be done has no experience with testing. It could be a coder who wants "the correct codepath" tested. It could be a middle-manager who wants to feel important. It could be someone who is trying to reduce the cost of testing a feature. Regardless of who, it is the right of the test department to decide how the feature should be tested. There is nothing wrong with taking the suggestion under advisement and use it as a guide towards one way of testing a feature, but if you are going to be held responsible for successfully testing a feature, you should at least be able to determine your own plan.
10. You have the right to be treated like a human being, both inside and outside your department.
Testers have to deal with the worst of all worlds. Their jobs are often temporary, low-paying and offer no benefits. As part of their jobs, they are forced to be the bearer of bad news. Their department is often the first to be blamed for any defect with any product, even if the issue was discovered and reported during testing. Starting from that, the job is already on a bit of a down note. It gets worse when you receive condescending comments from other departments, like "You're just a bunch of keyboard monkeys," or, "This system is so complex, I don't think you'd be able to understand it." There have been times when meals were provided for all employees working twelve or more hours except for the temp testers. (All of these examples are unfortunately real.) It can be just as bad between full-time testers and temporary testers. You are not your career and your worth is not defined by your profession. You are a human being, and you should be treated like one, whether you're full-time, contract on location, or outsourced in Shanghai.
Remember, we are all test, as bugs affect us all.
December 16, 2006
The CPL Scoop
Here's the scoop from the CPL:
- CPL is going global again next year. CounterStrike 1.6 is the focus; $500k purse.
- CPL is going to be creating their own game. Tom Mustaine is running the studio behind it. Game is codenamed "Severity."
- CPL needs to test their PA and video playback systems before running anything. We saw Romero's talk, but didn't hear it.
Losing The Game That Matters
There have been rumors that Doug Lowenstein is stepping down from his position as head of the ESA. While it's great to point back to his track record as head of the ESA and see the legal victories and acceptance of the ratings system grow over the last several years, the games industry is in one of the worst places it could be because Doug never played the one game that mattered...the public game.
We never had a vocal counterpoint to our critics from the ESA. Our critics were allowed to take center stage and put us on the defensive, forcing a twenty-fold increase in penalties from the ESRB and putting game developers in a self-censoring mode for the last year.
We had an online presence from bloggers, a single paper from a gentleman at MIT trying to debunk gaming myths, and more messageboard postings than you can shake a stick at, but the ESA has been so focused on showing who buys and plays games and trumpeting their legal victories and anti-piracy efforts that they've let everyone think of us as legally-protected hunter-killer-trainers rather than a form of entertainment. If the courts are our only allies, we are losing the war.
As developers and publishers, we haven't been much better. Our titles get targeted, and our public response is...nothing. It makes us look like guilty children cowering in the corner from the scolding we're receiving.
As an industry, we've taken a "wait and see" response because we keep seeing from the past that every generation has their strawman for its ills. However, if we don't start opening our mouths and fighting back, we aren't going to end up like the movie industry...embedded in the mainstream. We'll end up like Dungeons and Dragons...marginalized and mostly forgotten.
Long story short...if you want to protect gaming for the future, don't just type...open your fucking mouth.
We never had a vocal counterpoint to our critics from the ESA. Our critics were allowed to take center stage and put us on the defensive, forcing a twenty-fold increase in penalties from the ESRB and putting game developers in a self-censoring mode for the last year.
We had an online presence from bloggers, a single paper from a gentleman at MIT trying to debunk gaming myths, and more messageboard postings than you can shake a stick at, but the ESA has been so focused on showing who buys and plays games and trumpeting their legal victories and anti-piracy efforts that they've let everyone think of us as legally-protected hunter-killer-trainers rather than a form of entertainment. If the courts are our only allies, we are losing the war.
As developers and publishers, we haven't been much better. Our titles get targeted, and our public response is...nothing. It makes us look like guilty children cowering in the corner from the scolding we're receiving.
As an industry, we've taken a "wait and see" response because we keep seeing from the past that every generation has their strawman for its ills. However, if we don't start opening our mouths and fighting back, we aren't going to end up like the movie industry...embedded in the mainstream. We'll end up like Dungeons and Dragons...marginalized and mostly forgotten.
Long story short...if you want to protect gaming for the future, don't just type...open your fucking mouth.
December 15, 2006
Gearbox/Tom/CPL
Congratulations to Gearbox on getting the deal with Sega to work on an Aliens FPS. You've got a kick-ass team up there, and I know you'll be able to pull off a shooter that will make the Dallas-area game development community proud.
I'm fairly certain that is going to be one of the things that Randy Pitchford is going to be talking about tomorrow evening during the CPL "Open Discussion" during the CPL Finals.
Of course, Tom Mustaine is going to be up there as well. I wonder what he's going to be saying?
(Edit, Sunday 12/17: Fixed typo. Thanks Chris.)
I'm fairly certain that is going to be one of the things that Randy Pitchford is going to be talking about tomorrow evening during the CPL "Open Discussion" during the CPL Finals.
Of course, Tom Mustaine is going to be up there as well. I wonder what he's going to be saying?
(Edit, Sunday 12/17: Fixed typo. Thanks Chris.)
Economics Of $20 Price Point
There's been a lot of speculation on the net lately regarding the status of a certain product I've been known to be associated with. I'm not going to say anything one way or another regarding said product, the number of departures over the last few months, or the sales figures of said title. As I'm no longer associated with the company in question, it isn't my place to say anything one way or another regarding any of the previous items. That said, there is one misconception that I want to talk about, and that is sales expectations at the $20 price point.
There are generally three price-point ranges that PC games tend to start out at. There is the premium price-point, which is $49.99-$59.99, the mid-range price-point, which is $29.99-$39.99, and the budget price-point, which is $19.99 and under. The three major price-point ranges have very unique sales curves.
The high-end games start high, sell heavy for the first month or two, then start having tiered price drops. If they sold well, they'll drop to mid-range and get reprinted, otherwise, they drop to budget for a sell-off. Most are gone from shelves after six months unless they're being reprinted.
Mid-range games tend to keep their price-point for a longer period of time, but they have a lower margin per unit. Generally, they'll keep their price-point for about four to six months before a price drop to the bargain price-point. Nine to ten months is usually the outside.
Games that start at the budget price-point rarely have that up-front "oomph." They're "investment titles." They have extremely low margins per unit, but they stay on shelves for an extremely long time. Most games with a budget price point sell slightly heavier up front, but then fall into a steady unit sales per week. In other words, they result in a steady income flow to even out a relatively unstable sales category.
As a result, the initial sales on a budget price-point won't "wow" anybody, but the game will stay on shelves for as long as the per-week sales are relatively level. As long as weekly sales worldwide are over 1,000 units a week, that title isn't going anywhere for at least a year. If the unit sales remain steady up to the end of the year, it will usually get repackaged in a cheaper jewel-case SKU for $9.99 and stick around for even longer. I can still go into Wal-Mart and Target and purchase budget price-point games that were released five years ago. The games are there because they still sell.
Now, does this mean that a certain title is going away? Not my place to say. I can say that I was doing some work related to it prior to my untimely departure, but not what you might think.
And with that, I wash my hands of the title in question. You want more information? Convince certain people that silence isn't a viable PR strategy.
There are generally three price-point ranges that PC games tend to start out at. There is the premium price-point, which is $49.99-$59.99, the mid-range price-point, which is $29.99-$39.99, and the budget price-point, which is $19.99 and under. The three major price-point ranges have very unique sales curves.
The high-end games start high, sell heavy for the first month or two, then start having tiered price drops. If they sold well, they'll drop to mid-range and get reprinted, otherwise, they drop to budget for a sell-off. Most are gone from shelves after six months unless they're being reprinted.
Mid-range games tend to keep their price-point for a longer period of time, but they have a lower margin per unit. Generally, they'll keep their price-point for about four to six months before a price drop to the bargain price-point. Nine to ten months is usually the outside.
Games that start at the budget price-point rarely have that up-front "oomph." They're "investment titles." They have extremely low margins per unit, but they stay on shelves for an extremely long time. Most games with a budget price point sell slightly heavier up front, but then fall into a steady unit sales per week. In other words, they result in a steady income flow to even out a relatively unstable sales category.
As a result, the initial sales on a budget price-point won't "wow" anybody, but the game will stay on shelves for as long as the per-week sales are relatively level. As long as weekly sales worldwide are over 1,000 units a week, that title isn't going anywhere for at least a year. If the unit sales remain steady up to the end of the year, it will usually get repackaged in a cheaper jewel-case SKU for $9.99 and stick around for even longer. I can still go into Wal-Mart and Target and purchase budget price-point games that were released five years ago. The games are there because they still sell.
Now, does this mean that a certain title is going away? Not my place to say. I can say that I was doing some work related to it prior to my untimely departure, but not what you might think.
And with that, I wash my hands of the title in question. You want more information? Convince certain people that silence isn't a viable PR strategy.
December 14, 2006
Service Pack Pending
Visual Studio 2005 Service Pack 1 is going to be coming tomorrow, according to Ars Technica.
This service pack has been anxiously awaited by many people, but:
This service pack has been anxiously awaited by many people, but:
- Vista support won't be contained in the service pack. For that, we're going to have to wait until Q1 2007. A public beta could be released tomorrow, but don't quote me on that.
- If you are developing against certain consoles using native code and not XNA, you'll have to hold off on installing the service pack until it is supported. (Same goes for previous iterations of said console and SP1 for previous iteration of the compiler...)
When the service pack is ready (supposedly, around 9am Pacific), you'll be able to download it here.
(Update 12/15, 8:55a: Well, it's a couple of hours early, but the service pack is up for download.)
December 13, 2006
Ding Dong, 9x is Dead
Hey, testers...
Working on a game that uses DirectX? Still worrying about Win9x support (95/98/ME)?
Well, don't worry too much. Starting with the December 2006 DirectX SDK, portions of DirectX will no longer work on Win9x.
Support for Shader Model 1.x is going away as well without a compiler flag override.
The legacy tail is getting shorter and shorter.
Working on a game that uses DirectX? Still worrying about Win9x support (95/98/ME)?
Well, don't worry too much. Starting with the December 2006 DirectX SDK, portions of DirectX will no longer work on Win9x.
Support for Shader Model 1.x is going away as well without a compiler flag override.
The legacy tail is getting shorter and shorter.
Spit Out
Mahatma Gandhi once said that "you must be the change you wish to see in the world." There may be many things in the world that I am cynical about, but that nugget of truth is the only thing that has kept me going in the games industry for most of my time in it. After all, the games industry is known for picking up young, idealistic talent, chewing them up with long hours and low pay, and then spitting them out.
When I accepted the job at Ritual, I wrote up a document I called the "Game QA Bill of Rights." It was a list of ten items that I had either heard about from friends at other game developers or experienced at Microsoft that I never wanted to experience again. I'll be posting that document later, but please keep in mind for a few minutes that I had an agenda when I came back into the game industry...I wanted to change it by example. I wanted to pull back the veil and show a glimpse inside the industry, as well as prod the industry into changing for the better.
For the most part, I feel that my efforts over the last two years have been a success. The conversations that needed to happen are happening. People are starting to realize that quality assurance isn't a department, it's a process that every department needs to participate in or the customer suffers. Companies are starting to see good quality assurance as an investment in their future. It's just unfortunate that it's going to be one to two years before customers will be able to benefit from what is going on.
I've pushed a lot of buttons over the last two years, and I expect to continue to over the next few years...but at least for the near future, I'm going to be doing so from outside the profession and even the industry.
When I sent out my resumes over the last week, my QA experience was at the forefront of my resume, but it wasn't what interested employers. It was my development experience that people were interested in. Developers wanted to bring QA into their development department, and so that's where I'm going to spread the quality gospel. Once I've rebuilt my warchest, I'll come back...and this time, I'll bring friends.
When I accepted the job at Ritual, I wrote up a document I called the "Game QA Bill of Rights." It was a list of ten items that I had either heard about from friends at other game developers or experienced at Microsoft that I never wanted to experience again. I'll be posting that document later, but please keep in mind for a few minutes that I had an agenda when I came back into the game industry...I wanted to change it by example. I wanted to pull back the veil and show a glimpse inside the industry, as well as prod the industry into changing for the better.
For the most part, I feel that my efforts over the last two years have been a success. The conversations that needed to happen are happening. People are starting to realize that quality assurance isn't a department, it's a process that every department needs to participate in or the customer suffers. Companies are starting to see good quality assurance as an investment in their future. It's just unfortunate that it's going to be one to two years before customers will be able to benefit from what is going on.
I've pushed a lot of buttons over the last two years, and I expect to continue to over the next few years...but at least for the near future, I'm going to be doing so from outside the profession and even the industry.
When I sent out my resumes over the last week, my QA experience was at the forefront of my resume, but it wasn't what interested employers. It was my development experience that people were interested in. Developers wanted to bring QA into their development department, and so that's where I'm going to spread the quality gospel. Once I've rebuilt my warchest, I'll come back...and this time, I'll bring friends.
RIP Peter Boyle
The best incarnation of "Frankenstein's Monster" is dead.
If it's any comfort, just think that he's gone into the afterlife and is currently causing Madeline Kahn to sing, "Oh, sweet mystery of life, I've found you..."
If it's any comfort, just think that he's gone into the afterlife and is currently causing Madeline Kahn to sing, "Oh, sweet mystery of life, I've found you..."
December 12, 2006
December 11, 2006
Smaller != Better
Okay, this link is entirely inappropriate as it discusses items that generally are not discussed in public, but come on...entitling your report "Indian Men Don't Measure Up" is hardly going to reassure anybody.
Congrats to the XNA Team!
For those of you who have been waiting for it, XNA Game Studio Express Edition has been released.
Go. Download. Create.
Go. Download. Create.
December 10, 2006
Still Not Next-Gen?
A lot of people have been saying, "Now that all three 'next-gen' consoles are out, can we please stop saying 'next-gen' in reference to the games?" My answer to that would be that while we now have the consoles, we don't necessarily have the games yet.
All consoles have three points where they could be considered next-gen milestones. The first milestone is when we see graphics that just weren't capable under the previous generation. The second milestone is when the extra power provides a gameplay experience that was simply impossible with the previous generation. Finally, there is the point where next-gen actually arrives, when the gameplay and the graphics meld in a single title. Let's look back at the Super Nintendo and use it as a basis for comparison.
At launch, there was PilotWings, which graphically was doing something that simply could not be done with the previous generation because of the new graphics chip in the SNES.
A short time later, there was Super Castlevania IV, which while it may have had a very odd visual style compared to the rest of the series, used the power of the console to allow new gameplay challenges (navigating a level that rotated), new player capabilities (diagonal attacks, swinging, etc.), and enemies that would have been impossible on previous platforms (an enemy growing as it was attacked, restricting the area available for the player to move around). However, there were many locations in the game where graphics weren't all they could be.
About two years later, we got StarFox. One of the first console games to provide true 3D graphics, StarFox used the power of the console to open entirely new avenues of gameplay: wingman protection missions, multiple routes through the game via difficulty level selection, ship damage that affected navigation and combat, a mixture of rail-based and free-roaming combat, and more.
So the question you need to ask yourself when deciding whether or not next-gen has arrived is which is important to you: the graphics, the gameplay or the fusion of the two. For me, Super Castlevania IV was the point where next-gen arrived for that generation because it let me do things that I wanted to do in previous games in the series, and the power of the platform let the developers do it.
What will it take for next-gen to arrive for you?
All consoles have three points where they could be considered next-gen milestones. The first milestone is when we see graphics that just weren't capable under the previous generation. The second milestone is when the extra power provides a gameplay experience that was simply impossible with the previous generation. Finally, there is the point where next-gen actually arrives, when the gameplay and the graphics meld in a single title. Let's look back at the Super Nintendo and use it as a basis for comparison.
At launch, there was PilotWings, which graphically was doing something that simply could not be done with the previous generation because of the new graphics chip in the SNES.
A short time later, there was Super Castlevania IV, which while it may have had a very odd visual style compared to the rest of the series, used the power of the console to allow new gameplay challenges (navigating a level that rotated), new player capabilities (diagonal attacks, swinging, etc.), and enemies that would have been impossible on previous platforms (an enemy growing as it was attacked, restricting the area available for the player to move around). However, there were many locations in the game where graphics weren't all they could be.
About two years later, we got StarFox. One of the first console games to provide true 3D graphics, StarFox used the power of the console to open entirely new avenues of gameplay: wingman protection missions, multiple routes through the game via difficulty level selection, ship damage that affected navigation and combat, a mixture of rail-based and free-roaming combat, and more.
So the question you need to ask yourself when deciding whether or not next-gen has arrived is which is important to you: the graphics, the gameplay or the fusion of the two. For me, Super Castlevania IV was the point where next-gen arrived for that generation because it let me do things that I wanted to do in previous games in the series, and the power of the platform let the developers do it.
What will it take for next-gen to arrive for you?
December 9, 2006
Summary of the Week
Ten in-person interviews. Thirty-four phone interviews. Finding out on Monday about four different positions.
Interesting mix as well...do I stay in the industry in an SDET-style role (like the one I've unofficially had for the better part of my career), do I move to web development and try to integrate a QA process into one of the most dysfunctional development processes I've ever seen, do I take a contract-to-hire position writing no-brainer data entry apps, or do I take a chance on a position that pays better than any of them, but would require me to beg for help to move back to Utah so I can take it?
What sucks most is that this is the absolute worst time of the year to be unemployed. So many people go on vacation the latter half of December that if a person isn't employed during the first two weeks of the month, they're essentially fucked until early next year. And with my savings depleted after a year of doubled utility bills and significantly higher rent, that doesn't put me in a good place.
Personally, I want to stay in the industry, but in the end, it comes down to what is best for my family.
Interesting mix as well...do I stay in the industry in an SDET-style role (like the one I've unofficially had for the better part of my career), do I move to web development and try to integrate a QA process into one of the most dysfunctional development processes I've ever seen, do I take a contract-to-hire position writing no-brainer data entry apps, or do I take a chance on a position that pays better than any of them, but would require me to beg for help to move back to Utah so I can take it?
What sucks most is that this is the absolute worst time of the year to be unemployed. So many people go on vacation the latter half of December that if a person isn't employed during the first two weeks of the month, they're essentially fucked until early next year. And with my savings depleted after a year of doubled utility bills and significantly higher rent, that doesn't put me in a good place.
Personally, I want to stay in the industry, but in the end, it comes down to what is best for my family.
December 8, 2006
Post 750 Miscellanea
Hope to hear more from [a company in Plano] on Monday. Keep your fingers crossed with me.
I've seen many anime music videos with great lipsync...but here's a human music video with great lipsync for the "Weird Al" fans.
Parts 4 and 5 of my Automating Games QA series are still under construction. Job search has taken priority.
I've seen many anime music videos with great lipsync...but here's a human music video with great lipsync for the "Weird Al" fans.
Parts 4 and 5 of my Automating Games QA series are still under construction. Job search has taken priority.
Indirect Slashdotting
Welcome, visitors from Slashdot via The Escapist.
Don't take silence to mean inactivity. There are many times when things cannot be said, either because things are not ready, or there may be contract or legal issues involved, or the news may just not be ready. Don't jump to conclusions just yet.
And now...I have to get to a job interview in Plano.
Don't take silence to mean inactivity. There are many times when things cannot be said, either because things are not ready, or there may be contract or legal issues involved, or the news may just not be ready. Don't jump to conclusions just yet.
And now...I have to get to a job interview in Plano.
Test Tool Types
Over the last decade, I've found that all test tools, regardless of what they do, can be split into one of four seperate categories.
First, there are tools that test needs to test. Things like debug spew readers, memory trackers, fault simulation tools, DVD emulation suites...all of these allow testers to either see the state of the program or exercise certain functionality in the program.
Second, there are the tools that makes test easier or more efficient. In games, things that compare maps and show areas that have changed since the last iteration, or quick buttons to set up massive battles between unit types for balance testing, or directed dependency graphs derived from code changes since the last build...anything that lets test reduce the amount of time they set up for tests or reduces the surface area required for testing falls into this category.
Third, there are the tools that make test cheaper. Automated BVT's and other types of automated testing, continuous integration servers...anything that keeps test testing instead of stuck on a bad build or lets a machine do the work of a tester falls into this area.
Finally, there are the tools that free test up to focus on finding harder bugs. Content testing tools like level-leak checkers, code testing like unit tests and integration tests, and even spell-check for help documentation...anything that keeps a bug from getting to the testers in the first place falls into this area.
Where does your company invest its time?
First, there are tools that test needs to test. Things like debug spew readers, memory trackers, fault simulation tools, DVD emulation suites...all of these allow testers to either see the state of the program or exercise certain functionality in the program.
Second, there are the tools that makes test easier or more efficient. In games, things that compare maps and show areas that have changed since the last iteration, or quick buttons to set up massive battles between unit types for balance testing, or directed dependency graphs derived from code changes since the last build...anything that lets test reduce the amount of time they set up for tests or reduces the surface area required for testing falls into this category.
Third, there are the tools that make test cheaper. Automated BVT's and other types of automated testing, continuous integration servers...anything that keeps test testing instead of stuck on a bad build or lets a machine do the work of a tester falls into this area.
Finally, there are the tools that free test up to focus on finding harder bugs. Content testing tools like level-leak checkers, code testing like unit tests and integration tests, and even spell-check for help documentation...anything that keeps a bug from getting to the testers in the first place falls into this area.
Where does your company invest its time?
December 7, 2006
Media Rambling Rant
The RIAA is trying to reduce the amount of royalties being paid to songwriters.
This is a difficult topic to talk about. On one side, the RIAA member companies are doing exactly what their stockholders expect them to do...find a way to increase the amount of money coming in so that they can maintain or increase their stock price.
On the flipside, because of the sheer number of exemptions and chargebacks in the music industry, artists rarely see dime one beyond their advances until they've sold over three million units.
In the middle is the public domain. Labels and artists both want to increase the maximum length of copyright protection because both want to make more money (or in the case of most artists, some money) off of the same item, but at the same time, the most recent increase in copyright protection combined with the sheer quantity of samples being used in media is increasing costs for artists and accounting complexity for labels.
This issue is beginning to creep into game development as game costs increase. Namco has said that it will take an average sell-through of 500,000 units for a next-generation title to break even, in part because of increased development costs (estimated to be approximately $5 million). Compare that to the last generation, where 350,000 to 400,000 units was generally required to recoup publisher development costs of $2 million.
More costs are getting pushed back to the developer as well. Every time you contact publisher technical support, it takes away the royalties on as many as seven copies from the developer. Staff used to fill in holes are being charged back against royalties at rates as high as $60 an hour. If you fail certification more than twice, the cost for further certifications is getting charged back.
With the percentage of games breaking even per year decreasing slowly over the last few years, you're going to start seeing some fairly interesting shifts in the industry in an effort to control costs. I'm actually expecting a lot of the majors to gradually shift to a hub-and-spoke model over a studio model so that certain departments (audio, animation, etc.) can benefit from economies of scale. We're already seeing the rise of the designer again (Wright, Jaffe, etc.), where the designer is being pushed more than the team itself.
Moves like this are beginning to simulate a shift to more of a movie industry model. As technology and formats begin to stabilize, it is very possible that the shift will become even more dramatic. Outsourcing for asset production is already at record highs in this industry, with some designers going all out in support of outsourcing.
So the question is fairly simple. When the industry restructures, will we be looking at several ILM's, or several BMG's? It's unfortunate that the answer is not as simple...
This is a difficult topic to talk about. On one side, the RIAA member companies are doing exactly what their stockholders expect them to do...find a way to increase the amount of money coming in so that they can maintain or increase their stock price.
On the flipside, because of the sheer number of exemptions and chargebacks in the music industry, artists rarely see dime one beyond their advances until they've sold over three million units.
In the middle is the public domain. Labels and artists both want to increase the maximum length of copyright protection because both want to make more money (or in the case of most artists, some money) off of the same item, but at the same time, the most recent increase in copyright protection combined with the sheer quantity of samples being used in media is increasing costs for artists and accounting complexity for labels.
This issue is beginning to creep into game development as game costs increase. Namco has said that it will take an average sell-through of 500,000 units for a next-generation title to break even, in part because of increased development costs (estimated to be approximately $5 million). Compare that to the last generation, where 350,000 to 400,000 units was generally required to recoup publisher development costs of $2 million.
More costs are getting pushed back to the developer as well. Every time you contact publisher technical support, it takes away the royalties on as many as seven copies from the developer. Staff used to fill in holes are being charged back against royalties at rates as high as $60 an hour. If you fail certification more than twice, the cost for further certifications is getting charged back.
With the percentage of games breaking even per year decreasing slowly over the last few years, you're going to start seeing some fairly interesting shifts in the industry in an effort to control costs. I'm actually expecting a lot of the majors to gradually shift to a hub-and-spoke model over a studio model so that certain departments (audio, animation, etc.) can benefit from economies of scale. We're already seeing the rise of the designer again (Wright, Jaffe, etc.), where the designer is being pushed more than the team itself.
Moves like this are beginning to simulate a shift to more of a movie industry model. As technology and formats begin to stabilize, it is very possible that the shift will become even more dramatic. Outsourcing for asset production is already at record highs in this industry, with some designers going all out in support of outsourcing.
So the question is fairly simple. When the industry restructures, will we be looking at several ILM's, or several BMG's? It's unfortunate that the answer is not as simple...
Tags:
Rant
Thank You [deleted]
Thank you very much, [company in Plano], for the interview this morning.
You guys have a kick-ass staff, and anybody would be honored to work for you, myself included.
It was great to see [censored], [censored] and [censored] again and to share war stories with veterans of both digital and physical wars.
Hope to hear back from you soon.
You guys have a kick-ass staff, and anybody would be honored to work for you, myself included.
It was great to see [censored], [censored] and [censored] again and to share war stories with veterans of both digital and physical wars.
Hope to hear back from you soon.
December 6, 2006
PEBSAO
One of the most common tech support acronyms you'll hear is "PEBKAC." It stands for Problem Exists Between Keyboard and Chair, and implies user error. However, I think it's time for a new acronym, "PEBSAO," which stands for Problem Exists Between Software and OS.
Most software testing is done in a clean-room environment. Does "X" work on a clean install of Windows? For games, driver versions are generally tested as well because of the number of issues fixed and caused by various driver versions.
However, there are many third-party applications that interfere with applications running. Antivirus software can modify files by appending data to the end, interfering with CRC checks. Uninstallation utilities can cause temporary files to get locked and cause massive slowdown during unpacks. Firewall software can cause loopback addresses to wig out. Packet burning software can interfere with copy-protection checks.
We can't test all the combinations. I know of about thirty different anti-virus solutions, eighty different uninstallers, sixty different personal firewalls, a few dozen packet burning programs. It adds up.
So again, we return to what most people are going to have. Unfortunately, not a lot of people have antivirus. Several testing locations test McAfee, Norton and TrendMicro, but not all. Very few places test interactions with uninstallation utilities. Because of the number of firewalls and various interfaces, most places stick with testing the built-in Windows Firewall at most. And packet burning software...well, at least until now, we haven't had to worry about it being in the operating system...of course, with Vista...
These are just four of the various permutations of third-party software on machines that hook into the operating system. I'm sure that there are hundreds more.
As testers, we shoot to be as complete as we can for our product, and we shoot to ensure that in our supported environment, we work well. When we find a problem with third-party software, we'll often contact the vendor to try to find a workaround on our end or a fix on their end. But even so, our primary responsibility is to our software and our supported environment.
So next time your custom-written firewall rules cause "X" to crash on loading, or your uninstaller daemon causes "Y" to take four or more hours to load, or the browser plugin you installed to get free viewings of "Debbie Does Dallas" causes every single sound effect to be replaced with an obscene version of "Oh Come All Ye Faithful," remember "PEBSAO" when you're calling support.
Most software testing is done in a clean-room environment. Does "X" work on a clean install of Windows? For games, driver versions are generally tested as well because of the number of issues fixed and caused by various driver versions.
However, there are many third-party applications that interfere with applications running. Antivirus software can modify files by appending data to the end, interfering with CRC checks. Uninstallation utilities can cause temporary files to get locked and cause massive slowdown during unpacks. Firewall software can cause loopback addresses to wig out. Packet burning software can interfere with copy-protection checks.
We can't test all the combinations. I know of about thirty different anti-virus solutions, eighty different uninstallers, sixty different personal firewalls, a few dozen packet burning programs. It adds up.
So again, we return to what most people are going to have. Unfortunately, not a lot of people have antivirus. Several testing locations test McAfee, Norton and TrendMicro, but not all. Very few places test interactions with uninstallation utilities. Because of the number of firewalls and various interfaces, most places stick with testing the built-in Windows Firewall at most. And packet burning software...well, at least until now, we haven't had to worry about it being in the operating system...of course, with Vista...
These are just four of the various permutations of third-party software on machines that hook into the operating system. I'm sure that there are hundreds more.
As testers, we shoot to be as complete as we can for our product, and we shoot to ensure that in our supported environment, we work well. When we find a problem with third-party software, we'll often contact the vendor to try to find a workaround on our end or a fix on their end. But even so, our primary responsibility is to our software and our supported environment.
So next time your custom-written firewall rules cause "X" to crash on loading, or your uninstaller daemon causes "Y" to take four or more hours to load, or the browser plugin you installed to get free viewings of "Debbie Does Dallas" causes every single sound effect to be replaced with an obscene version of "Oh Come All Ye Faithful," remember "PEBSAO" when you're calling support.
A Tale of Two Recruiters
I had two interviews today for two different development positions, set up by two different recruiters.
The first one really liked my attitude, felt I had a strong grasp on the SDLC, but then started asking me lots of questions related to Java. I said, "Java? I thought this was a C# position." He said it was a Java position. I pulled out my printout describing the position and he pulled out my resume. Every reference on my resume that said C# had been changed to read Java...but the job opening I had in my hand said C#. After a curt phonecall from both of us, both of us terminated our relationship with the recruiter.
The second one really liked my attitude, but was wondering what games experience brought to "the real world." I asked him how many data records his office processed in an average day, and he estimated 2 million. So I told him that I come from an industry where we process that many records thirty times a second on consumer-level hardware; that the user can and often does change and distort that information; that we have intelligent agents acting on the data in real-time and interacting with the user; and we do all of that with the user experience in mind.
I have a follow-up interview on Monday.
The first one really liked my attitude, felt I had a strong grasp on the SDLC, but then started asking me lots of questions related to Java. I said, "Java? I thought this was a C# position." He said it was a Java position. I pulled out my printout describing the position and he pulled out my resume. Every reference on my resume that said C# had been changed to read Java...but the job opening I had in my hand said C#. After a curt phonecall from both of us, both of us terminated our relationship with the recruiter.
The second one really liked my attitude, but was wondering what games experience brought to "the real world." I asked him how many data records his office processed in an average day, and he estimated 2 million. So I told him that I come from an industry where we process that many records thirty times a second on consumer-level hardware; that the user can and often does change and distort that information; that we have intelligent agents acting on the data in real-time and interacting with the user; and we do all of that with the user experience in mind.
I have a follow-up interview on Monday.
December 5, 2006
Evangelists
Over at Coding Horror, Jeff Atwood has a great post about evangelists.
If software development is a religion, I guess that means that I'm in a metaphorical exile after nailing a list of grievances to the chapel door. We are stuck in battle between those who believe that programs spring whole from the minds of their creators, and those who believe that programs evolve through trial and error.
Is there a place in this world for a Quality Assurance Evangelist?
If software development is a religion, I guess that means that I'm in a metaphorical exile after nailing a list of grievances to the chapel door. We are stuck in battle between those who believe that programs spring whole from the minds of their creators, and those who believe that programs evolve through trial and error.
Is there a place in this world for a Quality Assurance Evangelist?
My Ritual Legacy
I spent some time last night looking over my legacy at Ritual, and I actually feel pretty good about what I managed to accomplish before I left.
I managed to test a AAA-title with a QA department one-fourth the size of the one on "Bookworm Adventures" and with a five-figure budget. I managed to handle all support calls and forum posts for the first three days after launch by myself, and by the time I left, had total support contacts across all methods of contact under 1% of units sold. On an unannounced project that we were working on with a publisher, our milestones were never rejected, in part because I looked at the milestone deliverable lists we received from the eyes of the publisher, not the letter of the contract. Ritual was also the first developer to publish any sort of Vista compatibility lists for its legacy catalog.
I did a lot outside of my department as well. I wrote two full manuals. I wrote a patch for "SiN 1.12" to add functionality that we promised to the community. (Whether it will be released or not isn't up to me anymore.)
I did a lot on this blog as well to help out on the PR front, although sometimes the accompanying flare-up was so much that I didn't get the backup I was supposed to get. I was asked to go forward with the piracy stats, and it succeeded in getting people talking about piracy. Unfortunately, the pirates were louder and Something Awful decided to be the assholes they normally are, so I was essentially hung out to dry on that one. It's too bad, too, because if you look at some other employee blogs, you'll see that Ritual's employees have never been ones to shy away from controversy. You'll see discussions of everything from asset theft by former employees to some items that shouldn't be discussed on a (relatively) family-friendly blog.
Of course, there was also "YAUSB." I worked hand-in-hand with PR on that one to get the message out.
If I had to say that I had one success while at Ritual, it was that I added a human face not only to the company, but to quality assurance as well. I tried to keep a conversational tone in all of my communications because I believe that talking to someone and teaching them about a topic is more effective than just talking at someone.
Oh, well...such is life. Back to looking...
I managed to test a AAA-title with a QA department one-fourth the size of the one on "Bookworm Adventures" and with a five-figure budget. I managed to handle all support calls and forum posts for the first three days after launch by myself, and by the time I left, had total support contacts across all methods of contact under 1% of units sold. On an unannounced project that we were working on with a publisher, our milestones were never rejected, in part because I looked at the milestone deliverable lists we received from the eyes of the publisher, not the letter of the contract. Ritual was also the first developer to publish any sort of Vista compatibility lists for its legacy catalog.
I did a lot outside of my department as well. I wrote two full manuals. I wrote a patch for "SiN 1.12" to add functionality that we promised to the community. (Whether it will be released or not isn't up to me anymore.)
I did a lot on this blog as well to help out on the PR front, although sometimes the accompanying flare-up was so much that I didn't get the backup I was supposed to get. I was asked to go forward with the piracy stats, and it succeeded in getting people talking about piracy. Unfortunately, the pirates were louder and Something Awful decided to be the assholes they normally are, so I was essentially hung out to dry on that one. It's too bad, too, because if you look at some other employee blogs, you'll see that Ritual's employees have never been ones to shy away from controversy. You'll see discussions of everything from asset theft by former employees to some items that shouldn't be discussed on a (relatively) family-friendly blog.
Of course, there was also "YAUSB." I worked hand-in-hand with PR on that one to get the message out.
If I had to say that I had one success while at Ritual, it was that I added a human face not only to the company, but to quality assurance as well. I tried to keep a conversational tone in all of my communications because I believe that talking to someone and teaching them about a topic is more effective than just talking at someone.
Oh, well...such is life. Back to looking...
December 4, 2006
Rumor Control
Okay, rumor control.
Yes, I got a little internal flak for the Sony QA debacle. The flak consisted of a total of two (2) E-mails. Most of the edits to the original post were made prior to those E-mails because of people taking select pieces out of context and reading their own bias into them. Unfortunately, some of my wording was...colorful enough that people could make it mean almost anything.
Yes, I am no longer with Ritual Entertainment.
One has nothing to do with the other. More information later this week. I'd say more now, but I made an agreement with someone inside Ritual that I respect greatly, and I honor my agreements.
Yes, I got a little internal flak for the Sony QA debacle. The flak consisted of a total of two (2) E-mails. Most of the edits to the original post were made prior to those E-mails because of people taking select pieces out of context and reading their own bias into them. Unfortunately, some of my wording was...colorful enough that people could make it mean almost anything.
Yes, I am no longer with Ritual Entertainment.
One has nothing to do with the other. More information later this week. I'd say more now, but I made an agreement with someone inside Ritual that I respect greatly, and I honor my agreements.
Request
If you are currently hiring for a QA manager, QA lead or QA engineer and offer relocation benefits, please contact me at this E-mail address.
Thank you for reading.
Thank you for reading.
Change of Plans
Parts 2, 3 and 4 of the Sin/USK series will show up over at Ritualistic instead of here. I'm unsure as to the timeline.
The other portions of the Automating Games QA series will be posted over the next two weeks.
If you have questions regarding the change, search the site for the answer.
The other portions of the Automating Games QA series will be posted over the next two weeks.
If you have questions regarding the change, search the site for the answer.
Tune In Later...
When I get into work, part two of the SiN/USK series is going to go live.
Also, I made a mistake last week. I pointed to mistakes that someone else was making instead of pointing the camera at my own department and showing the mistakes that we made. I also pointed out a negative example over a positive example.
Today, I'm going to attempt to rectify both.
Also, I made a mistake last week. I pointed to mistakes that someone else was making instead of pointing the camera at my own department and showing the mistakes that we made. I also pointed out a negative example over a positive example.
Today, I'm going to attempt to rectify both.
December 3, 2006
P-V-P-D-M-C-A
The button has been pressed.
That is the first time I have ever seen a DMCA violation described as part of a comic strip. Nifty.
That is the first time I have ever seen a DMCA violation described as part of a comic strip. Nifty.
Sunday Links
Geek haiku; not just
for computer literate
people anymore
The mind craves variety. (By the way, if you used to read Sheldon on uComics, this is his new home...)
Why reading your employment contract is very important (bottom of page 1). (And at least according to the kids that I've spoken to, Bratz aren't succeeding because of their attitudes, but because they look more like kids than Barbie dolls do, so they're easier to relate to.)
Hrm...$59.99 game with all content required to be unlocked via microtransactions, or free download with limited content and the remaining content required to be unlocked via microtransactions? Good choice.
Mu? Meh.
for computer literate
people anymore
The mind craves variety. (By the way, if you used to read Sheldon on uComics, this is his new home...)
Why reading your employment contract is very important (bottom of page 1). (And at least according to the kids that I've spoken to, Bratz aren't succeeding because of their attitudes, but because they look more like kids than Barbie dolls do, so they're easier to relate to.)
Hrm...$59.99 game with all content required to be unlocked via microtransactions, or free download with limited content and the remaining content required to be unlocked via microtransactions? Good choice.
Mu? Meh.
November 30, 2006
The Box
I want you to stop reading this for a second and go play "Asteroid's Revenge." I'll wait.
Welcome back. Now, while the game did not have a lot of spit and polish (missing a life meter, shots originate from the center of the ships instead of the nose, horrid UI), it was still fun, even though it was just the original game of "Asteroids" flipped inside out.
This is the kind of thing that can spawn a successful game...taking another game and reversing the roles of the protagonists. However, it doesn't often happen in our industry. Why?
We try to pigeonhole our games into these set categories. (FPS, run-and-gun, stop-and-pop, RTS, RPS, RPG, MMO, match-3, card battle, top-down shooter, etc.) Mashups and reverses rarely fit into such narrow categories, but they can be so fun.
Even simple non-game time-wasters like "Kitten Cannon," with the infringetastic Daft Punk music, are difficult to categorize. (Yes, it isn't a real game...but it's damn fun.)
So, the industry insists on pigeonholing games into these set categories to be marketed, but they're fun? How do we get the games out?
We have at least three choices. We can go full-bore with a full development budget and hope that when we ship, the public happens to be in the mood for feline artillery or rock-beats-ship. We can go cheap and create the fun game with the gameplay only but no assets to speak of, and not make any money because people don't like paying money for something that looks like crap, but then watch a competitor slap a fresh coat of paint on our idea and have it take off.
Think outside the box for a second...what do you think the third option could be?
(And yes, while I did not take the cat picture, I captioned it. It seemed appropriate. You can get a metric litterbox-ton of cat pictures, including the source for the one I used, here.)
Welcome back. Now, while the game did not have a lot of spit and polish (missing a life meter, shots originate from the center of the ships instead of the nose, horrid UI), it was still fun, even though it was just the original game of "Asteroids" flipped inside out.
This is the kind of thing that can spawn a successful game...taking another game and reversing the roles of the protagonists. However, it doesn't often happen in our industry. Why?
We try to pigeonhole our games into these set categories. (FPS, run-and-gun, stop-and-pop, RTS, RPS, RPG, MMO, match-3, card battle, top-down shooter, etc.) Mashups and reverses rarely fit into such narrow categories, but they can be so fun.
Even simple non-game time-wasters like "Kitten Cannon," with the infringetastic Daft Punk music, are difficult to categorize. (Yes, it isn't a real game...but it's damn fun.)
So, the industry insists on pigeonholing games into these set categories to be marketed, but they're fun? How do we get the games out?
We have at least three choices. We can go full-bore with a full development budget and hope that when we ship, the public happens to be in the mood for feline artillery or rock-beats-ship. We can go cheap and create the fun game with the gameplay only but no assets to speak of, and not make any money because people don't like paying money for something that looks like crap, but then watch a competitor slap a fresh coat of paint on our idea and have it take off.
Think outside the box for a second...what do you think the third option could be?
(And yes, while I did not take the cat picture, I captioned it. It seemed appropriate. You can get a metric litterbox-ton of cat pictures, including the source for the one I used, here.)
Tags:
Game Development
Closed Blog Now Live
My private blog is now "live." If you are a friend or family member, send me an E-mail. I'll give you rights to the blog and a link to it.
Other QA Blogs To Check Out
I figure while everyone is playing "Ooh, look at the guy who spoke up," I'd use the chance to plug the blogs of some other people who advocate for the black-sheep in a family.
For languages, look no further than Cory Smith over at AddressOf.com. Like me, Cory is a tireless advocate of Visual Basic. It's not a kid's language anymore, and hasn't been for quite some time.
For development platforms, check out Managed DirectX and XNA with Andy over at TheZBuffer.com. Performant, stable, and it makes it very easy to eliminate several common types of errors.
For other QA bloggers, you've got a choice. On the app side, check out The Braidy Tester. On the games side, GameQABlog. In the middle, check out Sam Kalman.
For people who think I'm full of shit, check out Francesco Poli.
Tell them "the future industry pariah who dared to open his mouth about that which should not be mentioned" sent you.
For languages, look no further than Cory Smith over at AddressOf.com. Like me, Cory is a tireless advocate of Visual Basic. It's not a kid's language anymore, and hasn't been for quite some time.
For development platforms, check out Managed DirectX and XNA with Andy over at TheZBuffer.com. Performant, stable, and it makes it very easy to eliminate several common types of errors.
For other QA bloggers, you've got a choice. On the app side, check out The Braidy Tester. On the games side, GameQABlog. In the middle, check out Sam Kalman.
For people who think I'm full of shit, check out Francesco Poli.
Tell them "the future industry pariah who dared to open his mouth about that which should not be mentioned" sent you.
December Agenda
Monday, I'm going to be posting part 2 of 6 of the SiN/USK series. The entire series will be posted prior to Christmas.
Sometime next week, I'm going to be posting part 4 of 5 of the Automating Games QA series. (Yes, I split part 4 in two.) Both parts will be posted prior to Christmas.
Beyond that, I intend to take a break from the blog for most of December. With XNA Game Studio Express Edition coming out December 11, my 12-year anniversary shortly after that, Christmas shortly after that, and a week off in there as well, my hope is to spend time reconnecting with my family, focusing on my work, and planning out the next year...well, as much as I can.
Oh, and the observant among you may have noticed some additional changes made to my blog today. The changes were made as an additional step towards integrating this blog into my primary site.
Sometime next week, I'm going to be posting part 4 of 5 of the Automating Games QA series. (Yes, I split part 4 in two.) Both parts will be posted prior to Christmas.
Beyond that, I intend to take a break from the blog for most of December. With XNA Game Studio Express Edition coming out December 11, my 12-year anniversary shortly after that, Christmas shortly after that, and a week off in there as well, my hope is to spend time reconnecting with my family, focusing on my work, and planning out the next year...well, as much as I can.
Oh, and the observant among you may have noticed some additional changes made to my blog today. The changes were made as an additional step towards integrating this blog into my primary site.
Tags:
Personal
Apologies to Sony
Dear Sony:
I want to apologize for what I said earlier this week.
As I said, no company truly has QA figured out. Most people inside Sony QA actually read the post instead of reading the news and realized what I was trying to say, but there are several people out there who are taking in the headlines only and not digging beyond it.
I'm sorry for any grief that I may have caused you or your employees.
Sincerely,
Michael Russell
Seperation of Topics
As part of my process of refining this blog, I've decided that effective immediately, personal posts are going to be going to a private, friends-only blog.
Industry anecdotes, testing tips and discussion, and discussion about products will remain on this blog.
However, don't expect the removal of one topic to affect how far I go on these other topics. As this recent dust-up proves, people are not used to straight talk coming from this industry. Everyone assumes that we have an agenda, so we must be lying our asses off. One thing that caused issues for "SiN Episodes" was that we promised 4-5 hours of gameplay, so everyone assumed that it meant 2-3 hours because of "time inflation." Right now, our average playtime is 4h57m, so I'd say we were dead on.
I'm a very straightforward person. I do my best to say things as they are. While it may drive PR departments insane when I open my mouth, I'd be doing a disservice to myself and to quality assurance if I toned down my words, omitted more than was legally necessary, or intentionally misled people.
And as for an agenda, I do have one. My agenda is to bring quality assurance out of the basement and into the light. QA has become an army of disposable temps in this industry, and is seen as the invisible enemy of most development teams and the automatic scapegoat for most customers when something goes wrong. This perception will only fester and grow if nothing is said or done about it.
Industry anecdotes, testing tips and discussion, and discussion about products will remain on this blog.
However, don't expect the removal of one topic to affect how far I go on these other topics. As this recent dust-up proves, people are not used to straight talk coming from this industry. Everyone assumes that we have an agenda, so we must be lying our asses off. One thing that caused issues for "SiN Episodes" was that we promised 4-5 hours of gameplay, so everyone assumed that it meant 2-3 hours because of "time inflation." Right now, our average playtime is 4h57m, so I'd say we were dead on.
I'm a very straightforward person. I do my best to say things as they are. While it may drive PR departments insane when I open my mouth, I'd be doing a disservice to myself and to quality assurance if I toned down my words, omitted more than was legally necessary, or intentionally misled people.
And as for an agenda, I do have one. My agenda is to bring quality assurance out of the basement and into the light. QA has become an army of disposable temps in this industry, and is seen as the invisible enemy of most development teams and the automatic scapegoat for most customers when something goes wrong. This perception will only fester and grow if nothing is said or done about it.
November 29, 2006
The Wrong Comparison
Games developed at the same time for similar platforms are not a valid basis of comparison when it comes to image quality.
If I'm developing a game for the 360 and for the PS3 at the same time, I'm going to use the same assets. I'm going to be using the same shaders where possible. I'm not going to bust ass to uprez any of my assets. If anything, I'm going to drop the resolution of assets to get them to fit in memory or on the media if there is a problem. But if they're being developed at the same time, the assets are being tuned to the lowest common denominator as part of their creation.
This comparison is like putting the same ingredients into two seperate sausages, cooking them seperately, then comparing the tastes. What's the point?
If I'm developing a game for the 360 and for the PS3 at the same time, I'm going to use the same assets. I'm going to be using the same shaders where possible. I'm not going to bust ass to uprez any of my assets. If anything, I'm going to drop the resolution of assets to get them to fit in memory or on the media if there is a problem. But if they're being developed at the same time, the assets are being tuned to the lowest common denominator as part of their creation.
This comparison is like putting the same ingredients into two seperate sausages, cooking them seperately, then comparing the tastes. What's the point?
Tags:
Consoles
Open Letter To QA
I've been watching the news today, and I've been seeing my comments regarding my brush with Sony's FPQA department taken with various levels of sensationalism and intrigue, so I figured I'd take a brief moment to address several of the comments I've been seeing and then drop this sordid matter before it goes much further.
To all developers: QA should be an integral part of any development process, not an afterthought. This doesn't just mean developer testing (unit tests, integration tests, etc.). This doesn't mean the certification process by the platform holder. This means real testing with real testers. Playtesting should not be how you find bugs. Shipping the product should not be how you find bugs. There are people out there who excel at finding these types of problems before they pound your review scores to dust...get them, keep them happy, and put them to work. Most importantly, listen to them. Testing without action is masturbation.
To Sony QA: I realize that your staffing structure is a direct result of cost cutting measures. However, several people in your in-house development houses joke about the bugs they receive. A big part of the reason they get laughable bugs is that when you're bringing that many people on for such a short period of time, the quality of training the testers receives suffers, as does the quality of bugs. Test leads do what they can to filter bad bugs from getting through, but there is a finite number of hours in the day and the longer the hours are that your testers work, the more items that are going to slip through their fingers.
If you want to adjust the perception, bite the bullet. Hire great testers, bring them on full-time, work them a reasonable number of hours, pay the benefits. It takes time to change a culture, but change has to start somewhere. A defeatist attitude like "the 5% rule" I was told about only proves the culture's point. (And if testing is going on from day 1, 5% should never happen.)
To the Press: A lot of people place blame on any bugs in a shipping product solely at the feet of quality assurance. Some people believe that bugs making it out are the result of QA sloppiness, or QA "not fighting hard enough" for the customer. To be honest, there are times when that is the case. However, knee-jerk accusations towards QA don't help anyone. In fact, it is reactions like that which have led many publishers to believe that since the highly-paid testers "missed this issue," they may as well employ "controller monkeys" instead. After all, they're cheaper, work longer hours, and are disposable.
And when you get an article like this, don't just take my word for it! While I stand by everything that I said, nowhere have I seen any attempts to contact Sony for a statement. Nowhere have I seen a response from Sony. The only response I've seen have been from former Sony QA members who said, "Yep, sounds right." Please try to present a balanced viewpoint.
To my regular readers: Sorry for the distraction. I didn't think sharing my experiences would lead to such a hubbub.
To all developers: QA should be an integral part of any development process, not an afterthought. This doesn't just mean developer testing (unit tests, integration tests, etc.). This doesn't mean the certification process by the platform holder. This means real testing with real testers. Playtesting should not be how you find bugs. Shipping the product should not be how you find bugs. There are people out there who excel at finding these types of problems before they pound your review scores to dust...get them, keep them happy, and put them to work. Most importantly, listen to them. Testing without action is masturbation.
To Sony QA: I realize that your staffing structure is a direct result of cost cutting measures. However, several people in your in-house development houses joke about the bugs they receive. A big part of the reason they get laughable bugs is that when you're bringing that many people on for such a short period of time, the quality of training the testers receives suffers, as does the quality of bugs. Test leads do what they can to filter bad bugs from getting through, but there is a finite number of hours in the day and the longer the hours are that your testers work, the more items that are going to slip through their fingers.
If you want to adjust the perception, bite the bullet. Hire great testers, bring them on full-time, work them a reasonable number of hours, pay the benefits. It takes time to change a culture, but change has to start somewhere. A defeatist attitude like "the 5% rule" I was told about only proves the culture's point. (And if testing is going on from day 1, 5% should never happen.)
To the Press: A lot of people place blame on any bugs in a shipping product solely at the feet of quality assurance. Some people believe that bugs making it out are the result of QA sloppiness, or QA "not fighting hard enough" for the customer. To be honest, there are times when that is the case. However, knee-jerk accusations towards QA don't help anyone. In fact, it is reactions like that which have led many publishers to believe that since the highly-paid testers "missed this issue," they may as well employ "controller monkeys" instead. After all, they're cheaper, work longer hours, and are disposable.
And when you get an article like this, don't just take my word for it! While I stand by everything that I said, nowhere have I seen any attempts to contact Sony for a statement. Nowhere have I seen a response from Sony. The only response I've seen have been from former Sony QA members who said, "Yep, sounds right." Please try to present a balanced viewpoint.
To my regular readers: Sorry for the distraction. I didn't think sharing my experiences would lead to such a hubbub.
Tags:
Testing
SiN and USK (Part 1)
We're going to start out this discussion with the living: civilians and unarmed enemies.
For the U.S. release, we had a couple of set behaviors for anyone who wasn't armed if gunfire went off near them or they witnessed an attack. They would run to safety, crouch and express fear. If they had no "safety zone," they would just crouch and express fear.
For the most part, our playtesters left the unarmed people alone. One person made a concerted effort to kill everyone, armed or not, but that was an anomaly.
In the final iteration of the game, there was only one civilian in the game after you received your magnum, and he is run over by a semi truck within a few moments, so very few people get a chance to try to kill him.
When we got our USK report back, one of the items discussed was that you could kill people who have expressed fear. The logic behind this being that they felt that it was inhumane to kill someone who was obviously afraid of you and was not attacking you.
At this point, we had two choices. We could take the easy road and just make civilians and unarmed enemies (like the U4 technicians) invincible. Had we done this, people who purchased the USK version would get an experience similar to other games where when they come across a civilian, they can't really do anything…the civilian is just for show. The second choice would be to make it so that civilians don’t express fear to the player.
We decided to go the second route. We did this for two reasons. First, it seemed really dumb to us to have something in the game that couldn’t be shot, especially given how important gunplay was in "SiN Episodes." Second, there was one place in U4 Labs 02 where if you were unable to shoot the lab workers, the player could be blocked from progressing through the remainder of the game.
The final result for owners of the USK version is that civilians ignore threats caused by the player. They’ll still fear mutants in cutscenes, but they no longer beg for their lives before you kill them.
For the U.S. release, we had a couple of set behaviors for anyone who wasn't armed if gunfire went off near them or they witnessed an attack. They would run to safety, crouch and express fear. If they had no "safety zone," they would just crouch and express fear.
For the most part, our playtesters left the unarmed people alone. One person made a concerted effort to kill everyone, armed or not, but that was an anomaly.
In the final iteration of the game, there was only one civilian in the game after you received your magnum, and he is run over by a semi truck within a few moments, so very few people get a chance to try to kill him.
When we got our USK report back, one of the items discussed was that you could kill people who have expressed fear. The logic behind this being that they felt that it was inhumane to kill someone who was obviously afraid of you and was not attacking you.
At this point, we had two choices. We could take the easy road and just make civilians and unarmed enemies (like the U4 technicians) invincible. Had we done this, people who purchased the USK version would get an experience similar to other games where when they come across a civilian, they can't really do anything…the civilian is just for show. The second choice would be to make it so that civilians don’t express fear to the player.
We decided to go the second route. We did this for two reasons. First, it seemed really dumb to us to have something in the game that couldn’t be shot, especially given how important gunplay was in "SiN Episodes." Second, there was one place in U4 Labs 02 where if you were unable to shoot the lab workers, the player could be blocked from progressing through the remainder of the game.
The final result for owners of the USK version is that civilians ignore threats caused by the player. They’ll still fear mutants in cutscenes, but they no longer beg for their lives before you kill them.
Tags:
SiN Episodes
November 28, 2006
My Stance On OpenAL And Vista
There has been a lot of talk about developers moving to OpenAL after learning that Vista does not support DirectSound 3D hardware mixing. The biggest reason for the functionality going away was that unstable audio drivers were causing almost as many blue-screens as unstable video drivers. Direct hardware access doesn't mean a damn thing if the machine won't work because of either an app or driver bug.
I've got five major issues with OpenAL and Vista.
The first is that it is bypassing the HAL in Vista so we will yet again be at the mercy of the driver developers. With Vista, Microsoft has made system stability a cornerstone of the operating system. I don't like the idea of opening a backdoor to companies that can't even be bothered to fix blue-screen causing crashes in their current drivers.
The second is that while it started as a joint effort between the Loki Entertainment and Creative Labs, it has turned into a vendor-driven API. Remember Glide? Vendor-driven API's can be dangerous, especially when the vendor driving it has their stock price is barely $1.50 over their 52-week low, over $10 below their 5-year high, and facing stiff competition on nearly every front.
Third, audio config testing efforts are going to go through the roof with OpenAL games. I go to Creative's developer forums and read things like, "Well, it works on a Creative card, but on this other card, it crashes," and the response back is, "I'm glad to hear that it works on a Creative card." Piss-poor support is what led to the creation of UAA in the first place.
Fourth, because OpenAL's functionality maps to resources available on the host sound card, there are additional issues with the API design. For example, there is no way to query what the maximum number of allowed buffers is. Instead, you have to try to create the buffers, and if the API doesn't cause an access violation because of a null pointer exception (a common occurance), you then check the return value and if the return code happens to be "that's too high," then you decrement and start over...blech.
Finally, some of the titles listed may use OpenAL, but they also use Miles Sound System as a fallback for cards that don't support OpenAL. Why the hell would I want to use an audio API that doesn't work on several popular sound chipsets? If I have to license another sound library just to fill the config gaps in yours, I may as well use what I'm paying for to the fullest.
I've got five major issues with OpenAL and Vista.
The first is that it is bypassing the HAL in Vista so we will yet again be at the mercy of the driver developers. With Vista, Microsoft has made system stability a cornerstone of the operating system. I don't like the idea of opening a backdoor to companies that can't even be bothered to fix blue-screen causing crashes in their current drivers.
The second is that while it started as a joint effort between the Loki Entertainment and Creative Labs, it has turned into a vendor-driven API. Remember Glide? Vendor-driven API's can be dangerous, especially when the vendor driving it has their stock price is barely $1.50 over their 52-week low, over $10 below their 5-year high, and facing stiff competition on nearly every front.
Third, audio config testing efforts are going to go through the roof with OpenAL games. I go to Creative's developer forums and read things like, "Well, it works on a Creative card, but on this other card, it crashes," and the response back is, "I'm glad to hear that it works on a Creative card." Piss-poor support is what led to the creation of UAA in the first place.
Fourth, because OpenAL's functionality maps to resources available on the host sound card, there are additional issues with the API design. For example, there is no way to query what the maximum number of allowed buffers is. Instead, you have to try to create the buffers, and if the API doesn't cause an access violation because of a null pointer exception (a common occurance), you then check the return value and if the return code happens to be "that's too high," then you decrement and start over...blech.
Finally, some of the titles listed may use OpenAL, but they also use Miles Sound System as a fallback for cards that don't support OpenAL. Why the hell would I want to use an audio API that doesn't work on several popular sound chipsets? If I have to license another sound library just to fill the config gaps in yours, I may as well use what I'm paying for to the fullest.
Tags:
Game Development,
OpenAL
November 27, 2006
SiN and USK (Intro)
As some of you may know, we went through a fairly painful process of getting "SiN Episodes: Emergence" ready to ship in Germany.
Our initial submission was denied a USK rating, which essentially meant that USK felt that BPjM would have indexed it, and as a result, we had to go back to the drawing board and try to find the smallest possible changes that we could make that would get the game passed USK.
There were very few changes made to the USK version, and I'm going to be discussing six of those changes over the next few weeks. I'll be discussing the change itself, the reasoning behind the change, and show that there are no gameplay ramifications from the changes.
The six changes that we are going to be going over are ragdoll physics, fire, gibs, jetpack deaths, civilians/unarmed characters and applied forces. Saddle up, this is going to get interesting...
A few ground rules. First, I will not be posting any instructions on how to revert these changes. Second, comments linking to instructions on how to revert these changes will be deleted immediately. It's unfortunate, but these are the rules that I have to abide by in order to bring you this list.
Our initial submission was denied a USK rating, which essentially meant that USK felt that BPjM would have indexed it, and as a result, we had to go back to the drawing board and try to find the smallest possible changes that we could make that would get the game passed USK.
There were very few changes made to the USK version, and I'm going to be discussing six of those changes over the next few weeks. I'll be discussing the change itself, the reasoning behind the change, and show that there are no gameplay ramifications from the changes.
The six changes that we are going to be going over are ragdoll physics, fire, gibs, jetpack deaths, civilians/unarmed characters and applied forces. Saddle up, this is going to get interesting...
A few ground rules. First, I will not be posting any instructions on how to revert these changes. Second, comments linking to instructions on how to revert these changes will be deleted immediately. It's unfortunate, but these are the rules that I have to abide by in order to bring you this list.
Tags:
SiN Episodes
Quality Assurance at Sony
READ ME FIRST: Given that so many news outlets have been taking portions of this out of context, I need to say this.
1) I have nothing against Sony's QA department, contrary to what some reporters have said. I was commenting on the impression I got of how QA was perceived within Sony, not QA in Sony.
2) I talk about the impressions that I got seven months ago. Things may have changed, I don't know.
3) The department in question is the "last line of defense" inside Sony. From what I have been told, individual internal developers may have their own QA staffs on top of these.
4) These were my impressions, and are not necessarily the opinions of my past, present or future employers.
Sam Kalman made a post on November 22nd about a bug in Genji found by Chris Kohler, and it begs for the following story to be told.
Back in April, I was interviewed for a FPQA Manager position at Sony Computer Entertainment America's San Diego office. Sony was extremely nice. They flew me down and back first-class, took me out to lunch, etc.
Everyone I met there was a consumate professional, but there was a lot of underlying tension. I signed an NDA so I can't go into specifics, but there was talk about issues that only came up on production UMD's for PSP games, major friction between test and development teams with little to no management backing for test, little to no shared technology, extremely lax "user effect" bug metrics for determining whether or not to fix something, and a variety of other fairly hefty issues, not just from a process standpoint, but a overall culture standpoint. Microsoft is known for giving QA a bit too much say in the products that are developed, but the feeling I got inside Sony was that QA was seen as nothing but a bunch of monkeys with controllers.
The straw that broke the camel's back came in the last hour of my interview. I was told that the way that Sony tests their games is that there are one or two test leads on a project starting at about six months out. At T-8 weeks, between 80 and 100 temporary testers are brought on to test the game for those eight weeks. That's it. This was done for financial reasons, and as a QA Manager, I would be expected to run test the same way. Obviously, I didn't feel that was a valid way of handling QA.
The following morning, I sent an E-mail to Sony removing myself from consideration for the position because I didn't feel that I could run test the way that they wanted me to.
At Microsoft, the stringent QA processes often strangle creativity. At Sony, the lax QA process allows creativity to squash quality. It's hard to walk a middle ground where QA and creativity work hand in hand, but it is a tightrope that this industry is going to have to learn to walk if it is going to succeed in the 21st century and beyond.
(Update: Welcome, visitors from Sony/Psygnosis and readers of the Escapist. Please don't take this as criticism of Sony, just of the practices as they were described to me. No company has QA perfected, and Sony has released some wonderful titles over the years. However, past success is not a guarantee of future success as this incident proves. Trust in Sony's ability to deliver is already shaken, not only from a consumer standpoint, but a developer standpoint as well. [Hell, I still haven't received my taxi fare reimbursement...]
First-party games are supposed to push the envelope with killer gameplay, crystal-clear graphics and first-rate quality. First-party games are supposed to sell not only the abilities of the console, but the promise of the platform.
Consider this a prod towards delivering the true promise of the platform: next-generation gaming for the masses. The masses don't like patching.)
1) I have nothing against Sony's QA department, contrary to what some reporters have said. I was commenting on the impression I got of how QA was perceived within Sony, not QA in Sony.
2) I talk about the impressions that I got seven months ago. Things may have changed, I don't know.
3) The department in question is the "last line of defense" inside Sony. From what I have been told, individual internal developers may have their own QA staffs on top of these.
4) These were my impressions, and are not necessarily the opinions of my past, present or future employers.
Sam Kalman made a post on November 22nd about a bug in Genji found by Chris Kohler, and it begs for the following story to be told.
Back in April, I was interviewed for a FPQA Manager position at Sony Computer Entertainment America's San Diego office. Sony was extremely nice. They flew me down and back first-class, took me out to lunch, etc.
Everyone I met there was a consumate professional, but there was a lot of underlying tension. I signed an NDA so I can't go into specifics, but there was talk about issues that only came up on production UMD's for PSP games, major friction between test and development teams with little to no management backing for test, little to no shared technology, extremely lax "user effect" bug metrics for determining whether or not to fix something, and a variety of other fairly hefty issues, not just from a process standpoint, but a overall culture standpoint. Microsoft is known for giving QA a bit too much say in the products that are developed, but the feeling I got inside Sony was that QA was seen as nothing but a bunch of monkeys with controllers.
The straw that broke the camel's back came in the last hour of my interview. I was told that the way that Sony tests their games is that there are one or two test leads on a project starting at about six months out. At T-8 weeks, between 80 and 100 temporary testers are brought on to test the game for those eight weeks. That's it. This was done for financial reasons, and as a QA Manager, I would be expected to run test the same way. Obviously, I didn't feel that was a valid way of handling QA.
The following morning, I sent an E-mail to Sony removing myself from consideration for the position because I didn't feel that I could run test the way that they wanted me to.
At Microsoft, the stringent QA processes often strangle creativity. At Sony, the lax QA process allows creativity to squash quality. It's hard to walk a middle ground where QA and creativity work hand in hand, but it is a tightrope that this industry is going to have to learn to walk if it is going to succeed in the 21st century and beyond.
(Update: Welcome, visitors from Sony/Psygnosis and readers of the Escapist. Please don't take this as criticism of Sony, just of the practices as they were described to me. No company has QA perfected, and Sony has released some wonderful titles over the years. However, past success is not a guarantee of future success as this incident proves. Trust in Sony's ability to deliver is already shaken, not only from a consumer standpoint, but a developer standpoint as well. [Hell, I still haven't received my taxi fare reimbursement...]
First-party games are supposed to push the envelope with killer gameplay, crystal-clear graphics and first-rate quality. First-party games are supposed to sell not only the abilities of the console, but the promise of the platform.
Consider this a prod towards delivering the true promise of the platform: next-generation gaming for the masses. The masses don't like patching.)
Tags:
Testing
No Respect...
Just getting ready to go into work this morning, but I saw this article over at Kotaku and had to say something.
Look closely at the linked picture, and tell me what department is missing. (Hint: Playtesting != QA)
Look closely at the linked picture, and tell me what department is missing. (Hint: Playtesting != QA)
Tags:
Testing
November 25, 2006
Level 60
Today, after 16 days, 14 hours and 9 minutes /played, I finally reached level 60 with my first character.
Most of today was spent in Felwood killing Toxic Horrors. I told my wife what I was killing, but she thought I said "Toxic Whores." I riffed with that for a bit, saying, "Nah, they're over in Auberdine. Better be careful when dealing with them, though, or you'll end up with a bad case of sylphilis."
Thanks for making a game worth paying almost $17 a month for after tax, Blizzard. Looking forward to The Burning Crusade.
Tags:
Personal,
World of Warcraft
November 22, 2006
Runner-Up
I just got an E-mail from CMP, the people who run the Game Developers Conference. Turns out I was first runner-up in their haiku competition. No prize for being a runner-up, but given that I wrote my 8-haiku entry in under 20 minutes, being first runner-up isn't bad.
They'll post the haiku on the GDC website shortly.
Update, 2:16p: Looks like the one haiku of mine they didn't post was the one that cost me first place. My original entries, with the omitted one bolded:
They'll post the haiku on the GDC website shortly.
Update, 2:16p: Looks like the one haiku of mine they didn't post was the one that cost me first place. My original entries, with the omitted one bolded:
Annual absense
Has caused career turmoil
So I must attend
The keynote speeches
Lead to a renewed sense of
Purpose for our games
The featured classes
#define our purpose: Coding
Experiences
The panels bring out
The curmudgeon and the sheep
To battle it out
Casual Games Summit
Connects the hardcore with mom
To bring fun to all
IGF: Where next
Year's ripped-off gameplay is seen
Today in the flesh
GDC Awards:
We celebrate each other
But only one wins
So many classes...
So much information, but
Why ignore testing?
November 20, 2006
Escape of the Bird
About fifteen minutes ago, I received a phone call from my wife. She was in hysterics. I couldn't understand a thing that she was saying. It took me five minutes to calm her down to the point where I could understand what she was saying. Her bird, Chris, escaped. He pried open the door to his cage and flew away.
Yvonne has a small cage that she puts her birds in so she can put them out on the balcony and they can enjoy the weather and interact with other birds. Last year, Elmo escaped and she was devastated. Last Christmas, I gave her the money to go get herself another bird, and she got Chris.
These two events have been really hard on Yvonne because she hasn't really been able to make any friends down here in Dallas. Elmo was the last physical link she had with Utah, and now with her out of work because of her fibromyalgia and really just being stuck in the house, she's going to be alone again.
I guess it's good that I'm going to be home for a 4-day weekend this weekend...I can help her cope with this loss.
Yvonne has a small cage that she puts her birds in so she can put them out on the balcony and they can enjoy the weather and interact with other birds. Last year, Elmo escaped and she was devastated. Last Christmas, I gave her the money to go get herself another bird, and she got Chris.
These two events have been really hard on Yvonne because she hasn't really been able to make any friends down here in Dallas. Elmo was the last physical link she had with Utah, and now with her out of work because of her fibromyalgia and really just being stuck in the house, she's going to be alone again.
I guess it's good that I'm going to be home for a 4-day weekend this weekend...I can help her cope with this loss.
Tags:
Personal
November 17, 2006
Automating Games QA (part 3)
In this installment, we're going to be talking about combination testing.
Most games nowadays have some sort of customization system, be it your character, your "crib," your vehicle, etc. Testing the entire gamut of combinations by hand can actually get to the point where it is impossible to test within the time available for testing.
For example, let's say that you have a standard human avatar with a customizable shirt and pants. There are 10 different shirts available and 10 different pants available. That is 100 combinations right there. Let's add 10 hairstyles. That bumps it up to 1,000 combinations. Add 10 different fleshtones...10,000 combinations. Add a second gender...20,000 combinations. Add 5 different faces per gender...100,000 combinations. It adds up quickly.
Combination testing is designed to hit the two simplest types of bugs: single-value and two-value settings.
Now, if you look at the example above, while there are 100,000 combinations, there only a few individual settings: 10 shirts, 10 pants, 10 hairstyles and 5 faces per gender. That can mean either 40 settings if you assume everything else equal between the genders beside the faces, or 70 settings if everything is seperate between the genders. An automation script to individually cycle each of these single-value settings can quickly help eliminate bad items, and if the script screencaps each item, manual verification of item appearance will go fairly quickly.
One last thing: while handling the single-item tests, check the amount of memory that each item uses. A good additional test is to set all of your settings to their most memory-intensive setting and play the game that way to check for borderline out-of-memory conditions.
The second most-common type of bug is when two values are interdependent on each other. For example, you may have a hairstyle that clips through some geometry on a certain shirt in this example. Now even with automation, you're still going to have to manually verify the screenshots, so you want to minimize the number of shots you are looking at. This is where all-pairs testing comes in. This type of testing is also called pairwise testing, and there is a very in-depth example here.
This gets very easy if your combination lists are data-driven. Feed your lists into a tool like ALLPAIRS from James Bach and pass those lists into the game for testing. Pass the created list into your framework and have at it.
Automation testing is all about getting the grunt work done by the machine so you can focus on the non-automatable tasks. Always be on the lookout for tasks like this that you can automate.
The next games QA automation column will be on automating content testing. It's probably the hardest type of automation, and it will not be a good fit for most studios, which is why I left it for the end. Stay tuned.
Most games nowadays have some sort of customization system, be it your character, your "crib," your vehicle, etc. Testing the entire gamut of combinations by hand can actually get to the point where it is impossible to test within the time available for testing.
For example, let's say that you have a standard human avatar with a customizable shirt and pants. There are 10 different shirts available and 10 different pants available. That is 100 combinations right there. Let's add 10 hairstyles. That bumps it up to 1,000 combinations. Add 10 different fleshtones...10,000 combinations. Add a second gender...20,000 combinations. Add 5 different faces per gender...100,000 combinations. It adds up quickly.
Combination testing is designed to hit the two simplest types of bugs: single-value and two-value settings.
Now, if you look at the example above, while there are 100,000 combinations, there only a few individual settings: 10 shirts, 10 pants, 10 hairstyles and 5 faces per gender. That can mean either 40 settings if you assume everything else equal between the genders beside the faces, or 70 settings if everything is seperate between the genders. An automation script to individually cycle each of these single-value settings can quickly help eliminate bad items, and if the script screencaps each item, manual verification of item appearance will go fairly quickly.
One last thing: while handling the single-item tests, check the amount of memory that each item uses. A good additional test is to set all of your settings to their most memory-intensive setting and play the game that way to check for borderline out-of-memory conditions.
The second most-common type of bug is when two values are interdependent on each other. For example, you may have a hairstyle that clips through some geometry on a certain shirt in this example. Now even with automation, you're still going to have to manually verify the screenshots, so you want to minimize the number of shots you are looking at. This is where all-pairs testing comes in. This type of testing is also called pairwise testing, and there is a very in-depth example here.
This gets very easy if your combination lists are data-driven. Feed your lists into a tool like ALLPAIRS from James Bach and pass those lists into the game for testing. Pass the created list into your framework and have at it.
Automation testing is all about getting the grunt work done by the machine so you can focus on the non-automatable tasks. Always be on the lookout for tasks like this that you can automate.
The next games QA automation column will be on automating content testing. It's probably the hardest type of automation, and it will not be a good fit for most studios, which is why I left it for the end. Stay tuned.
Tags:
Automation,
Testing
November 16, 2006
Game-Themed Wedding
Okay, two of your friends are getting married. Both are gamers to the n-th degree. What do you do?
Evidently, you make them play a video game in order to get married.
Devious.
Evidently, you make them play a video game in order to get married.
Devious.
Halfway There...
I received an E-mail today from Blogger. They've agreed that my blog isn't a spam blog, but my entry screen still has the stupid spam blog CAPTCHA image which means that if it takes me longer than three minutes to write my entry, I have to re-enter the CAPTCHA again...
What a pain in the ass.
What a pain in the ass.
Tags:
Blogger
Cat-Scratch Fever
I'm still working on the final two Automated Games QA posts, but in the meantime...
Cute cats.
That is all.
Cute cats.
That is all.
Tags:
Cute
November 15, 2006
Pyramid Head
In my morning comic run, I came across this wonderful piece of Silent Hill fan-art (note: some images on site NSFW). Pyramid Head's words sounded like some lyrics from They Might Be Giants' "Particle Man," so I figured I'd flesh it out and try to keep it canon. Please forgive the musical interlude and the embedded spoilers.
Silent Hill, Silent Hill,
Doing the things that a town will,
What's it like is not important
Silent Hill
Is a town or is it in hell?
When darkness comes, does it feel well,
Or does the fog come from a well?
Nobody knows, Silent Hill...
Pyramid Head, Pyramid Head
Pyramid Head hates Silent Hill
They have a fight, nobody wins
Pyramid Head
Konami Man, Komani Man
Designs the entire universe man
Likes to scare the common man
Konami Man
The clock puzzle has a minute hand
An hour hand and a second hand
When they lock, the sanity ends
Powerful man, Konami Man
Sunderland, Sunderland
"Killed his wife with a pillow" man
Lives his life in a foggy land
Sunderland
He is depressed, and he is a mess,
and feels totally worthless.
Silent Hill will judge the man,
Try to redeem Sunderland.
Pyramid Head, Pyramid Head
Pyramid Head judges Sunderland
They have a fight, Pyramid wins
Pyramid Head
Tags:
Games
November 14, 2006
Right/Wrong vs. Right/Left
A lot of people go into games QA with one of two mentalities. Either they come in thinking, "cool, I get to play games for a living," in which case they generally leave the industry after five months never to return. Or they come in thinking, "there is so much wrong with games, this is my chance to make things right."
There is a lot of potential if you come into the games industry believing that you can fight the good fight and win the war against poor quality crapware, but I've found that people who keep that attitude burn out fairly quickly. It isn't that it's a wrong attitude, but the way that the games industry works, "right vs. wrong" just isn't...right.
I bring this up because I like making fun of commercials. Recently, they've been showing a commercial for DVD boxsets for the old Superman TV series and the last season of Lois and Clark. In the commercial, Dean Cain as Superman states in a matter-of-fact fashion, "I stand for what is right!" I always reply saying, "I stand for what is left!" For the most part, my reply is a joke, but I started thinking about it, and it does actually apply to how people tend to survive in QA.
Everyone on a team can pretty much agree about the "right" bugs to fix. Everyone on the team can also agree on the "wrong" bugs to fix: the ones that will result in additional instability, the ones that nobody will ever see, the ones that only occur if you noclip out of the world, the stupid shitty bugs that never work. However, between the "right" and the "wrong" bugs are the bugs in the grey area...the bugs that are left.
As testers, we stand for the bugs that are left. We fight for the bugs that aren't slam-dunk "must fix," but will have a serious impact on our customers. We wade into the grey, and escort our issues into the light.
Shifting from a "right/wrong" mentality to a "right/left" mentality isn't easy, but it makes survival in this industry so much easier.
There is a lot of potential if you come into the games industry believing that you can fight the good fight and win the war against poor quality crapware, but I've found that people who keep that attitude burn out fairly quickly. It isn't that it's a wrong attitude, but the way that the games industry works, "right vs. wrong" just isn't...right.
I bring this up because I like making fun of commercials. Recently, they've been showing a commercial for DVD boxsets for the old Superman TV series and the last season of Lois and Clark. In the commercial, Dean Cain as Superman states in a matter-of-fact fashion, "I stand for what is right!" I always reply saying, "I stand for what is left!" For the most part, my reply is a joke, but I started thinking about it, and it does actually apply to how people tend to survive in QA.
Everyone on a team can pretty much agree about the "right" bugs to fix. Everyone on the team can also agree on the "wrong" bugs to fix: the ones that will result in additional instability, the ones that nobody will ever see, the ones that only occur if you noclip out of the world, the stupid shitty bugs that never work. However, between the "right" and the "wrong" bugs are the bugs in the grey area...the bugs that are left.
As testers, we stand for the bugs that are left. We fight for the bugs that aren't slam-dunk "must fix," but will have a serious impact on our customers. We wade into the grey, and escort our issues into the light.
Shifting from a "right/wrong" mentality to a "right/left" mentality isn't easy, but it makes survival in this industry so much easier.
Tags:
Testing
"We" vs. "IFQ"
Managers have lots of little tips and tricks that they use to help them evaluate the frame of mind of their employees. One of my favorites is one I learned at Microsoft. It's called "the 'We' index." Next time you're talking to someone at another company, give this a shot.
To check the "we" index, strike up a conversation with a person about the company that they are at, and listen to how they verbally refer to their company.
If they say "we" a lot, they're happy with the company itself, and they feel like they belong. If they say "they" a lot, they're unhappy with the company, but still feel like they belong. If they say "I" a lot, they're happy with the company, but feel like they're alone in what they do. Finally, if they say the company's name a lot, they're not only unhappy with work, but they're extremely unhappy.
This works because the person who is speaking is completely unaware that they are doing it. It falls apart when they're writing E-mails, unfortunately, but in conversation, it can be an invaluable tool for determining the state of mind of a person related to their employment.
Now, there is a way that an employee can roughly evaluate themselves on this scale using what is called the "IFQ" index, but there is a catch. People who thought they were unhappy because of work have started feeling optimistic; others who have tried this method and thought they were happy started looking for other jobs. If you are open to that sort of experience, then advance to the next paragraph. Otherwise, see you next post.
To determine your "IFQ" index, go home. Kick everyone out of the house for ten to fifteen minutes. Go into the restroom and look straight into the mirror. Finally, say in the most convincing way that you can, "I fucking quit." Now think about how you feel after saying that.
If a pit has formed in your stomach and/or you feel slightly ill, then you would be miserable if you left where you are. If you aren't happy at work but you feel ill at the thought of leaving, that generally means that there is a specific something that is dragging you down and if you can find and isolate it, you'll enjoy work a lot more.
If you felt like a major weight has been lifted off your shoulders, you have two choices. The first choice is you can start looking for another job. Something has pissed you off to the point where even saying "you quit" has made you happy. The other option is to try to figure out what you want to get away from and try to correct those issues.
If you felt no change, sorry, no answer at this time...try again later.
To check the "we" index, strike up a conversation with a person about the company that they are at, and listen to how they verbally refer to their company.
If they say "we" a lot, they're happy with the company itself, and they feel like they belong. If they say "they" a lot, they're unhappy with the company, but still feel like they belong. If they say "I" a lot, they're happy with the company, but feel like they're alone in what they do. Finally, if they say the company's name a lot, they're not only unhappy with work, but they're extremely unhappy.
This works because the person who is speaking is completely unaware that they are doing it. It falls apart when they're writing E-mails, unfortunately, but in conversation, it can be an invaluable tool for determining the state of mind of a person related to their employment.
Now, there is a way that an employee can roughly evaluate themselves on this scale using what is called the "IFQ" index, but there is a catch. People who thought they were unhappy because of work have started feeling optimistic; others who have tried this method and thought they were happy started looking for other jobs. If you are open to that sort of experience, then advance to the next paragraph. Otherwise, see you next post.
To determine your "IFQ" index, go home. Kick everyone out of the house for ten to fifteen minutes. Go into the restroom and look straight into the mirror. Finally, say in the most convincing way that you can, "I fucking quit." Now think about how you feel after saying that.
If a pit has formed in your stomach and/or you feel slightly ill, then you would be miserable if you left where you are. If you aren't happy at work but you feel ill at the thought of leaving, that generally means that there is a specific something that is dragging you down and if you can find and isolate it, you'll enjoy work a lot more.
If you felt like a major weight has been lifted off your shoulders, you have two choices. The first choice is you can start looking for another job. Something has pissed you off to the point where even saying "you quit" has made you happy. The other option is to try to figure out what you want to get away from and try to correct those issues.
If you felt no change, sorry, no answer at this time...try again later.
Tags:
Employment,
Management
Subscribe to:
Posts (Atom)