December 31, 2006
I haven't been getting several of my bills, but all of my junk mail has been arriving. As a result, I've had to pay several late fees because I'm one of those guys who sends in the bill when it arrives, and if I don't get a bill, I don't think about it.
Of course, that's a bit difficult to do now because I still haven't received my first paycheck, even though it was mailed from inside Dallas several days ago. Now the earliest it can arrive is Wednesday because the Post Office won't be delivering any mail on Monday because of New Year's Day, and no deliveries on Tuesday because of the state funeral for Gerald Ford.
Now watch my first Google AdSense check get "misdirected" as well when it ships out in February...
December 29, 2006
Issue: Player trapped after Radek leaves office
Repro Steps: There are two ways that this could happen with the initial release of the game.
- The player is on a very slow machine, and does not watch Radek go through the door. (28 reports)
- The player made a massive mess in the crane control room, and Radek has to teleport past some debris. (11 reports)
We didn't see that behavior at all during testing. Even with less than .1% of our customers at the time experiencing it, we still went through and got it fixed in the first update over Steam.
After that update, we were still getting reports of this behavior. Obviously it shouldn't happen, because the trigger was grown to encompass Radek's final pathnode, so there was absolutely no way that the Elexis trigger brush wouldn't be tripped.
We managed to track down a copy of the warez version of the game, and used that to isolate out the problem. In this case, the same behavior as above was still happening, but the cause was changing because of how the warez morons decided to crack the game.
The last four months of testing on the game were spent using an internal Steam server. The reason that we were testing over Steam was that Source and Steam act differently when shown loose files as opposed to files inside GCF's. When a file is inside a GCF, it is considered an official release. When it's a loose file, it is considered a mod. The warez morons unpacked the files from the GCF's and modified a DLL file. As a result, it triggered the behavior change in the engine.
Here's what happened. "SiN Episodes" loaded up SE1_U4Lab02 and tried to load the pathnode file. The pathnode file was missing some file system metadata, so the engine thought that the pathnode file was out-of-date. The engine loaded the pathnode file, but then created a new "quick" pathnode file. The "quick" pathnode file is there so that map authors can quickly test their maps before building the full pathnode file. Because of how the map was constructed (dynamic bridge, sealed doors, etc.), the "quick" pathnode file had no link between Radek's start point and his end point.
As a result, if someone using the warez version plays the level and does not die, Radek uses the "out-of-date" pathnode file and completes his path, but is still subject to the previously mentioned potential causes. However, if the player exits the game after loading the level or restarts, or dies on the level and reloads, or starts a new game after getting to U4Lab02 once, Radek will never trigger the Elexis portion of the cutscene, and the player will be trapped.
In this case, a bug was introduced by the pirates because it was causing a codepath to be executed with the production data that would never be executed with production data.
On the upside, because of how the savegame files work, we could tell by looking at the savegame whether or not the player was using a legitimate copy of the game. On the downside, the vast majority of the people I spoke with who were asking for support for this issue knew that they were using a pirated version of the game...they just didn't care.
And with that, I weep for the future of the PC games industry. People felt that they were entitled to support, even for a copy of the game that they didn't buy. Once a sense of entitlement falls on a crowd, no amount of education will change their ways.
December 28, 2006
How would you guys feel if I set up a "Piracy Wall of Shame" for these guys who decide to warez our products and still contact support?
The rules would be:
1) Once you have verified that someone stole your game and contacted support, you'd forward the entire support thread (minus any confidential information) to the Wall of Shame E-mail address.
2) Once the WoS verifies the claim, the E-mail thread is posted on the site.
3) Each individual game would have its own WoS page that contained all the threads.
4) Each WoS page would have a link to where they can buy the game legally via Amazon.com. (The link would have a commission link back to cover the costs of running the site.)
5) If someone who was posted decides to purchase the game after being posted on the site, they can be removed by E-mailing a digital picture of the game package, receipt and their name/E-mail address written on the receipt to the WoS site.
6) Once the WoS verifies the purchase, the E-mail is removed, and a notice is put up that a legitimate sale was created through use of the site.
What do I receive in my E-mail box?
From: wilton wilfred [address removed]My reply:
how can i get pass through the office of viktor radek? i've been standing for half an hour after he spoke and i realise that something wrong here. after searching for an answer, somebody said that i have to install steam. is this required? heck,i just want to finish this game.
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
I'm glad that you want to finish the game.I'd been trying to decide whether or not to post my post-mortem of this particular issue or not. Guess I should finish writing that fucker up so I'll stop getting E-mails like this one.
However, the issue you are encountering was introduced by the pirate group Provision when they broke the Steam copy protection on the game.
If you want to finish the game, please purchase a legitimate copy. If you purchase over Steam, it is available for $14.99, or if you want a copy at retail, it's $19.99.
Perhaps if more people had bought the game instead of pirating it, I wouldn't have been laid off from Ritual.
And people thought I was making shit like this up...
December 27, 2006
grdDeleted.AllowPaging = dsDeleted.Tables.Rows.Count <= 10 ? false : true;...or nine different ways of pulling from the same
DataReaderin the same function...
...or hand-coded time parsing routines that think 18:00 AM is a valid time...
...or repeated code to translate boolean
falsevalues into the words "Yes" and "No" for storage as strings in the database...
...or Hungarian prefixes that could never have applied in a sane world (
cbxStuffis the name of a floating point variable, for example)...
...if I ever think about outsourcing code to the company that created this abomination against maintenance coders, I may just have to shoot someone.
December 24, 2006
|What American accent do you have? |
Your Result: The Midland
"You have a Midland accent" is just another way of saying "you don't have an accent." You probably are from the Midland (Pennsylvania, southern Ohio, southern Indiana, southern Illinois, and Missouri) but then for all we know you could be from Florida or Charleston or one of those big southern cities like Atlanta or Dallas. You have a good voice for TV and radio.
|The Inland North|
|What American accent do you have?|
Quiz Created on GoToQuiz
The sad thing is that all it really did was make me think of how much progress I've lost over the last two years with my speech.
You see, when I was a child, I had quite a few speech impediments. It took over three years of working with a speech therapist to get to the point where I sounded "normal." A big reason why it took so long was that I never really had a lot of "face-to-face" interaction with people back when I was a child. I was pretty withdrawn, and spent more time with books than anything else.
Over the last two years, my interaction with people has been mostly through E-mail or over forums. My previous job didn't leave me a lot of time for a social life of any sort so I never really had a chance to get out and interact with people, and unfortunately, some minor impediments have started creeping back into my speech patterns unless I really concentrate on my speech.
Oh well...it happens. At least I have time now to get out and interact with people again. Last night, I even went to a Shackmeet over in Fort Worth. It was fun geeking out with the holiday spirit. After all, how often can you make a "Hand of Vecna" joke during the Christmas season and have it be appropriate?
December 21, 2006
That said, I think that this story takes the cake. Only in a virtual world could someone protest what is going on by launching a "flying penis attack."
December 20, 2006
The codebase that I'm working on updating was outsourced to a coding sweatshop overseas. While I've seen some wonderful code come back from outsourcing partners, this code is closer to the code I'd write on a Dvorak keyboard while high on NyQuil and NoDoz.
My current task is to add two pieces of additional functionality without breaking any of the old functionality. Unfortunately, the codebase is so fragile that looking at it funny causes the server to dump core and fling it at the sysadmins.
The developer in me is thinking that a trained chimp could do better than this monstrosity, and that I'd be best served in the long run by rewriting the mess of code in a clean three-tier setup rather than the partial-one-tier abomination I'm working in. The tester in me is trying to minimize change as much as possible. After all, while the code is fragile, it does work. The code is in production. The tester in me wants to restrict change as much as possible to limit the chances of my code breaking anything else.
I've ended up compromising with myself. The new code that is going in is being written as close to a clean three-tier architecture as possible. If I can cleanly plug into the current system with a one or two line change, I'm doing that. If not, I'm planning out how to refactor that section of code to allow the clean add, refactoring the code bit by bit and testing each step of the way, and finally integrating my new code. I'm also adding in comments as I delve into the old codebase so that later, if a full refactor/rewrite is done, there are some landmarks that can be used as guideposts.
It's slower, and the old code isn't improving very much, but I'm not affecting the stability of the product and I'm getting the functionality in.
Hrm...never heard of "The Popcorn Factory"...
Thanks for the good vibes...I've been feeling significantly more relaxed since I found something to tide me over.
How can you tell you live in a Web 2.0 world? Everything is packaged in spam.
A handtied bow that my wife wouldn't let me untie. I had to slide it off. (After twelve years of marriage, you'd think she'd let me unwrap my own present my own way...but nooooo....)
Cookies and English toffee. For me, this is comfort food. My grandmother used to buy packages similar to this all the time, so I grew up with this.
December 19, 2006
My challenge to you is to take one day between now and the end of the month, and try to find as many bugs in this game as possible. Find ways to reproduce those bugs, and report them.
Bug reports go here, or get E-mailed to Sam. Follow this template when submitting bugs for maximum bug damage.
Consider this a way for you to find out if game testing really is the dream job that you think it is.
One request for Sam, though...publish the spec. It's hard enough to test a game with a spec...without one, testing is mostly guesswork...
...but then again, I haven't worked on a game with a spec yet...why start now?
December 18, 2006
December 17, 2006
I wouldn't say that I have messianic fantasies. I will say that I've been trying to fill a perceived hole.
Over the last decade, I've seen quality assurance get crammed lower and lower on the priority list. The tester week-to-production week ratio at has moved from a semi-reasonable 1:10 average to 1:35 and above at some studios. During that time, complexity has increased exponentially, and the industry has answered the challenge of testing this by throwing inexperienced bodies at the problem rather than approach it the way they approach every other discipline: with experienced, knowledgable people who are passionate about their work.
Designers, developers, artists, all disciplines have had someone to champion their cause...but not testing. The end result is that testing has turned into a faceless department filled with temps whose sole purpose is to pound on their controllers until something breaks and then hope that the development team doesn't ridicule their bugs and waive them outright.
Long story short (too late), quality assurance in this industry is broken. Nobody disputes that. I hoped to drive some positive change, and I've seen some change happen. It's been at the local level mostly, but it's a start.
Now I'm not going to be able to push for change from the inside for awhile. To be honest, I'm okay with that. But now people in games QA have to make a decision. Is the cause of reform worth pushing for or not? If it is, are you going to champion change in your own organization? Are you going to champion change in your area?
The industry can change for the better, but you have to make it happen. That's what I tried to do.
What will you do?
If quality is to improve in the products that the games industry creates, the way that we treat those responsible for maintaining the quality bar must improve as well. With that in mind, I present for your consideration "The Games Quality Assurance Bill of Rights."
1. You have the right to work no more than ten hours per day.
As a tester, your primary tools of the trade are your eyes and ears. Your powers of observation are crucial to your success as a tester. Studies have shown that after six hours, a person's ability to be observant begins to decline. In practice, any tester starts being more of a hindrance than a help after about ten hours of testing. Experience has shown that each hour spent by a tester after ten hours adds an additional 2.5 hours to the overall testing load to account for double-checking questionable and bad bugs and having to retest the area that was tested.
2. You have the right to at least one full day off per week.
Everyone needs at least one day off per week, be it to rest, attend religious services, do your laundry, pay bills, etc. Just because you're testing doesn't mean that your life outside of work stops. Your family misses you, your friends miss you...you need "you" time. Take it. It will help you recharge so you'll be more effective when you come back in.
3. You have the right to advance notice of expected overtime.
Testers need to have a life outside of work. You work to live, not live to work. However, there are times when extra hours are needed, be it for milestone testing or the final push to get a product out the door. I always tried to provide at least 48 hours notice of any extra hours coming up, and if I wasn't able to give that notice and the hours were required, I excused the extra hours for testers who had plans already.
4. You have the right to information about the game you are testing.
If we don't have information about what we're testing, if we're expected to test based on "feel" or "gut," we may as well just be monkeys with controllers. Testing isn't just throwing random input at a game until something breaks...it's a directed effort into the innards of the game trying to find the flaws that our customers are going to encounter so that they can be fixed. In order to have that directed effort, we need information. We need maps, behavior descriptions, system descriptions for systems that affect the user experience but are not exposed via UI elements, interaction charts...we need to know how the game is supposed to work so that when we are in the game, we'll know if something isn't working right.
5. You have the right to access your development team.
In this industry, we have a horrible habit of segregating test from the rest of the company. Builds aren't delivered, they're "thrown over the wall." There are times when text is not the ideal way to describe a bug. There are times when a tester should be able to go to a developer and say, "I've got this odd behavior...does this seem like a bug to you?" There are times when a tester should be able to go to a developer and ask, "I'm not sure how this feature is supposed to work...can you describe it for me?" Some people think that interaction like this is a waste of the programmer's time, but ask yourself this...would you rather spend two minutes talking to the person testing your feature, or a week handling "bad" bugs? Would you rather spend an hour talking to the tester who is going to be working on breaking your system and work out most of the major kinks before the system is even written, or four months rewriting your system after a major flaw is discovered by test?
6. You have the right to be evaluated on more than just your bug count.
A lot of QA managers evaluate their testers on bug count alone, but that doesn't paint an accurate picture of your performance as a tester, and can actually lead to an increase in counter-productive activities like bug padding, bug repetition, and skipping verifying repro steps in an effort to get the bug in faster. Activities such as test casing, bug triages, meetings with developers, and buddy testing reduce the amount of time that testers can use to find bugs, but still positively affect the quality of the project. In addition, not all bugs are created equal. Is one crash bug the same as a missing period in a dialog box? Also, not all areas will be as buggy as others. I used bug counts as an indicator of a potential problem, but only as an indicator that I need to dig deeper to see if there really is a problem.
7. You have the right to know why it was decided that your bugs would not be fixed.
In most companies, there is a triage process during the endgame where bugs "live or die." There may be times when a bug is just deemed too small to fix. Other times, fixing a bug may be seen as too risky because of the likelihood of other bugs being introduced by the fix. Other times, bugs may be offered up as trade in an effort to get other bugs fixed. Regardless of the decision, testers deserve the right to know why their bugs were waived. It's demoralizing to have a bug come back "won't fix," but having an understanding as to the process that led to that decision often helps.
8. You have the right to appeal a decision made about your bugs.
There have been many times when a bug was waived because it wasn't seen to meet a certain level of severity, even though the actual bug itself was significantly worse than people thought. Either the subject line of the bug didn't make it seem harsh enough, or the description came across as soft, or just the bug wasn't properly understood. There have been many times when a tester's appeal has caused a bug to be seen in a new light, and led to the bug being fixed, resulting in a higher-quality product.
9. You have the right to have your department decide how a feature should be tested.
Many people have a low opinion of testers. They believe that people with limited to no training and/or extremely low wages in comparison to themselves couldn't possibly understand the complications and/or ramifications of a system. What happens on occasion is that the means of how something should be tested will be passed down to the test manager or test lead as an "edict from on high." The main problem with this scenario is that the person deciding how the testing should be done has no experience with testing. It could be a coder who wants "the correct codepath" tested. It could be a middle-manager who wants to feel important. It could be someone who is trying to reduce the cost of testing a feature. Regardless of who, it is the right of the test department to decide how the feature should be tested. There is nothing wrong with taking the suggestion under advisement and use it as a guide towards one way of testing a feature, but if you are going to be held responsible for successfully testing a feature, you should at least be able to determine your own plan.
10. You have the right to be treated like a human being, both inside and outside your department.
Testers have to deal with the worst of all worlds. Their jobs are often temporary, low-paying and offer no benefits. As part of their jobs, they are forced to be the bearer of bad news. Their department is often the first to be blamed for any defect with any product, even if the issue was discovered and reported during testing. Starting from that, the job is already on a bit of a down note. It gets worse when you receive condescending comments from other departments, like "You're just a bunch of keyboard monkeys," or, "This system is so complex, I don't think you'd be able to understand it." There have been times when meals were provided for all employees working twelve or more hours except for the temp testers. (All of these examples are unfortunately real.) It can be just as bad between full-time testers and temporary testers. You are not your career and your worth is not defined by your profession. You are a human being, and you should be treated like one, whether you're full-time, contract on location, or outsourced in Shanghai.
Remember, we are all test, as bugs affect us all.
December 16, 2006
- CPL is going global again next year. CounterStrike 1.6 is the focus; $500k purse.
- CPL is going to be creating their own game. Tom Mustaine is running the studio behind it. Game is codenamed "Severity."
- CPL needs to test their PA and video playback systems before running anything. We saw Romero's talk, but didn't hear it.
We never had a vocal counterpoint to our critics from the ESA. Our critics were allowed to take center stage and put us on the defensive, forcing a twenty-fold increase in penalties from the ESRB and putting game developers in a self-censoring mode for the last year.
We had an online presence from bloggers, a single paper from a gentleman at MIT trying to debunk gaming myths, and more messageboard postings than you can shake a stick at, but the ESA has been so focused on showing who buys and plays games and trumpeting their legal victories and anti-piracy efforts that they've let everyone think of us as legally-protected hunter-killer-trainers rather than a form of entertainment. If the courts are our only allies, we are losing the war.
As developers and publishers, we haven't been much better. Our titles get targeted, and our public response is...nothing. It makes us look like guilty children cowering in the corner from the scolding we're receiving.
As an industry, we've taken a "wait and see" response because we keep seeing from the past that every generation has their strawman for its ills. However, if we don't start opening our mouths and fighting back, we aren't going to end up like the movie industry...embedded in the mainstream. We'll end up like Dungeons and Dragons...marginalized and mostly forgotten.
Long story short...if you want to protect gaming for the future, don't just type...open your fucking mouth.
December 15, 2006
I'm fairly certain that is going to be one of the things that Randy Pitchford is going to be talking about tomorrow evening during the CPL "Open Discussion" during the CPL Finals.
Of course, Tom Mustaine is going to be up there as well. I wonder what he's going to be saying?
(Edit, Sunday 12/17: Fixed typo. Thanks Chris.)
There are generally three price-point ranges that PC games tend to start out at. There is the premium price-point, which is $49.99-$59.99, the mid-range price-point, which is $29.99-$39.99, and the budget price-point, which is $19.99 and under. The three major price-point ranges have very unique sales curves.
The high-end games start high, sell heavy for the first month or two, then start having tiered price drops. If they sold well, they'll drop to mid-range and get reprinted, otherwise, they drop to budget for a sell-off. Most are gone from shelves after six months unless they're being reprinted.
Mid-range games tend to keep their price-point for a longer period of time, but they have a lower margin per unit. Generally, they'll keep their price-point for about four to six months before a price drop to the bargain price-point. Nine to ten months is usually the outside.
Games that start at the budget price-point rarely have that up-front "oomph." They're "investment titles." They have extremely low margins per unit, but they stay on shelves for an extremely long time. Most games with a budget price point sell slightly heavier up front, but then fall into a steady unit sales per week. In other words, they result in a steady income flow to even out a relatively unstable sales category.
As a result, the initial sales on a budget price-point won't "wow" anybody, but the game will stay on shelves for as long as the per-week sales are relatively level. As long as weekly sales worldwide are over 1,000 units a week, that title isn't going anywhere for at least a year. If the unit sales remain steady up to the end of the year, it will usually get repackaged in a cheaper jewel-case SKU for $9.99 and stick around for even longer. I can still go into Wal-Mart and Target and purchase budget price-point games that were released five years ago. The games are there because they still sell.
Now, does this mean that a certain title is going away? Not my place to say. I can say that I was doing some work related to it prior to my untimely departure, but not what you might think.
And with that, I wash my hands of the title in question. You want more information? Convince certain people that silence isn't a viable PR strategy.
December 14, 2006
This service pack has been anxiously awaited by many people, but:
- Vista support won't be contained in the service pack. For that, we're going to have to wait until Q1 2007. A public beta could be released tomorrow, but don't quote me on that.
- If you are developing against certain consoles using native code and not XNA, you'll have to hold off on installing the service pack until it is supported. (Same goes for previous iterations of said console and SP1 for previous iteration of the compiler...)
When the service pack is ready (supposedly, around 9am Pacific), you'll be able to download it here.
(Update 12/15, 8:55a: Well, it's a couple of hours early, but the service pack is up for download.)
December 13, 2006
Working on a game that uses DirectX? Still worrying about Win9x support (95/98/ME)?
Well, don't worry too much. Starting with the December 2006 DirectX SDK, portions of DirectX will no longer work on Win9x.
Support for Shader Model 1.x is going away as well without a compiler flag override.
The legacy tail is getting shorter and shorter.
When I accepted the job at Ritual, I wrote up a document I called the "Game QA Bill of Rights." It was a list of ten items that I had either heard about from friends at other game developers or experienced at Microsoft that I never wanted to experience again. I'll be posting that document later, but please keep in mind for a few minutes that I had an agenda when I came back into the game industry...I wanted to change it by example. I wanted to pull back the veil and show a glimpse inside the industry, as well as prod the industry into changing for the better.
For the most part, I feel that my efforts over the last two years have been a success. The conversations that needed to happen are happening. People are starting to realize that quality assurance isn't a department, it's a process that every department needs to participate in or the customer suffers. Companies are starting to see good quality assurance as an investment in their future. It's just unfortunate that it's going to be one to two years before customers will be able to benefit from what is going on.
I've pushed a lot of buttons over the last two years, and I expect to continue to over the next few years...but at least for the near future, I'm going to be doing so from outside the profession and even the industry.
When I sent out my resumes over the last week, my QA experience was at the forefront of my resume, but it wasn't what interested employers. It was my development experience that people were interested in. Developers wanted to bring QA into their development department, and so that's where I'm going to spread the quality gospel. Once I've rebuilt my warchest, I'll come back...and this time, I'll bring friends.
If it's any comfort, just think that he's gone into the afterlife and is currently causing Madeline Kahn to sing, "Oh, sweet mystery of life, I've found you..."
December 12, 2006
December 11, 2006
December 10, 2006
All consoles have three points where they could be considered next-gen milestones. The first milestone is when we see graphics that just weren't capable under the previous generation. The second milestone is when the extra power provides a gameplay experience that was simply impossible with the previous generation. Finally, there is the point where next-gen actually arrives, when the gameplay and the graphics meld in a single title. Let's look back at the Super Nintendo and use it as a basis for comparison.
At launch, there was PilotWings, which graphically was doing something that simply could not be done with the previous generation because of the new graphics chip in the SNES.
A short time later, there was Super Castlevania IV, which while it may have had a very odd visual style compared to the rest of the series, used the power of the console to allow new gameplay challenges (navigating a level that rotated), new player capabilities (diagonal attacks, swinging, etc.), and enemies that would have been impossible on previous platforms (an enemy growing as it was attacked, restricting the area available for the player to move around). However, there were many locations in the game where graphics weren't all they could be.
About two years later, we got StarFox. One of the first console games to provide true 3D graphics, StarFox used the power of the console to open entirely new avenues of gameplay: wingman protection missions, multiple routes through the game via difficulty level selection, ship damage that affected navigation and combat, a mixture of rail-based and free-roaming combat, and more.
So the question you need to ask yourself when deciding whether or not next-gen has arrived is which is important to you: the graphics, the gameplay or the fusion of the two. For me, Super Castlevania IV was the point where next-gen arrived for that generation because it let me do things that I wanted to do in previous games in the series, and the power of the platform let the developers do it.
What will it take for next-gen to arrive for you?
December 9, 2006
Interesting mix as well...do I stay in the industry in an SDET-style role (like the one I've unofficially had for the better part of my career), do I move to web development and try to integrate a QA process into one of the most dysfunctional development processes I've ever seen, do I take a contract-to-hire position writing no-brainer data entry apps, or do I take a chance on a position that pays better than any of them, but would require me to beg for help to move back to Utah so I can take it?
What sucks most is that this is the absolute worst time of the year to be unemployed. So many people go on vacation the latter half of December that if a person isn't employed during the first two weeks of the month, they're essentially fucked until early next year. And with my savings depleted after a year of doubled utility bills and significantly higher rent, that doesn't put me in a good place.
Personally, I want to stay in the industry, but in the end, it comes down to what is best for my family.
December 8, 2006
I've seen many anime music videos with great lipsync...but here's a human music video with great lipsync for the "Weird Al" fans.
Parts 4 and 5 of my Automating Games QA series are still under construction. Job search has taken priority.
Don't take silence to mean inactivity. There are many times when things cannot be said, either because things are not ready, or there may be contract or legal issues involved, or the news may just not be ready. Don't jump to conclusions just yet.
And now...I have to get to a job interview in Plano.
First, there are tools that test needs to test. Things like debug spew readers, memory trackers, fault simulation tools, DVD emulation suites...all of these allow testers to either see the state of the program or exercise certain functionality in the program.
Second, there are the tools that makes test easier or more efficient. In games, things that compare maps and show areas that have changed since the last iteration, or quick buttons to set up massive battles between unit types for balance testing, or directed dependency graphs derived from code changes since the last build...anything that lets test reduce the amount of time they set up for tests or reduces the surface area required for testing falls into this category.
Third, there are the tools that make test cheaper. Automated BVT's and other types of automated testing, continuous integration servers...anything that keeps test testing instead of stuck on a bad build or lets a machine do the work of a tester falls into this area.
Finally, there are the tools that free test up to focus on finding harder bugs. Content testing tools like level-leak checkers, code testing like unit tests and integration tests, and even spell-check for help documentation...anything that keeps a bug from getting to the testers in the first place falls into this area.
Where does your company invest its time?
December 7, 2006
This is a difficult topic to talk about. On one side, the RIAA member companies are doing exactly what their stockholders expect them to do...find a way to increase the amount of money coming in so that they can maintain or increase their stock price.
On the flipside, because of the sheer number of exemptions and chargebacks in the music industry, artists rarely see dime one beyond their advances until they've sold over three million units.
In the middle is the public domain. Labels and artists both want to increase the maximum length of copyright protection because both want to make more money (or in the case of most artists, some money) off of the same item, but at the same time, the most recent increase in copyright protection combined with the sheer quantity of samples being used in media is increasing costs for artists and accounting complexity for labels.
This issue is beginning to creep into game development as game costs increase. Namco has said that it will take an average sell-through of 500,000 units for a next-generation title to break even, in part because of increased development costs (estimated to be approximately $5 million). Compare that to the last generation, where 350,000 to 400,000 units was generally required to recoup publisher development costs of $2 million.
More costs are getting pushed back to the developer as well. Every time you contact publisher technical support, it takes away the royalties on as many as seven copies from the developer. Staff used to fill in holes are being charged back against royalties at rates as high as $60 an hour. If you fail certification more than twice, the cost for further certifications is getting charged back.
With the percentage of games breaking even per year decreasing slowly over the last few years, you're going to start seeing some fairly interesting shifts in the industry in an effort to control costs. I'm actually expecting a lot of the majors to gradually shift to a hub-and-spoke model over a studio model so that certain departments (audio, animation, etc.) can benefit from economies of scale. We're already seeing the rise of the designer again (Wright, Jaffe, etc.), where the designer is being pushed more than the team itself.
Moves like this are beginning to simulate a shift to more of a movie industry model. As technology and formats begin to stabilize, it is very possible that the shift will become even more dramatic. Outsourcing for asset production is already at record highs in this industry, with some designers going all out in support of outsourcing.
So the question is fairly simple. When the industry restructures, will we be looking at several ILM's, or several BMG's? It's unfortunate that the answer is not as simple...
You guys have a kick-ass staff, and anybody would be honored to work for you, myself included.
It was great to see [censored], [censored] and [censored] again and to share war stories with veterans of both digital and physical wars.
Hope to hear back from you soon.
December 6, 2006
Most software testing is done in a clean-room environment. Does "X" work on a clean install of Windows? For games, driver versions are generally tested as well because of the number of issues fixed and caused by various driver versions.
However, there are many third-party applications that interfere with applications running. Antivirus software can modify files by appending data to the end, interfering with CRC checks. Uninstallation utilities can cause temporary files to get locked and cause massive slowdown during unpacks. Firewall software can cause loopback addresses to wig out. Packet burning software can interfere with copy-protection checks.
We can't test all the combinations. I know of about thirty different anti-virus solutions, eighty different uninstallers, sixty different personal firewalls, a few dozen packet burning programs. It adds up.
So again, we return to what most people are going to have. Unfortunately, not a lot of people have antivirus. Several testing locations test McAfee, Norton and TrendMicro, but not all. Very few places test interactions with uninstallation utilities. Because of the number of firewalls and various interfaces, most places stick with testing the built-in Windows Firewall at most. And packet burning software...well, at least until now, we haven't had to worry about it being in the operating system...of course, with Vista...
These are just four of the various permutations of third-party software on machines that hook into the operating system. I'm sure that there are hundreds more.
As testers, we shoot to be as complete as we can for our product, and we shoot to ensure that in our supported environment, we work well. When we find a problem with third-party software, we'll often contact the vendor to try to find a workaround on our end or a fix on their end. But even so, our primary responsibility is to our software and our supported environment.
So next time your custom-written firewall rules cause "X" to crash on loading, or your uninstaller daemon causes "Y" to take four or more hours to load, or the browser plugin you installed to get free viewings of "Debbie Does Dallas" causes every single sound effect to be replaced with an obscene version of "Oh Come All Ye Faithful," remember "PEBSAO" when you're calling support.
The first one really liked my attitude, felt I had a strong grasp on the SDLC, but then started asking me lots of questions related to Java. I said, "Java? I thought this was a C# position." He said it was a Java position. I pulled out my printout describing the position and he pulled out my resume. Every reference on my resume that said C# had been changed to read Java...but the job opening I had in my hand said C#. After a curt phonecall from both of us, both of us terminated our relationship with the recruiter.
The second one really liked my attitude, but was wondering what games experience brought to "the real world." I asked him how many data records his office processed in an average day, and he estimated 2 million. So I told him that I come from an industry where we process that many records thirty times a second on consumer-level hardware; that the user can and often does change and distort that information; that we have intelligent agents acting on the data in real-time and interacting with the user; and we do all of that with the user experience in mind.
I have a follow-up interview on Monday.
December 5, 2006
If software development is a religion, I guess that means that I'm in a metaphorical exile after nailing a list of grievances to the chapel door. We are stuck in battle between those who believe that programs spring whole from the minds of their creators, and those who believe that programs evolve through trial and error.
Is there a place in this world for a Quality Assurance Evangelist?
I managed to test a AAA-title with a QA department one-fourth the size of the one on "Bookworm Adventures" and with a five-figure budget. I managed to handle all support calls and forum posts for the first three days after launch by myself, and by the time I left, had total support contacts across all methods of contact under 1% of units sold. On an unannounced project that we were working on with a publisher, our milestones were never rejected, in part because I looked at the milestone deliverable lists we received from the eyes of the publisher, not the letter of the contract. Ritual was also the first developer to publish any sort of Vista compatibility lists for its legacy catalog.
I did a lot outside of my department as well. I wrote two full manuals. I wrote a patch for "SiN 1.12" to add functionality that we promised to the community. (Whether it will be released or not isn't up to me anymore.)
I did a lot on this blog as well to help out on the PR front, although sometimes the accompanying flare-up was so much that I didn't get the backup I was supposed to get. I was asked to go forward with the piracy stats, and it succeeded in getting people talking about piracy. Unfortunately, the pirates were louder and Something Awful decided to be the assholes they normally are, so I was essentially hung out to dry on that one. It's too bad, too, because if you look at some other employee blogs, you'll see that Ritual's employees have never been ones to shy away from controversy. You'll see discussions of everything from asset theft by former employees to some items that shouldn't be discussed on a (relatively) family-friendly blog.
Of course, there was also "YAUSB." I worked hand-in-hand with PR on that one to get the message out.
If I had to say that I had one success while at Ritual, it was that I added a human face not only to the company, but to quality assurance as well. I tried to keep a conversational tone in all of my communications because I believe that talking to someone and teaching them about a topic is more effective than just talking at someone.
Oh, well...such is life. Back to looking...
December 4, 2006
Yes, I got a little internal flak for the Sony QA debacle. The flak consisted of a total of two (2) E-mails. Most of the edits to the original post were made prior to those E-mails because of people taking select pieces out of context and reading their own bias into them. Unfortunately, some of my wording was...colorful enough that people could make it mean almost anything.
Yes, I am no longer with Ritual Entertainment.
One has nothing to do with the other. More information later this week. I'd say more now, but I made an agreement with someone inside Ritual that I respect greatly, and I honor my agreements.
The other portions of the Automating Games QA series will be posted over the next two weeks.
If you have questions regarding the change, search the site for the answer.
Also, I made a mistake last week. I pointed to mistakes that someone else was making instead of pointing the camera at my own department and showing the mistakes that we made. I also pointed out a negative example over a positive example.
Today, I'm going to attempt to rectify both.
December 3, 2006
for computer literate
The mind craves variety. (By the way, if you used to read Sheldon on uComics, this is his new home...)
Why reading your employment contract is very important (bottom of page 1). (And at least according to the kids that I've spoken to, Bratz aren't succeeding because of their attitudes, but because they look more like kids than Barbie dolls do, so they're easier to relate to.)
Hrm...$59.99 game with all content required to be unlocked via microtransactions, or free download with limited content and the remaining content required to be unlocked via microtransactions? Good choice.