[SiN] Patch Live, Bug Post-Mortem #1

The first patch for "SiN Episodes: Emergence" is now live.

I've also noticed a few posts around the blogosphere calling my competency into question because of the Personal Challenge glitch. Frankly, I can't blame them. I played the game once we triggered what they were running into, and if I had played that, I know I'd be cursing up one side and down the other.

So, what caused this glitch, and why was it missed? The cause and the miss are actually related. I can't go into complete details, but I'll summarize as best I can.

We had over 100 playtests for this first episode. During the playtests, we had monitoring software in the background running that measured when the players had passed certain "pacing" triggers in the game. This software let us analyze trends, and say things like "The average player went through this section in 60 seconds." Later in development, these triggers were updated with the timing data itself. Finally, every time a trigger was passed, it would delete itself so that it wouldn't be triggered again. This last part is important.

What I'm about to say is a gross simplification, but the Personal Challenge System is based off of advice from software components called "advisors." The "Challenge" slider that you adjust at the beginning of the game tells these advisors what would make them happy. One advisor watches your health, one watches your weapons and ammo, one watches your accuracy, etc. We had one advisor that watched these triggers. If you were going through the triggers faster than average, he'd get unhappy and recommend making the game harder.

Before ship, we were trying to get a base implementation of "Arena Mode" in place for E3, as well as get the game ready to ship. We were fixing the last few bugs, etc. Our primary focus was the single player game, not Arena Mode. However, one of the designers working on Arena Mode needed an extra spawn flag on the pacing trigger that would let him make an indestructible trigger. The spawn flag was added rather than creating a new trigger type. The issue there: we had already preloaded all of the maps. There were two triggers that because of the way they were created had the bits set already.

Second, these triggers are very small. The two in question were less than 4 inches wide, and placed in locations that users normally just blow through.

Finally, the code change for this went in just a couple of days before we uploaded the final files to Valve.

This is still a gross simplification, but every time you triggered a pacing trigger, the advisor would get happier or angrier. The issue with these two triggers was that if you just walked through them, the advisor would get just a little miffed. But if you stayed in the trigger for any duration of time, he'd get more and more upset every frame of server time. He'd keep thinking, "Hey, you were supposed to take three minutes to get through this trigger, and you got through in .1 second! I'm mad!" and he'd get even angrier as it went on.

The end result: people who backtracked through these triggers multiple times or lingered in this one section had this one advisor who was extremely pissed. It would have taken almost a day and a half to get him un-pissed.

So how did we miss this? This definitely was a miss. I can't and won't argue that one iota. During the end-game, we try to focus our testing on bug fixes that are being put in. We pour through check-in mails and regress any issues that come up. I saw the checkin that said, "Dynamic difficulty triggers now have a spawn flag that allows you to make them not destroy themselves on triggering," spot checked triggers in the game and in Hammer to see if they were being destroyed, verified they were, and considered it a moot issue.

Obviously, that was a mistake. So, we're working with the team to try to make sure nothing like this happens again. We can't change the past, but we can learn from it.