May 31, 2006

[Personal] The Boundaries Have Been Breached

I've been a pretty open person in the forums and in this blog. I try my best to help people, to answer people's questions, and in general to be a decent member of the digital communities that I exist in.

However, there are some lines that I'm going to have to draw based off of circumstances that have occured over the last forty-eight hours.

I will not be responding to any work-related E-mails to my home account. If you have a question related to work, please go to the forums at and post there.

I will continue to answer questions via Messenger, but on a limited basis only, and only when I'm set to "Online," not "Busy" or "Away." I have work to do, and the constant IM pop-ups tend to reduce my productivity.

Finally, visits to my home are strictly out-of-bounds. To the gentleman that I passed this morning who was looking for my house, I hope you didn't find it.

I want to be of as much help as possible, but I have to put these limits out there to not only ensure a base level of service, but to also protect my family.

May 30, 2006

[Games] Haiku Review: Tomb Raider: Legend (Xbox 360)

If the ending to
Halo 2 made you angry
Do not play this game

Great game, but damn, the ending pissed me off as much as the ending to "XIII."

May 29, 2006

[Personal] The Mark Of Shame

I remember every bug I ever let slip through without my knowledge. Bugs like allowing a malformed player-created image to crash "Links Extreme," or a weird multiplayer handshake bug that bit a couple of guys with "Microsoft Golf 2001 Edition."

At least with those, both involved unsupported behavior.

For "SiN Episodes," the personal challenge bug was perhaps the worst bug that I've ever let slip, I've ordered myself a wearable "mark of shame." After all, anytime a T-shirt is made available advertising not only the existence of, but the nature of your bug, it's a valid "mark of shame."

I've already instituted some policies to stop a bug like this from getting out again, but subtle reminders like this never hurt.

May 28, 2006

[Games] Between a blockhead and a hard case...

Last week, I spent two days training Ritual employees on the ins and outs of the various ratings boards across the globe.

What drove the training was that there is a lot of hatred in this industry towards ratings boards, including the ESRB, but not a lot of understanding.

On one side, we've got groups who see scandals like "Hot Coffee" and "Hot Mead" and think that the ESRB isn't doing a good job of controlling the content inside video games.

On the other side, we've got developers who see the ESRB as their enemy, with the looming threat of an "AO" rating or severe descriptors ready to force them to remove portions of their artistic vision and/or eliminate a massive chunk of sales.

To be honest, I see both sides, but I don't think that anyone really has stopped to look at things from the ESRB's point of view.

The ESRB was created as an independent ratings board in an effort to prevent the government from stepping in and ratings games themselves. In that way, the ESRB is identical to the MPAA.

There are other similarities to the MPAA. There is an ad code which says how you can and cannot advertise your titles, as well as limitations on marketing for certain ratings. The ESRB owns their rating mark and merely licenses it out, so if you violate the terms, they can force a recall or resticker of the title. The people who do the rating itself have no professional association with the industry. And finally, the ESRB is funded by the submission process.

How the rating system works is that if I want to rate my game, I get a submission packet from the ESRB. Based off of my knowledge of the content in my game, I fill out the submission packet. I also create some extra exhibits as necessary, be it a soundtrack CD and lyric sheet, the voice-over script, a list of cheat codes, etc. Finally, I create a video cassette that shows normal gameplay, as well as the worst items from the submission packet. Everything on the tape must be in the submission packet somewhere. I assemble all of this, pay a fee to the ESRB, and in a couple of weeks, I get my rating and descriptors. Outside of those two pieces of information, I receive no information as to why the ESRB granted that rating. After the product ships, I'm required to provide copies of the game to the ESRB. They double-check the rating at that time.

Movies work in a similar fashion. Ratings are granted several months in advance based off of a roughly-edited version of the movie, but movies are often edited up to the week before the movie initially screens.

In both cases, if something that would materially change the rating is introduced, a resubmission is required. In the film industry, "Snakes on a Plane" is a good example. They were initially shooting for a "PG-13" rating, but because fans were demanding Samuel L. Jackson to use some more...ahem...severe profanity, they added the scenes and the movie was upgraded to an "R" rating.

Likewise, if in "SiN Episodes: Emergence," we decided that we wanted Elexis to be nude, we'd have to resubmit because we don't have a "Nudity" descriptor.

I guess the point is that while everyone in this industry wants a say in how the ESRB rates our titles, we can't have a say. Our "say" consists of either accepting the rating, or making changes and resubmitting. If we were able to do any more than that, the ESRB would lose their biggest defense...that they are independent. They have to remain seperate, lodged between blockheads who would ban or regulate us, and the hard cases who believe that rating is equivalent to censorship.

May 22, 2006

[Sin] I love this community...

We've been getting a lot of random crash dumps from people in the "SiN Episodes" community. When I say "random," I mean "random." There was no commonality to the dumps whatsoever. The only common thread for most were that they were running a dual-core processor.

Fortunately, one of our forum-goers found this trinket about dual-core processors, which links to a fix.

Thanks, MagicFire. Credit goes where credit is due.

May 21, 2006

[Personal] The Month Ahead...

This next month is going to be insane, so I'm not sure what my posting schedule is going to be.

Besides handling tech support on the second launch for "SiN Episodes" in Europe this next week, we've also got the merged codebase getting closer to being ready for testing (we need to do a lot of testing before we release it), some other internal projects that require testing efforts, some additional bugfixes for some bugs that either were reproduced after the patch or weren't reported until the patch, as well as Episode 2 feedback collection and review.


May 19, 2006

[SiN] Tech Support

Picture of the Tech Support Wiki Page I've been trying to keep on top of the problems that have been facing our community. Aside from some minor lingering issues, the vast majority of people who were having problems seem to have had them fixed by the patch.

There's even been a lull this afternoon, which has been nice. I've been using that time capturing solutions to common issues to our internal wiki for quick reference and retrieval.

This wiki page has two major purposes. First off, it gives us a common area to keep solutions and problems. If a solution that works in v1.1 won't work in v1.2, we can easily change it in one place without any real problems.

The second function is to make it so that if necessary, we can hand support duties off to other people for brief periods. I've been working for nineteen days straight, and I know I'm going to need some downtime soon.

All in all, this has been a pretty smooth launch. I remember the aftermath of some of the "Links" launches...ugh, the stuff that nightmares are made of. I mean, if one person can handle all the tech support duties for the first five days of a product's lifetime, how bad can the launch have really been?

May 18, 2006

[Request] Training Materials

I'm doing some worldwide ratings training next week, and I'm looking for a variety of screenshots from the ending cinematics from "Dead or Alive 4" to use as visual aids.

Any idea where I can find some?

[SiN] Bug Postmortem #2

People seem surprised that I’m actually being open about bugs on this blog. I can understand why my being forthright about these issues would cause some confusion. In this industry, “bugs” is a four-letter word in more than one way.

But one of the big things that I have been an advocate of is information sharing. I believe that the more lessons we can learn from others, the less likely we are to repeat their mistakes. My goal throughout this process has been to make sure that everything I do is as transparent as possible. With that disclosure out of the way, let’s postmortem another bug.

The evening we released, we started hearing reports that the in-engine cinematic at the end of the second U4 Labs level wouldn’t progress for some people. This was extremely odd, because the only time we had seen this happen was back before Radek wasn’t able to walk. The bug hadn’t surfaced in over seven months of testing, and never surfaced in any of our playtests.

While many things could have caused this bug, we narrowed it down to three major causes, two of which were related. To talk about the causes, we need to first discuss abusive players.

Every game out there has abusive players. These are guys who will do everything they can to try to exploit the game; to try to beat the design of the game rather than the game itself. It’s like a guy playing volleyball who intentionally spikes the ball at the head of the slowest person on the other team in an attempt to get it to bounce off the guy’s head back into his area so he can try to keep the round going. It’s completely legal and within the rules, but a pretty piss-poor way to play.

Back to the game. During testing, we set aside time to essentially act like assholes towards the levels and do everything possible to break and/or exploit them. We had exploits ranging from being able to club mission-critical NPC’s to death using a cardboard box to being able to jump on top of the lighthouse to being able to flip the car up into the upper stories of Highrise and have it still be usable.

The vast majority of the exploits were easily fixable, but for the rest, we sat down and looked at the amount of effort that was required to trigger the exploit. Oh, this one can be done in two minutes easily? Must fix. This one requires ten minutes of split-second jumping and exploiting the physics push of four primed grenades? Fix if we have time. This one requires seventy-five minutes of preparation to break? No.

These last ones were the worst ones. It is possible to break interactive cinematics in pretty much every single game that has been released ever, especially if you really really really really really really try to. We decided that if someone was intentionally trying to break the game by doing a bunch of stupid shit, we’d let them. After all, this part was a single-player game, and if someone really wanted to trap themselves, why should we stop them? All they’d be affecting is their own experience. If someone wants to smash themselves in the genitals with a pickaxe while singing “O Come All Ye Faithful,” who are we to decide what is fun for them?

This led to cause number one: abusive players. We got a lot of save games from people related to this issue, and the number one cause that we saw was people taking tons of U4 storage containers up into the crane control room and dropping them there. Radek and his grunts had to be able to go through the room. If he couldn’t, the scene couldn’t progress. We timed it, and it took us about 20 minutes to get every single U4 container in that area into the room.

Unfortunately, we missed cause number two: physics. We received one save game where a cardboard box from inside the room shot over, collapsed, but collapsed in such a way that the door itself was jammed shut. We received another where one of the larger propane tanks had flown in and jammed the door. In both cases, the player wasn’t trying to break the sequence, but a confluence of one-in-a-billion events had caused them to be stuck in an 8x10 room unless they replayed the submarine area. Not good.

The third cause was odd to say the least though. For some people, Radek would never leave. This was a bug we had never seen. We had no idea what was causing it. We had two reviewers get stuck with it as well, which was discouraging to say the least. The only commonality we were able to find was that all of the people who were affected by the issue had preloaded the game.

There was a drastic solution that worked, but wasn’t really practical: deleting the ClientSettings.blob file, all "SiN Episodes" GCF’s, and redownloading the game. So, as part of the map recompile, we updated the nodegraph again and sent it down as well so everyone would get the update. Much better to send down a few kilobytes than force the redownload of several gigabytes.

Our lessons learned from this one: 1) If an abusive player can exploit it, it can also hammer an innocent. 2) There are some bugs that you will never see until it gets out into the real world, and you have to go out of your way to diagnose and fix those.

May 17, 2006

[SiN] Patch Live, Bug Post-Mortem #1

The first patch for "SiN Episodes: Emergence" is now live.

I've also noticed a few posts around the blogosphere calling my competency into question because of the Personal Challenge glitch. Frankly, I can't blame them. I played the game once we triggered what they were running into, and if I had played that, I know I'd be cursing up one side and down the other.

So, what caused this glitch, and why was it missed? The cause and the miss are actually related. I can't go into complete details, but I'll summarize as best I can.

We had over 100 playtests for this first episode. During the playtests, we had monitoring software in the background running that measured when the players had passed certain "pacing" triggers in the game. This software let us analyze trends, and say things like "The average player went through this section in 60 seconds." Later in development, these triggers were updated with the timing data itself. Finally, every time a trigger was passed, it would delete itself so that it wouldn't be triggered again. This last part is important.

What I'm about to say is a gross simplification, but the Personal Challenge System is based off of advice from software components called "advisors." The "Challenge" slider that you adjust at the beginning of the game tells these advisors what would make them happy. One advisor watches your health, one watches your weapons and ammo, one watches your accuracy, etc. We had one advisor that watched these triggers. If you were going through the triggers faster than average, he'd get unhappy and recommend making the game harder.

Before ship, we were trying to get a base implementation of "Arena Mode" in place for E3, as well as get the game ready to ship. We were fixing the last few bugs, etc. Our primary focus was the single player game, not Arena Mode. However, one of the designers working on Arena Mode needed an extra spawn flag on the pacing trigger that would let him make an indestructible trigger. The spawn flag was added rather than creating a new trigger type. The issue there: we had already preloaded all of the maps. There were two triggers that because of the way they were created had the bits set already.

Second, these triggers are very small. The two in question were less than 4 inches wide, and placed in locations that users normally just blow through.

Finally, the code change for this went in just a couple of days before we uploaded the final files to Valve.

This is still a gross simplification, but every time you triggered a pacing trigger, the advisor would get happier or angrier. The issue with these two triggers was that if you just walked through them, the advisor would get just a little miffed. But if you stayed in the trigger for any duration of time, he'd get more and more upset every frame of server time. He'd keep thinking, "Hey, you were supposed to take three minutes to get through this trigger, and you got through in .1 second! I'm mad!" and he'd get even angrier as it went on.

The end result: people who backtracked through these triggers multiple times or lingered in this one section had this one advisor who was extremely pissed. It would have taken almost a day and a half to get him un-pissed.

So how did we miss this? This definitely was a miss. I can't and won't argue that one iota. During the end-game, we try to focus our testing on bug fixes that are being put in. We pour through check-in mails and regress any issues that come up. I saw the checkin that said, "Dynamic difficulty triggers now have a spawn flag that allows you to make them not destroy themselves on triggering," spot checked triggers in the game and in Hammer to see if they were being destroyed, verified they were, and considered it a moot issue.

Obviously, that was a mistake. So, we're working with the team to try to make sure nothing like this happens again. We can't change the past, but we can learn from it.

May 15, 2006

[DRM] Annoyed Beyond Recognition

"Whose Line Is It Anyway?" is a bit of an evening tradition for me. We have it set up to record on ABC Family every night, and when I get home from work, we sit down and watch the episodes together. We've been doing that every night for the last eighteen months...except for tonight.

I got home after a long day of patch testing, sat down, and what do I see on the TV?

ABC Family won't let me record 'Whose Line'!
This is a twenty-four hour world. Not everyone works nine to five. It's bad enough that I can't watch episodes I used to record on other machines in my house due to the DRM, but the fact that I can't record at all anymore is frustrating to the point of insanity.

I watch television. I am not a criminal. Quit treating me like one.

May 14, 2006

[Personal] T-3

We're shooting for the Wednesday patch window for "SiN Episodes." The preliminary patch I tested today had most of the major fixes in place. A couple of minor fixes weren't working right, but we can iron out the kinks in those tomorrow.

Three of the bugs were actually pretty interesting, and I'm thinking about doing some "anatomy of a bug" posts after the patch goes live.

The first interesting bug (although a nightmare for those who were living it) was in certain unique circumstances, the Personal Challenge System would amp the difficulty up to previously unseen heights.

The second bug is very interesting to me, and that's the crash bug that about 1% of our users are experiencing going into U4 Lab 02. Given that the bug was introduced back in January, and we had done three full config passes since then, the lessons learned tracking this one down are ones that I'm definitely going to be applying going forward.

The third bug also takes place in U4 Lab 02, and it's Radek not triggering Elexis. This one is the strangest of the bunch, because the symptoms and solution have been different for almost every person who has reported it, but the end solution was pretty much the same. This bug is also going to have a profound effect on how we test levels in future episodes.

While several other bugs have come up, these are the three that fascinate me both from a clinical as well as a professional sense. So until the patch goes live...

May 13, 2006

[Anime, Pop Culture] Cosby Bebop

(via ShackNews)

I never missed "The Cosby Show" when I was growing up, and "Cowboy Bebop" was an animated masterpiece with striking imagery and a killer soundtrack.

So it's with no small consternation that I look at this and giggle uncontrollably.

[Personal] The 360 Cometh...

When I got off of work today from working on the patch for "SiN Episodes: Emergence," I stopped by EBGames and dropped almost $700 on something I'm not going to really use until Thursday...I bought an Xbox 360.

Being tech support is stressful. Working on a patch with a limited timeframe for release (Wednesday is our target, so we have to have the patch really done by Monday afternoon so we can send it up) is even more stressful.

So, I picked up an Xbox 360, an extra controller, 1600 Microsoft Points, "Dead or Alive 4," "Perfect Dark: Zero," a copy of "SiN Episodes" (hey, I bought the third copy on Steam, but I wanted a box as well), and a few other miscellaneous items...they're going to be my reward on Thursday for making it through this stressful period.

That way, once the patch is out, I can hopefully take a day off (working non-stop for two weeks tends to wear on one) and beat some shit up.

May 12, 2006

[TV] CSI Web Puzzle

By now, I'm sure that a lot of you have seen this web clue puzzle for the season finale of "CSI."

I've only found a few "interactive" clues in the picture...
  • Hovering over the backwards "PLAY" replaces the picture with the word "DEAD."
  • Hovering over Sara's picture causes her head to turn.
  • Hovering over Grissom's head causes him to turn to look at the body, and causes a crack in the window in the door in the back to vanish. It looks like he has a piece of paper in his left pocket.
  • Hovering over one of the casinos in the background causes that casino to vanish.
So what else am I missing from the picture besides the person being shot looking like Brass?

(Sorry, I needed a brief break from tracking down bug repros...)

May 11, 2006

[Personal] Whew...

Nearly 300 posts in 18 hours across multiple sites. My fingers and eyes hurt. Need sleep.

[Piracy] SiN: Ep (Not Work Safe)

[ temper on ]

Evidently, this lame-ass pirate group going by the handle of "PROViSiON" decided to dedicate some hours so that episode one of "SiN Episodes" would be what is known as a "0-day warez release." Fucking smacktard pieces of shit.

The game is $20. $20. All this industry has been hearing from fuckers like this is "Oh, drop the price and we won't pirate because we poor people won't have to." So, we self-fund the game entirely, drop the price to something to a level where we stand a chance of breaking even, and what do you know! They pirate our game anyway!

So if you know any members of this "PROViSiON" group, do me a favor...kick them in the fucking nuts and piss on their face. When they ask why, just say, "Well, normally I have to go to a fetish club and spend $20 to do that to someone, but you walked by and I figured, 'What the hell, I'll do it for free.'"


[ temper off ]

By the way, if anyone who downloads the warez version comes to one of the sites and asks for support and I find out that you warezed the game, just be warned that you haven't seen temper yet.

May 8, 2006

[Sin] Just under 48 hours to go...

At 12:00 noon CST on Wednesday, "SiN Episodes: Emergence" is going to be unlocked for the world. To say that I'm anxious and nervous is an understatement and a half.

I have faith in my abilities as a tester. I have faith in the ability of my team. But the real testing starts once the game is in the hands of the consumer. You never know what real users are going to do until they actually have their hands on the product.

Sure, you may have spent hours launching DUAG rounds into concrete ensuring that the penetration depth was correct, but a consumer will find the one area with a hidden material inside that prevents the proper amount of penetration.

Sure, you have have thought you sealed all of your map exploits, but a consumer will discover that if he stacks nineteen trash cans on end and bunny-hops in a particular pattern, that he can get outside the world.

So I'm nervous, but I'm excited as well. It's been sixteen months coming, but we'll finally get to see consumers having fun with our game.

May 4, 2006

[Miscellaneous] Spammers and Shaders

Two small items... one to appease me, one to potentially help a long-time visitor and friend.

First, there are many ways to get me to visit a site. Referral spam isn't one of them.
Casino site referral spam

Second, if you are someone who uses Managed DirectX and are having a hard time wrapping your head around shaders, The ZMan is going to be doing a webcast for Microsoft on writing shaders and using shaders with managed code. Should be a good time for all. If you are interested, you can go here to register for the DirectX webcast.

[Industry] ESRB Examples (May not be work-safe)

I'm trying to get approval to post the ESRB submission materials I used for the resubmission of "SiN," but in light of the "Oblivion" re-rating, I'm going over my submission materials, trying to double-check and make sure I didn't miss anything.

Violence was pretty straightforward for "SiN," but the "Sexuality" section was really funky to write, because you never know what people are going to think is suggestive or sexual in nature.

Some items are fairly straightforward...
Elexis Sinclaire, your main adversary through the game, has overly exaggerated breasts.

In the Mansion level, Elexis has turrets that look like giant metal penises. She also has a drawer in her bedroom that contains a non-interactive picture of marital aids.
Others, not so much...
In the end credits, there is a picture of one of the company owners kissing the boot of a woman dressed as Elexis. The same credit slide shows one of the other company owners taking a nap with an inflatable sheep.
Others only make sense if you can read badly pixelated text...
There is a milk carton with the badly pixelated text "100% BULL" above the word "MILK."
...or if you know what is supposed to be being said...
During the cutscene between Chapter 5 and Chapter 6, Blade is woozy from being beaten up. Elexis is taunting him, and he replies with slurred speech, "Bit meh flock ouee fargin' bit," but people can infer the actual meaning of "Bite my cock, you fucking bitch."
"Language" was another interesting one to write. I disclosed everything from...
We have one texture with the words "Fuck You" written in American Sign Language on a posting on a bulletin board.
During our opening cutscene, two criminals are talking about a policeman that they shot, saying "That's nothin' compared to last week. I shot this cop in the head…he never even saw it coming."
If you try to open a closed port-a-potty door, a voice can say, "Hey, I'm in here, wait your turn asshole."
So, for those of you who read my blog who have created these packets before? What's the strangest thing you've ever disclosed to the ESRB? My strangest:
We have several warning signs scattered around the world showing stick figures being injured.

May 3, 2006

[Industry] Take Two Strikes Out Again

So, let me see if I've got this right. Take Two first drives game ratings into the spotlight with hidden sexual content in "Grand Theft Auto: San Andreas." Then, about a year later, they do it again with "The Elder Scrolls: Oblivion?" And now Bethesda is using the same excuse that Rockstar did, saying that without using a mod, the content couldn't be accessed.


The ESRB has been telling people since last year that they were going to be rating everything on the disc, whether it could be accessed during gameplay normally or not. When in doubt, disclose, edit or remove. That's it. There are no other choices. Blocking off or covering up used to be a viable strategy, but it isn't anymore.

Bah...I don't have the energy right now to worry about this shit...