March 30, 2015

The Failures Of Binary Formats In Development

One thing I've wanted to do for the last few weeks is dive back into game development.  While I was back at Amazon, I could do game dev work that I had grandfathered in, but I couldn't technically do any new game dev work.  Fortunately, Netflix lets me do pretty much whatever I want outside the office, provided it isn't creating a new video streaming service.

With that said, I needed a solid project to drive me forward.  I had licenses for Unreal Engine 4, Unity 4/5, and Leadwerks 3.x, tons of assets I had been collecting and licensing over the last few years, and some larger projects in mind, but I wanted something small and contained...something where I could spend a week or two working on the project and have something done.  I wanted something completable.

While I was browsing through my assets looking for a little inspiration, I found quake_map_source.zip.  Basically, John Romero had released the original .map files for Quake v1.06 back in 2006.  I started remembering back when Tenebrae was released, people were going in and manually placing lights to try to get the effects, and here I am with the source files.  Why not see if I could get Quake maps functioning in these other engines?  And thus, a project was born.  I wouldn't be able to release a full project because that would virally taint the engines with the GPL, but I could release code and tools to let people recreate it themselves.

I decided to start with Leadwerks.  I had backed the v3 engine on Kickstarter, and Josh had been nice enough to release the runtime code to the Map class loader at my request.

The Leadwerks map class is a binary chunked format.  The first 32 bytes of the file include a magic number to identify it as a Leadwerks scene file (SCEN), a version number, and then offsets to where you will find each chunk.  Dealing with a chunked file is fairly easy to do:

  1. Serialize each chunk to an internal buffer
  2. Write out an empty directory chunk up front
  3. For each buffer that you want to write out...
    1. Store the current file location
    2. Write out the chunk
  4. Once you're done writing out all the chunks, seek back to the beginning and write the offsets for each chunk into the file

The first item that needed to be done was to write out an empty map file.  If I can't write out an empty file, I don't stand a chance of getting the rest done.

There are two chunks that are critical to working with a Leadwerks file: the String Table and the Map Info chunks.  The String Table is where strings are supposed to be stored, and they'll be referred to by IDs within the Leadwerks file.  The Map Info table has general information about the map.

Let's start with the Map Info chunk.  If you have a Leadwerks license, you can download the file in the above-linked post and follow along.  Please be aware that after a lot of headaches, I rolled back to map version 22 for what I was going to try to export.

Starting on line 352, it wants the number and resolution of light maps.  That's easy...I don't have any.  Two ints out.

Line 408 wants ambient lighting.  Four 1.0f values.

Line 413 wants a map size.  Inspection of the UI shows valid values of 1024, 2048, and 4096.  1024 it is.

Line 414 wants gravity.  The UI shows a positive value, but the value is stored in the map file as a negative value, so -25 is serialized.

Line 424 looks like wasted space, and I just serialized out 0 and 0f depending on the field, but we'll come back to this.

Now for the comment on 438:
// This just uses the regular string commands, not the fancy string index system
If you look at an actual file, it means that it's going to try to read in an actual string here, terminated by a Chr(13)/Chr(10) line break.  Okay...ReadLine() is used for integrated text, ReadString() is used for string table data...got it.

The String Table chunk starts with a string count, and is then just a dump of null-terminated ASCII strings.

I also serialized out empty object tables, navigation mesh tables, and terrain info chunks.  The end result is a fairly simple, empty file.  If I load the empty file in the engine itself, it works fine.   However, if I load the empty file in the editor, the editor crashes with an access violation.

So what happened here?  Let's go back and look at that line 424 item.  The engine is throwing away this data, but the editor requires some specific stuff there.  While I'm not going to guess what the first two int values are, I think the next four groupings are data about the four viewports.

So in this case, we have editor data intermingled with runtime data.  That's a bit of a problem.  I'll have to think about the editor as well as the engine representation.

I understand the appeal of keeping a binary file for your worlds.  You can keep all of your dependencies in a single file or location.  You can have things in a ready-to-run format for your engine.  Speed of iteration (time from edit to gameplay) seems to drop dramatically.  There are major benefits.  However, there are large drawbacks.

First, maintenance. This map shows other issues with maintaining binary formats over a long time.  Look at line 443:
if (version > 22 && version < 25)
In other words, for only those versions, read in an integrated string, split it out, and apply the post f/x shaders there.  If this had been a text file, the key/value pair could have been easily ignored or converted out to a new format on save.

Second, corruption.  Let's say I've got a hefty map and as part of a save, a small portion gets corrupted.  If I have a text file, I can generally go in, find the problem area, and either figure out how to fix it or how to excise it so I can rebuild.  If a binary file gets corrupted...oh, well.

Some of you are thinking that any game developer worth his or her salt would be using source control, which leads to number three, source control.  Source control systems work best with text files.  Some, like Perforce, do a good job with handling binary diffing.  Others, like git, really dislike binary files.  Hell, even Unity has started offering a text format for its levels because of this.  It's in YAML, which is a pain to work with, but it works.

Fourth, extensibility.  There's nothing wrong with using a binary package or a bake for modules that you want to share out in order to keep all the dependencies in one place or to make it easy to handle module licensing, but when it comes to working with the core engine itself, allowing full import of scenes from text can be crucial.  When I initially asked for the map class to be released, I was working on some procedural content stuff.  I wanted to be able to create a single file that I could import that had lights, geometry, triggers, script hookups, etc.  You can import your geometry easily enough in most of these engines, provided you can export into a proper format (FBX, COLLADA, etc.), but can you import gameplay primitives just as easily?

When I get home tonight, I'm going to try to bring in the 424 chunk, and I'm going to try to get this Leadwerks file writer out as code for people, but this has really reinforced why any game dev work I do going forward is going to try to keep plain text files for my non-runtime representations of game primitives where appropriate.  I'd rather spend a little extra time baking for runtime and keep the flexibility it gives me.

Minor update, 6:13pm: After extracting the values from another map, it does look like those values are used for the editor.  After adding those values in, the editor no longer aborts out with an access violation.  It just quits silently.

March 12, 2015

MacBook Air Soundflower Issue

I've got a 2011 MacBook Air that I use for travelling.

Since I updated to Yosemite, I've been having a weird issue where it will occasionally lose all audio.

The weird part is that if I go into Settings > Audio, and go to the Output tab, I see that the sound has changed from "Internal Speakers" to "Soundflower (2ch)."  This is odd, because "Soundflower" is meant to be used to capture audio.  It's an input pathway.

Any idea how to disable "Soundflower" for output?

WSOP Los Angeles 2015 Update Post (Done)

Update 1:

I played in the 8:30pm $230 satellite against 67 other players.  Top 6 got seats, 7th got $1500.  I busted out in 11th.  Not too bad given that I haven't played poker in person since last year's WSOP in Las Vegas.

Today, I'm going to be playing in event #6, the $365 no-limit hold-em tournament.  I paid my registration fee last night, and I was the first entrant.  This tourney can go two days.  I may also play another $230 satellite tonight to see if I can avoid paying the $1,675 for the main event on Saturday.

I'll be taking Sunday off to take care of some items for work, as well as to do some game development in complete isolation.

Anyway, time for a quick shower, then a run to IHOP for a ham and cheese omelette. 

Update 2:

Took the table lead early, but was rivered three times in a row that crippled me.  I pushed with 11 big blinds left and got outkicked to be out in 121st place.

Next tournament is a $230 satellite tomorrow at 3pm.  Going to do some game development for the rest of the night.

Update 3:

Decided to try another $230 satellite again tonight.  Rivers made sure it was the wrong choice.  Back to game dev.  Will do event #7 tomorrow at noon instead of the $230 satellite and will just buy into the event on Saturday.

Update 4:

Busted out in 97th in event #7.  Went to the tables and managed to claw back $565, bought in for event #8 day 1a, and called it an early night.  I've got plenty of time for a quick shower, a leisurely breakfast and lunch, and a bit of relaxing music before my last tourney.

If I cash, great.  If not, this was still a good experience.  I busted out 18 places away from the money last year in Vegas, and this has done a good job of exposing the weaknesses in my game.  Hopefully I've learned enough to help me out today.

Update 5:

Today was flight A of the main event. Took a dominating lead on my table and kept it through dinner time, eliminating four players.

However, at level 12, things got insane. The slow bleed took almost 40% of my chips, and then two hands knocked me out.

For the first one, the short stack shoved. I had a pair of 9's, and this guy had been shoving with crap all night, so I called. Everyone else folded, and he showed AK. It was a race that he won on the river.

Next hand, two dudes I had good reads on ahead of me were raising. I could tell they didn't have shit, and when I looked down at queens, I pushed. One dude behind me called, and the dudes with nothing folded. He had AK, and won the race on the river.

It's been a fun trip down here to the WSOP Circuit. While I didn't cash in any of the events, I'm up about $565 from my cash games.

Tomorrow will be spent working remotely, and the rest of this trip will be spent working in isolation on some game dev stuff.

March 11, 2015

Heading to LA for a week

I'm going to be in Los Angeles until March 18 participating in the World Series of Poker circuit tournament at the Bicycle Casino.

Updates to come.