With that said, I needed a solid project to drive me forward. I had licenses for Unreal Engine 4, Unity 4/5, and Leadwerks 3.x, tons of assets I had been collecting and licensing over the last few years, and some larger projects in mind, but I wanted something small and contained...something where I could spend a week or two working on the project and have something done. I wanted something completable.
While I was browsing through my assets looking for a little inspiration, I found quake_map_source.zip. Basically, John Romero had released the original .map files for Quake v1.06 back in 2006. I started remembering back when Tenebrae was released, people were going in and manually placing lights to try to get the effects, and here I am with the source files. Why not see if I could get Quake maps functioning in these other engines? And thus, a project was born. I wouldn't be able to release a full project because that would virally taint the engines with the GPL, but I could release code and tools to let people recreate it themselves.
I decided to start with Leadwerks. I had backed the v3 engine on Kickstarter, and Josh had been nice enough to release the runtime code to the Map class loader at my request.
The Leadwerks map class is a binary chunked format. The first 32 bytes of the file include a magic number to identify it as a Leadwerks scene file (SCEN), a version number, and then offsets to where you will find each chunk. Dealing with a chunked file is fairly easy to do:
- Serialize each chunk to an internal buffer
- Write out an empty directory chunk up front
- For each buffer that you want to write out...
- Store the current file location
- Write out the chunk
- Once you're done writing out all the chunks, seek back to the beginning and write the offsets for each chunk into the file
The first item that needed to be done was to write out an empty map file. If I can't write out an empty file, I don't stand a chance of getting the rest done.
There are two chunks that are critical to working with a Leadwerks file: the String Table and the Map Info chunks. The String Table is where strings are supposed to be stored, and they'll be referred to by IDs within the Leadwerks file. The Map Info table has general information about the map.
Let's start with the Map Info chunk. If you have a Leadwerks license, you can download the file in the above-linked post and follow along. Please be aware that after a lot of headaches, I rolled back to map version 22 for what I was going to try to export.
Starting on line 352, it wants the number and resolution of light maps. That's easy...I don't have any. Two ints out.
Line 408 wants ambient lighting. Four 1.0f values.
Line 413 wants a map size. Inspection of the UI shows valid values of 1024, 2048, and 4096. 1024 it is.
Line 414 wants gravity. The UI shows a positive value, but the value is stored in the map file as a negative value, so -25 is serialized.
Line 424 looks like wasted space, and I just serialized out 0 and 0f depending on the field, but we'll come back to this.
Now for the comment on 438:
// This just uses the regular string commands, not the fancy string index system
If you look at an actual file, it means that it's going to try to read in an actual string here, terminated by a Chr(13)/Chr(10) line break. Okay...ReadLine() is used for integrated text, ReadString() is used for string table data...got it.
The String Table chunk starts with a string count, and is then just a dump of null-terminated ASCII strings.
I also serialized out empty object tables, navigation mesh tables, and terrain info chunks. The end result is a fairly simple, empty file. If I load the empty file in the engine itself, it works fine. However, if I load the empty file in the editor, the editor crashes with an access violation.
So what happened here? Let's go back and look at that line 424 item. The engine is throwing away this data, but the editor requires some specific stuff there. While I'm not going to guess what the first two int values are, I think the next four groupings are data about the four viewports.
So in this case, we have editor data intermingled with runtime data. That's a bit of a problem. I'll have to think about the editor as well as the engine representation.
I understand the appeal of keeping a binary file for your worlds. You can keep all of your dependencies in a single file or location. You can have things in a ready-to-run format for your engine. Speed of iteration (time from edit to gameplay) seems to drop dramatically. There are major benefits. However, there are large drawbacks.
First, maintenance. This map shows other issues with maintaining binary formats over a long time. Look at line 443:
if (version > 22 && version < 25)
In other words, for only those versions, read in an integrated string, split it out, and apply the post f/x shaders there. If this had been a text file, the key/value pair could have been easily ignored or converted out to a new format on save.
Second, corruption. Let's say I've got a hefty map and as part of a save, a small portion gets corrupted. If I have a text file, I can generally go in, find the problem area, and either figure out how to fix it or how to excise it so I can rebuild. If a binary file gets corrupted...oh, well.
Some of you are thinking that any game developer worth his or her salt would be using source control, which leads to number three, source control. Source control systems work best with text files. Some, like Perforce, do a good job with handling binary diffing. Others, like git, really dislike binary files. Hell, even Unity has started offering a text format for its levels because of this. It's in YAML, which is a pain to work with, but it works.
Fourth, extensibility. There's nothing wrong with using a binary package or a bake for modules that you want to share out in order to keep all the dependencies in one place or to make it easy to handle module licensing, but when it comes to working with the core engine itself, allowing full import of scenes from text can be crucial. When I initially asked for the map class to be released, I was working on some procedural content stuff. I wanted to be able to create a single file that I could import that had lights, geometry, triggers, script hookups, etc. You can import your geometry easily enough in most of these engines, provided you can export into a proper format (FBX, COLLADA, etc.), but can you import gameplay primitives just as easily?
When I get home tonight, I'm going to try to bring in the 424 chunk, and I'm going to try to get this Leadwerks file writer out as code for people, but this has really reinforced why any game dev work I do going forward is going to try to keep plain text files for my non-runtime representations of game primitives where appropriate. I'd rather spend a little extra time baking for runtime and keep the flexibility it gives me.
Minor update, 6:13pm: After extracting the values from another map, it does look like those values are used for the editor. After adding those values in, the editor no longer aborts out with an access violation. It just quits silently.
