May 14, 2015

Enabling Nashorn Autocomplete/Intellisense In Eclipse

(Update 5/14/2015, 1:00pm: Discovered that for this to work, you should be running Eclipse Luna SR2 for Java EE Developers at a minimum.)

At Netflix, we have a system called Netflix Test Studio.  Under the hood, it uses Nashorn to execute test cases that are written in JavaScript.  Several of the objects that get instantiated within the Nashorn context are native Java objects that inherit from a couple of base classes or implement a specific interface.

One of the biggest downsides to the way our system was set up was that we didn't have autocompletion for our test cases.  I decided to spend a day to try to figure out what we needed to do, and here are the results of my investigation.

Please note that I won't be sharing my exact code here.  The code I wrote relies on some inner assumptions about our code base.  I'm here to talk you through what I did so that you can replicate the behavior in your own code base.  Also, I'm using Eclipse Luna SR2, so instructions may differ from version to version.  Finally, this is only a day's worth of work.  A final version will be significantly more robust than what I describe here.

The first step is enabling the JavaScript facet in your Java project.  To do this, right-click on your project in Project Explorer, go to Project Facets, select "JavaScript" and hit "Apply" and "OK."  This will also give you syntax error checking for JavaScript files inside your Java application, which is always a nice thing.

Second, you need to create a JsDoc file.  JsDoc is the JavaScript equivalent of JavaDoc.  What I did was I added a new main class to my project that I could run that would use the Reflections library to find all the classes in my project that inherited from our test step base class and generate a JsDoc stub like this:

/** @external com.netflix.nts.steps.tocap.NrdpCompareFriendlyName

Constructors:
NrdpCompareFriendlyName()

@class NrdpCompareFriendlyName
 */
function NrdpCompareFriendlyName() {}


/** Add com.netflix.nts.steps.tocap.NrdpCompareFriendlyName to JavaImporter */ NrdpCompareFriendlyName.prototype._ = function() {}
/** getTargetId()
*/
NrdpCompareFriendlyName.prototype.getTargetId = function() {};
/** promptUserForResponse(UserPrompt arg0)
*/
NrdpCompareFriendlyName.prototype.promptUserForResponse = function() {};
/** wait(long arg0, int arg1)
wait(long arg0)
wait()
*/

NrdpCompareFriendlyName.prototype.wait = function() {};
...

Let me walk you through each of these items.

NrdpCompareFriendlyName is a Java class.  /** */ is how you delimit a JsDoc comment block in JavaScript, so I put some metadata there.  Eventually, I'll be pulling in some data from our JavaDoc, but for now, I really only needed to add the @class attribute to help Eclipse know that this is a valid item to show up after a "new" statement.  I then declare an empty function with the same name.

Now, there's a bit of a bug in Eclipse Luna SR2 where if you hover over a class name, rather than pull up the JsDoc information that came immediately before the function definition, it will pull up the JsDoc for the method immediately following it.  So, I emit a dummy underscore so I can have the class name that needs to be added to the JavaImporter class immediately after.  This way, I can hover over the class name, then copy the full class name and easily paste it into my JavaImporter statement.

To associate the method with the class, I add it to the class prototype.

For each method in the class, I then emit a simple JsDoc block with all the overloads of the methods and their expected parameter types.

Finally, we add the JsDoc to the project references.  Right-click on JavaScript Resources in your project and pick "Properties" and add your JsDoc file there.  You should then get this data to show up as part of your Eclipse autocomplete/content assist.

A couple of additional recommendations before I leave.  In Preferences > JavaScript > Editor > Content Assist, check "Enable Auto Activation."  Also, if your JsDoc file is massive (ours is 4.6MB right now), you might need to increase the timeout under Preferences > JavaScript > Editors > Content Assist > Advanced.

Have fun, and let me know what kind of enhancements you guys can make with this.

April 5, 2015

Handling Quake 1 .map Files (Part 1)

Now, let's say you want to handle Quake 1 .map files.  Perhaps you want to do the project I'm doing: porting Quake to other engines.  Perhaps you want to use Quake 1-compatible level editors for your MonoGame or three.js projects.  Who knows?  Code for this series will be provided in C# for easy integration into tools and content pipelines.

First, let's look at what an entry in a .map file looks like.

{
 {
  ( x1 y1 z1 ) ( x2 y2 z2 ) ( x3 y3 z3 ) TEXTURENAME xofs yofs rotation xscale yscale
  ...
 }
 "key" "value" // Optional comment
}

Each entry in a .map file is enclosed in curly braces to encompass an entity.

Each entity can contain zero to many brushes.  Each brush is enclosed in curly braces.

Each brush must be convex can contain four to many faces.  (Why four?  Think of a three-sided pyramid and the base.)

Each face includes three vertices to define the plane, shown above as (x1,y1,z1) through (x3,y3,z3), a texture name, and all the information needed to calculate the texture coordinates for each final vertex.  Vertices are in clockwise order.  Most of the time, these vertices are the first three vertices of that face, but not always.  They're just there to define the plane.

Each entity can also contain one to many key/value pairs.  Each entity will have one key named "classname" which tells you what other key/value pairs you should expect.

If you find a classname of "worldspawn" in the entity, this will be all of your static geometry.  You can use this information to heavily optimize this information for rendering or other work: do texture batching, visibility calculations, collision calculations, etc.

While you can have comments in your .map files, you probably won't see them.

If you think about it, this gives you a good way to handle save game separation as well.  When you are saving your game, iterate through your world and serialize out the information for any entity that isn't worldspawn.  When you are loading the saved game, you can load only the worldspawn entity, then overlay your other entity information from your save game.

Now that we have that information, we know what we need to parse the .map file.  Here's a functioning map parser.  It fills a Map object with Entities that can contain key/value pairs and Brush objects containing Faces.

http://www.romsteady.net/quake1/Quake1Map-ParserOnly.zip

Next time, we'll turn this information to a polygon soup.

March 30, 2015

The Failures Of Binary Formats In Development

One thing I've wanted to do for the last few weeks is dive back into game development.  While I was back at Amazon, I could do game dev work that I had grandfathered in, but I couldn't technically do any new game dev work.  Fortunately, Netflix lets me do pretty much whatever I want outside the office, provided it isn't creating a new video streaming service.

With that said, I needed a solid project to drive me forward.  I had licenses for Unreal Engine 4, Unity 4/5, and Leadwerks 3.x, tons of assets I had been collecting and licensing over the last few years, and some larger projects in mind, but I wanted something small and contained...something where I could spend a week or two working on the project and have something done.  I wanted something completable.

While I was browsing through my assets looking for a little inspiration, I found quake_map_source.zip.  Basically, John Romero had released the original .map files for Quake v1.06 back in 2006.  I started remembering back when Tenebrae was released, people were going in and manually placing lights to try to get the effects, and here I am with the source files.  Why not see if I could get Quake maps functioning in these other engines?  And thus, a project was born.  I wouldn't be able to release a full project because that would virally taint the engines with the GPL, but I could release code and tools to let people recreate it themselves.

I decided to start with Leadwerks.  I had backed the v3 engine on Kickstarter, and Josh had been nice enough to release the runtime code to the Map class loader at my request.

The Leadwerks map class is a binary chunked format.  The first 32 bytes of the file include a magic number to identify it as a Leadwerks scene file (SCEN), a version number, and then offsets to where you will find each chunk.  Dealing with a chunked file is fairly easy to do:

  1. Serialize each chunk to an internal buffer
  2. Write out an empty directory chunk up front
  3. For each buffer that you want to write out...
    1. Store the current file location
    2. Write out the chunk
  4. Once you're done writing out all the chunks, seek back to the beginning and write the offsets for each chunk into the file

The first item that needed to be done was to write out an empty map file.  If I can't write out an empty file, I don't stand a chance of getting the rest done.

There are two chunks that are critical to working with a Leadwerks file: the String Table and the Map Info chunks.  The String Table is where strings are supposed to be stored, and they'll be referred to by IDs within the Leadwerks file.  The Map Info table has general information about the map.

Let's start with the Map Info chunk.  If you have a Leadwerks license, you can download the file in the above-linked post and follow along.  Please be aware that after a lot of headaches, I rolled back to map version 22 for what I was going to try to export.

Starting on line 352, it wants the number and resolution of light maps.  That's easy...I don't have any.  Two ints out.

Line 408 wants ambient lighting.  Four 1.0f values.

Line 413 wants a map size.  Inspection of the UI shows valid values of 1024, 2048, and 4096.  1024 it is.

Line 414 wants gravity.  The UI shows a positive value, but the value is stored in the map file as a negative value, so -25 is serialized.

Line 424 looks like wasted space, and I just serialized out 0 and 0f depending on the field, but we'll come back to this.

Now for the comment on 438:
// This just uses the regular string commands, not the fancy string index system
If you look at an actual file, it means that it's going to try to read in an actual string here, terminated by a Chr(13)/Chr(10) line break.  Okay...ReadLine() is used for integrated text, ReadString() is used for string table data...got it.

The String Table chunk starts with a string count, and is then just a dump of null-terminated ASCII strings.

I also serialized out empty object tables, navigation mesh tables, and terrain info chunks.  The end result is a fairly simple, empty file.  If I load the empty file in the engine itself, it works fine.   However, if I load the empty file in the editor, the editor crashes with an access violation.

So what happened here?  Let's go back and look at that line 424 item.  The engine is throwing away this data, but the editor requires some specific stuff there.  While I'm not going to guess what the first two int values are, I think the next four groupings are data about the four viewports.

So in this case, we have editor data intermingled with runtime data.  That's a bit of a problem.  I'll have to think about the editor as well as the engine representation.

I understand the appeal of keeping a binary file for your worlds.  You can keep all of your dependencies in a single file or location.  You can have things in a ready-to-run format for your engine.  Speed of iteration (time from edit to gameplay) seems to drop dramatically.  There are major benefits.  However, there are large drawbacks.

First, maintenance. This map shows other issues with maintaining binary formats over a long time.  Look at line 443:
if (version > 22 && version < 25)
In other words, for only those versions, read in an integrated string, split it out, and apply the post f/x shaders there.  If this had been a text file, the key/value pair could have been easily ignored or converted out to a new format on save.

Second, corruption.  Let's say I've got a hefty map and as part of a save, a small portion gets corrupted.  If I have a text file, I can generally go in, find the problem area, and either figure out how to fix it or how to excise it so I can rebuild.  If a binary file gets corrupted...oh, well.

Some of you are thinking that any game developer worth his or her salt would be using source control, which leads to number three, source control.  Source control systems work best with text files.  Some, like Perforce, do a good job with handling binary diffing.  Others, like git, really dislike binary files.  Hell, even Unity has started offering a text format for its levels because of this.  It's in YAML, which is a pain to work with, but it works.

Fourth, extensibility.  There's nothing wrong with using a binary package or a bake for modules that you want to share out in order to keep all the dependencies in one place or to make it easy to handle module licensing, but when it comes to working with the core engine itself, allowing full import of scenes from text can be crucial.  When I initially asked for the map class to be released, I was working on some procedural content stuff.  I wanted to be able to create a single file that I could import that had lights, geometry, triggers, script hookups, etc.  You can import your geometry easily enough in most of these engines, provided you can export into a proper format (FBX, COLLADA, etc.), but can you import gameplay primitives just as easily?

When I get home tonight, I'm going to try to bring in the 424 chunk, and I'm going to try to get this Leadwerks file writer out as code for people, but this has really reinforced why any game dev work I do going forward is going to try to keep plain text files for my non-runtime representations of game primitives where appropriate.  I'd rather spend a little extra time baking for runtime and keep the flexibility it gives me.

Minor update, 6:13pm: After extracting the values from another map, it does look like those values are used for the editor.  After adding those values in, the editor no longer aborts out with an access violation.  It just quits silently.

March 12, 2015

MacBook Air Soundflower Issue

I've got a 2011 MacBook Air that I use for travelling.

Since I updated to Yosemite, I've been having a weird issue where it will occasionally lose all audio.

The weird part is that if I go into Settings > Audio, and go to the Output tab, I see that the sound has changed from "Internal Speakers" to "Soundflower (2ch)."  This is odd, because "Soundflower" is meant to be used to capture audio.  It's an input pathway.

Any idea how to disable "Soundflower" for output?

WSOP Los Angeles 2015 Update Post (Done)

Update 1:

I played in the 8:30pm $230 satellite against 67 other players.  Top 6 got seats, 7th got $1500.  I busted out in 11th.  Not too bad given that I haven't played poker in person since last year's WSOP in Las Vegas.

Today, I'm going to be playing in event #6, the $365 no-limit hold-em tournament.  I paid my registration fee last night, and I was the first entrant.  This tourney can go two days.  I may also play another $230 satellite tonight to see if I can avoid paying the $1,675 for the main event on Saturday.

I'll be taking Sunday off to take care of some items for work, as well as to do some game development in complete isolation.

Anyway, time for a quick shower, then a run to IHOP for a ham and cheese omelette. 

Update 2:

Took the table lead early, but was rivered three times in a row that crippled me.  I pushed with 11 big blinds left and got outkicked to be out in 121st place.

Next tournament is a $230 satellite tomorrow at 3pm.  Going to do some game development for the rest of the night.

Update 3:

Decided to try another $230 satellite again tonight.  Rivers made sure it was the wrong choice.  Back to game dev.  Will do event #7 tomorrow at noon instead of the $230 satellite and will just buy into the event on Saturday.

Update 4:

Busted out in 97th in event #7.  Went to the tables and managed to claw back $565, bought in for event #8 day 1a, and called it an early night.  I've got plenty of time for a quick shower, a leisurely breakfast and lunch, and a bit of relaxing music before my last tourney.

If I cash, great.  If not, this was still a good experience.  I busted out 18 places away from the money last year in Vegas, and this has done a good job of exposing the weaknesses in my game.  Hopefully I've learned enough to help me out today.

Update 5:

Today was flight A of the main event. Took a dominating lead on my table and kept it through dinner time, eliminating four players.

However, at level 12, things got insane. The slow bleed took almost 40% of my chips, and then two hands knocked me out.

For the first one, the short stack shoved. I had a pair of 9's, and this guy had been shoving with crap all night, so I called. Everyone else folded, and he showed AK. It was a race that he won on the river.

Next hand, two dudes I had good reads on ahead of me were raising. I could tell they didn't have shit, and when I looked down at queens, I pushed. One dude behind me called, and the dudes with nothing folded. He had AK, and won the race on the river.

It's been a fun trip down here to the WSOP Circuit. While I didn't cash in any of the events, I'm up about $565 from my cash games.

Tomorrow will be spent working remotely, and the rest of this trip will be spent working in isolation on some game dev stuff.