November 23, 2016

The 1.3.4.3 Method Rewrite Post

While I was taking my Uber back from the Postmodern Jukebox concert, I figured out how I was going to "fix" Terraria and allow myself to release an update for RomTerraria at the same time: I'm going to rewrite the method that is introducing the issue.

I've patched bugs in the base game before. For the longest time, if you didn't have a My Games/Terraria folder in your Documents folder, the game would crash when trying to create the settings file.  I patched that with the original releases of RomTerraria.

Fortunately, I've got a good place to handle injecting my replacement method.  LiquidRenderer.PrepareDraw just calls the internal method.

I'm going to do this patch in two phases.

Phase one will be out tomorrow on Thanksgiving assuming nothing else blows up and will simply make PrepareDraw be a no-op.  This will allow everything else to work as is EXCEPT for the new water effects.

Phase two will take significantly longer.  I'm going to rewrite LiquidRenderer.InternalPrepareDraw.  It's a bit over 400 lines of, well, not good code.  I mean, it's dereferencing to memory before a pointer.  If I was going to be a total dick, I could craft a world file that could potentially turn into a native code exploit.  Fortunately, I'm not a total dick.  This will take me a lot longer to do and may end up moot if they fix the issue before my rewrite is done.

I've only got this weekend to do this, so here goes nothing.

Update 11/24/2016, 11:57am: Well, it's not going to happen this morning, but I am working on it...


Update 11/24/2016, 12:20pm: I've gotten the replacement class building as a standalone DLL.  I've got to do IL-patches to replace 14 instances of the class in six methods in two classes.  I'm going to try to get that done before I have to leave for my afternoon social obligation.

Update 11/24/2016, 4:38pm: I'm back from my social obligation and am resuming my coding.  I've almost got the replacement working.

Update 11/24/2016, 9:57pm: Stopping for the night.  I've got my replacement class patched in and am working on debugging, but I'm exhausted.  That said, making PrepareDraw a no-op has other side effects and isn't a good solution, so I'm dropping that part of the plan.

Update 11/25/2016, 12:18pm: Taking a lunch break.  Since I'm afraid I'm going to have to do this kind of thing again in the future, I've created a general purpose class replacement system. Will resume testing once I am fed.

Update 11/25/2016, 1:03pm: I'm now able to reproduce the crash inside my custom class inside the debugger.  Should help me fix it.

Update 11/25/2016, 1:09pm: Well, it's good that I made the class replacement system generic, because the crash is NOT caused by the code I suspected.  It appears that something else is running amok around memory.  Give me a bit of time to figure out the appropriate memory layout so I can figure out which object is immediately before this one in the heap.

Update 11/25/2016, 1:30pm: I've got a 1.2GB memory dump that I'm manually going through.  I'd be going faster, but the .NET memory tools require a minimum .NET version of 4.5 and Terraria is compiled to 4.0.

Update 11/25/2016, 2:25pm: Good news: managed to stop the game from crashing. Bad news: filled blocks of water aren't rendering.  Still going.

Update 11/25/2016, 3:53pm: I've checked my current code into Github.  Still trying to figure out why full blocks don't render.  Note that this doesn't truly fix the memory overwrite, but it leaves the old 160KB object around as a "safety buffer."  I'm going to try for another hour, then call it a day.  I've still got work to do around the house.

Update 11/25/2016, 5:51pm: Part of the problem is that the version of Telerik JustDecompile that I'm using doesn't properly decompile unsafe code evidently.  Testing a fix now.

Update 11/25/2016 6:00pm: Adding padding around the wave bank let me run with water effects for almost ten seconds before the game crashed in a spectacular fashion.  Looks like something is hardcoded with a max resolution and when you go above that resolution, it starts walking over memory it shouldn't.  Dinner shortly, then more research

Update 11/25/2016 6:09pm: Here's a test version.  This is not guaranteed to work, but appears to be working in my limited testing.  Please let me know if it crashes for you.

Update

I'm currently at a concert for Postmodern Jukebox and won't be home until almost midnight.

I've got tomorrow morning blocked out for rewriting the defective method in Terraria that's doing a negative address pointer dereference and swapping it out.  I've patched the base game before.  I'll do it again...

November 16, 2016

1.3.4 Support Coming

Sounds like Terraria v1.3.4.x broke RomTerraria again.

If I had to make a guess, I'd say that the new render pathway for water changed the number of RenderTarget2D's in the code.  If that is the case, it's probably a 20-30 minute fix.

That said, I haven't had a chance to look at it because I've been working, and I won't be getting home from work until late tonight.

I'll take a look at it after dinner tonight.

Update 11/16/2016, 8:23pm: I've got the resolution patch working, but the new water effects are causing the following crash that I need to fix first...

...removed for space...

Update 11/22/2016, 11:10am: Patch appears to be working, but I have to finish my test pass before I release it.  Hopefully tonight after dinner.

Update 11/22/2016, 6:41pm: I found a crash bug with 1.3.4.3. Working on patching it now.

Update 11/22/2016, 8:24pm: Okay, I've got a choice.  I can either release a version that will only work with all the visual bells and whistles turned off, or I can hold back and release it once I figure out what is causing the wave mask to get corrupted and crash approximately ten frames into rendering.  Right now, I'm planning on holding back until I get this tracked down.

I know that isn't what people want to hear, but I'd like you to remember two things.  First, the reason that RomTerraria works as well as it does across multiple versions of the game is that I don't do source decompiling/recompiling.  I do MSIL-level opcode patching.  This requires a lot more time to do right because if I do it wrong, I create invalid code and that breaks everything else.

Second, the reason that you never see anything about this mod over on anything official is because back when Terraria launched, some asshats were using my mod as a crack for Terraria and they bundled my mod with the cracked version.  Because of this, Re-Logic appears to see this mod as something bad.  As someone who spent a decade in the games industry fighting pirates, I can't blame them.  I'm absolutely certain that they see me as someone who has enabled bad behavior and they don't want to enable me to succeed in this venture.

I've been maintaining this mod as a spare-time project for over five years now, and it wears on me.  I don't make any money off of it, partly because I don't want to further enrage Re-Logic, and partly because it doesn't seem right to me to make money off of mods.  However, that means that this mod is secondary to other projects at both work and home.  Even tonight, I've had to spend time helping out a lady friend.

I spend the time that I can afford to on this, both from a financial point of view, but also a mental point of view.  Working on this mod isn't as simple as hitting F5 in Visual Studio and seeing what happens.  It requires a lot of time to trace through each failure.  It requires a lot of effort to inject diagnostics code into the assemblies.  It requires a lot of patience to track down crashes, support comments, etc.  I have a finite amount of willpower to get through each day and this mod gets the last few minutes of the day usually.

I'll keep working until I head to bed tonight, but I won't be able to work on it again until Thursday night or Friday morning at the earliest due to the holiday and social obligations.  I'm off Friday, however, so hopefully I can refocus my energies for a bit and get this fixed.

Update 11/22/2016, 8:50pm: Around 40 frames in, something is corrupting Terraria.GameContent.Liquid.LiquidRenderer._waveMask and changing it from an array of 40000 XNA Color structs to an array of 39936 XNA Color structs.  This is going to take a lot longer to figure out since Terraria uses unsafe code to work with that array.

Update 11/22/2016, 9:05pm: Well, the good news is that I figured out the root cause.  There's a bad bug in LiquidRenderer.InternalPrepareDraw when it's dealing with a fixed pointer to _waveMask.  This code has a lot of issues.  It's not cache coherent (looping over Y inside of a loop over X), and the root cause is a chunk of code that starts writing before the variable if you are on a screen with certain conditions set:

*(vISCOSITYMASK - 200 * sizeof(Color)) = *vISCOSITYMASK;

They're trying to set the pixel above a point to the value of that point.  That line should scare you.  Scares the hell out of me.  Until I can trace through what the hell led to this, or Re-Logic writes a cache coherent version of this method that doesn't write to memory before vISCOSITYMASK pointer, there will not be an update to RomTerraria.

Update 11/26/2016, 12:57am: A test version that seems to work with 1.3.4.3 is available over at the update post.

October 26, 2016

Electric Eye Open Source Release Update

I've been asked a few dozen times over the last month about where the open source release of Electric Eye is.  Here's a quick update.

I was in Japan and South Korea from October 1 through October 13.  Another individual working on Electric Eye was in China October 14 through October 21.  We were on site at our partners ensuring that the multicore rewrite that we did over the summer worked the way that it should.  A couple of minor bugs surfaced on site, including an interesting one where a S/PDIF renegotiation caused our audio interfaces to no longer receive audio.  We're both currently fixing the final spat of issues that surfaced from this trip.

On November 3, we're releasing the code to one partner here in the Bay Area.  This is a bit of a trial run to ensure that there's nothing egregiously off about our code.

Once we've gotten final feedback from our local partner about the code, we can resume working with legal to get the code released.

Sorry about the delay.  On the upside, we're going to be supporting several more webcam models as part of the release.  If you're trying to prepare for the release, I can provide you with system requirements...
  • Windows
    • Windows 7 SP1 64-bit or above with all Windows Updates installed
    • Intel i5 2.6 GHz processor or better, or the AMD equivalent
    • 8GB of RAM or more
    • OpenGL 3.2 compatible video card
    • Visual Studio 2015 Update 3
  • Mac
    • OS X Yosemite or El Capitan (macOS Sierra not yet supported)
    • mid-2014 MacBook Pro or better
    • Xcode 6.4
  • One of the following cameras (affiliate links)
I'm currently working on adding support for the c922x and c920 but cannot guarantee support will be in the open source release.

The Linux version isn't ready to go yet, and there are two chunks of code that we won't be able to release that won't impact general usage.  However, those should not stop the release.

Will share more once I know more.

October 2, 2016

In Japan

I'm currently sitting in my hotel room at the InterContinental Osaka getting ready for the next two weeks of work.  I'll be back in the United States after October 13.

During the next few weeks, I'll be unable to do any work on RomTerraria.  Even if I could do some work on it, I'm sitting here with a MacBook Air and wouldn't be able to test the high resolution features.