RTResolution will be released to the general public on Friday after being released to patrons early today. This post will collect the troubleshooting information from early adopters.
The game crashes on launch.
The RTResolution patcher now drops two files in your save game folder: Terraria.exe and RTRHooks.dll. Make sure you copy both files over to your save game folder.
Update 6/10 7:24pm: Fixed in RTResolution 0.4.2.
The game won't go over 4096 in any one direction.
There are memory constraints within the Terraria process when you start going larger than this. If you want to risk it, try the 8K experimental build. I'm aware of at least one crash at higher resolutions due to an issue inside of Terraria with ultra-high resolutions, though.
Update 6/10 7:24pm: Fixed in RTResolution 0.4.1.