Console Compatibility in ROM Manager
This page is intended as a resource to create console compatible romhacks (or port your own existing ones) using SM64 ROM Manager.
I am still researching the topic, so I may add more info as I find out about it.
Thanks to Hyena-chan for knowledge on console related issues, Falcobuster for creating Bowser's Blueprints and several patches related to console compatibility that I used to make this, and HeralayanSalty for helping me interpret C++ code.
The Magic Console Patch
I have created a patch that will make your romhack work on console, and also reduces console lag.
Mainly, the things it does are:
Reverts RCVI hack patch (which outright breaks console)
Fixes an issue where the soundbank would attempt to get loaded over framebuffer in RAM, which I was surprised is still a thing.
Properly reverts ROM Manager's laggy extended boundary patch, as the "official" way to do it is to patch a ppf included with versions 1.9.11 and older that breaks a function and crashes the game after file select.
Applies the extended boundary patch used for decomp and Bowser's Blueprints, which is a lot more console friendly.
Adds the "World scale" setting from Bowser's Blueprints, which helps to get rid of graphics overflow (Aka Z fighting / flickering issues that happen when a level is too big). This setting will be set to 3.5x.
Unfortunately, there are a few inconveniences that this patch causes, namely:
King Whomp won't deal damage when squishing Mario.
Death floors added with the ROM Manager setting won't work in-game. You'll need to manually add them to your model.
If your hack had Frame patch, walls will not exist anymore. There is currently no fix for this.
Due to the world scale patch, all the values in the "Camera frustrum" setting must be divided by 3.5 for graphics to work correctly.
This results in 29 for "near" and 8571 for "far" as the default, and 10721 if you're using the SM64 Editor value for increased draw distance (0x9292, or -28014 in-game). Feel free to calculate your own if you want different values.
You can download the patch here:
Graphics Overflow
The world scale setting is set to 3.5x by default. To change this setting, open the asm patch, find the line
"LUI AT, 0x4060", and change 0x4060 to the upper 2 bytes of the floating point you want to set world scale to, expressed in hex.
After Applying the Patch(es)
If you had used the "Slope Fix (both versions)" tweak, it won't work anymore. To fix this, apply it again, but make sure to pick the "No extended boundaries" script.
Apply the "Remove Some Console Anti-aliasing" tweak. From my testing, it doesn't seem to cause any differences on emu, as graphics plugins have their own anti aliasing, but it will improve performance on console.
Triangles that are too big may cause major graphical issues. If that happens, you can try subdividing them into multiple smaller triangles.
Remember that, while you shouldn't have more triangles than needed, triangles that overlap eachother on screen is a bigger issue.
Try not to have unnecessary geometry under structures, delete the back sides of objects at the edges of your level, and don't do the funny effect where you copypaste a bunch of transparent faces to make a gradient.
And most importantly, test your hack on Parallel plugin from time to time (or Angrylion if your PC can't run Parallel).
But also remember that lag on Parallel/Angrylion does not correlate with lag on console. Either of them may lag where the other doesn't.
What? That's it?
Yup.
Despite the common misconception that having your romhacks run at all on console is a super arduous process, I was able to make most of mine work by changing literally 4 bytes.
I can't count on my two hands the amount of times I've been told about a supposed extremely low triangle limit that is strictly required for romhacks to work, but there really is no such thing as that -- at least not beyond what ROM Manager itself can handle.
Happy console gaming!