Baking

Guide to bake groups

The theory:

A classic scenario, you've got your super awesome model ready to bake. You have your torus, cube, cone and plane models all ready to go!

You export your high and low res .FBX models and hit bake when, aww dang it! It's all messed up! Why is there a bunch of errors near where things touch? and why are the normals of the torus, cube and cone imprinted on the plane? I didn't want that!

The solution is to use bake groups (also known as matching by mesh name in Substance Painter). Bake groups restrict the low poly meshes to only consider their linked high poly meshes.

With this mesh naming scheme and with "Match" set to "By Mesh Name", Substance Painter will automatically arrange the four low poly meshes into bake groups, torus_low will only receive normals from torus_high, cube_low will only receive normals from cube_high and so on.

Hey that's better! the normals aren't intersecting anymore, but the ambient occlusion is leaving a shadow. That's because the normals and ambient occlusion can have different bake group settings. This is important, 90% of the time you will want to use bake groups for the normals but leave it off for the ambient occlusion, this allows you to have clean normals but the meshes can still receive shadowing from their neighbors.

Now for another experiment, let's take torus_high and rename it to plane_high2. 

Now the plane has two high poly meshes in its group, plane_high and plane_high2, so the torus (now named plane_high2) is imprinted once again. You'll also notice that torus_low has messed up ambient occlusion and normals, this is because it has no high poly meshes in its bake group to sample from anymore. Continue to count up the numbers on the end of your high poly mesh names and painter will continue to match them, you can have many multiple high poly models for every low poly one!

Practical demonstration:

This is Hal! get used to the sight, you'll be seeing him a lot through the rest of this page. You can see how I have the bake groups set up, I have maya groups (like collections in blender) for the body_high and knife_high meshes just for organisations sake, but they do not affect the bake groups inside painter. That is always done by the mesh name. 

Hal has one manifold low poly body, but in the high poly for his body, his shirt, neckscarf, pants and such are all different meshes. This is fine, we'll just name them all body_high to body_high6 and they'll be considered a part of the body_low bake group. In painter let's set the normals to match "By Mesh Name" and the ambient occlusion to match "Always" 

Awesome! the normals have avoided intersecting with their nearby meshes but we're still getting the right amount of AO shadowing. The head and the body are in different bake groups but are still shadowing each other, and the multiple body_high meshes are all casting down to the singular body_low mesh without a problem. But we can do one step better, now that we have multiple high poly meshes we can use it get us free masking later via the ID map bake. Set the Color source to "Mesh ID/Polygroup" and Color Generator to "Random" then hit bake again.

Nice, now you have a good ID from all those meshes you can use to instantly mask things. If you have stuff combined on your high poly or want to manually specify these you can also get around this by simply setting vertex colours on your high poly mesh before import and choosing Vertex color as the generator. 

Bake helpers:

That bake looked pretty good right? but let's take a closer look. 

The eyes have AO baked onto them that won't match when they start moving around and the scarf is leaving a permanent shadow on the torso that won't match when it moves on a jiggle bone. Hal's head and scarf are not manifold to his body mesh and where they intersect is making an ugly shadow as well.  

How are we gonna fix this? 

You could just paint these errors out after the bake but my favoured method is something I call bake helpers. We're going to take advantage of the ability to have multiple high poly meshes for every low poly mesh and use it to place fake occluders in the exact positions we want them. Then we turn "Self Occlusion" to "Only Same Mesh Name" for the ambient occlusion pass and let the helpers in each bake group take care of the shadowing from neighboring meshes.

A situation I use this on a lot is baking hats, if you leave them resting on the head they'll make intersection lines in the AO bake. You could simply move the low and high poly hat together up out of the way (this is known as "exploding" the mesh), but then every time you export a new high poly mesh you'll have to move it into position again. The advantage of bake helpers to me is that I set them up once and forget it, the bake will now sail through perfectly every time without any fixing. The meshes will also remain in the position they're supposed to be in for texturing. 

 So let's unhide my bake helpers group and show you what they look like. 

¡Madre mía! what is that! Let's isolate one of the bake groups and show you with and without bake helpers instead to make it a bit clearer. 

So in this case I've added an extra clone of the body_high mesh, renamed it to head_high3 and moved it down a couple of inches to allow for clearance. There'll still be some intersection at the base of the head but that bit will never be seen unlike the intersection near the edge. Let's have a look at another one. 

In this case we've duplicated head_high, renamed it to goggle_strap_high2 but we have not moved it. The downside to this technique is you'll need to duplicate every mesh that you want to share AO even if it doesn't require any special treatment. If you don't manually add it to the bake group, it will not show up. So let's get this into painter and see what it does. I like to make a separate maya group just for the bake helpers and export it as its own .fbx. 

Bake all your normals with just the fbx containing the high group loaded into your "High Definition Meshes" list, turn off baking on all the other maps, add the bake helpers then just bake only the ambient occlusion. Make sure you turn the ray distance down to as low as you can without getting artifacts. 

Now let's see the results of that!

Nice, at first glance the bake appears less realistic (and to be fair it is). But we no longer have any intersection problems on the scarf, the eyes don't have AO anymore (we did not place any additional helpers into their bake group) and the rest of the AO we wanted is still all in place. 

Don't consider bake helpers the default way of working, they're something I only resort to when I have to. If you can get away with leaving everything where they are and just letting AO bake on "always" for matching then do it!  

Candidate #1: Handplane

https://www.handplane3d.com/

Some slightly different names to other programs (Position map is called "Volumetric Gradient" here for example) but it sports a comprehensive set of bake options. The resolution goes up to 16k and it can also do non square textures. No UDIM tile support is a bummer for me but not abnormal for bakers at all. 

It's clear the focus of handplane is not an easy one button bake solution but instead on the user manually specifying the exact the result they want. Material ID maps require you to type in the names and assign them yourself. This is nice for consistency across different models but a random colour per Mesh ID option like in painter would be very handy for bakes where you don't care about that. "Ray offset" is what you'll use to specify the cage size when baking and requires hand setting per bake group without any form of visual guide.

CPU baking engine means it's not the fastest, the final 4k bake with 4x supersampling took me 28 minutes on a ryzen 5900x (this is mostly due to AO, the normals baked in 10 seconds). However, it is exceptionally well multithreaded, and CPU baking means you'll never have VRAM troubles like you can occasionally get in substance painter.

Handplane Results:

In a sense, excellent! It's totally possible to get bakes as good as anything you can do in Substance Painter.  The ambient occlusion and curvature bakes are perfect ( I actually like the curvature bake more than Toolbag's), and the normals/position/thickness and ID maps are exactly as you'd expect.

However, the amount of work, effort and knowledge it requires from the user makes it hard to recommend to beginners. There is no viewport or visual feedback for most of what you're doing, if you're not intimately familiar with the concepts of baking cages, ray distances or bake groups you may be very confused.

It also doesn't automatically split up texture sets and bake multiple at once like Painter or Toolbag, which means you'll need to do an individual bake for every set of maps/material assigned to your low poly, not a problem for Hal here as he's all one material but would get old fast.

Candidate #2: xNormal

https://xnormal.net/

Once the de facto baker for the games industry, xNormal is responsible for the bakes in thousands of shipped titles. This baker sure appears retro with it's funky windows media player appearance and unfortunately that extends to its baking performance in a few ways.

First of all there are no bake groups to speak of requiring you to "explode" the model to avoid intersections with baking, this also means there's no easy way to do ambient occlusion that takes into account all meshes, they are physically apart in your main bake and cannot influence each other. The GPU verion of the baker simply didn't work for me and the CPU version is multitudes slower than handplane's CPU engine (60 seconds for a 4k normal map at 4x supersampling, vs 10 seconds for the same settings in handplane). UDIMs and multiple material set baking are also not supported, much like handplane.

xNormal Results:

This is where things go further south for me, speed of render aside some maps from xNormal turned out to be very much useless for me in this test. Ambient occlusion was totally fine (outside of lack of neighboring occlusion as mentioned earlier) but I have never managed to get a very useful curvature bake out of xNormal no matter what settings I use. In this case as well xNormal seemed to have an issue with longer triangle shapes that handplane and all the other bakers covered with no issue. The normals are never properly overwritten and create funky artifacts. I checked and I was baking to Mikk - TSpace too,  the result was just off for some reason.

I don't really have a circumstance where I'd recommend baking in xNormal as long as handplane is there as a free alternative. Though it is worth noting that it includes a suite of utility tools that can be useful in processes (like the object space to tangent space converter or height to normal converter) , though frankly I'd just use Substance for most of these processes if you have it.

Candidate #3: Bystedt's Blender Baker addon

https://3dbystedt.gumroad.com/l/JAqLT

Hey wait a minute, didn't I say don't ever bake in blender? Well I'm wrong all the time so let's test that with the help of noted blender enthusiast and all around fantastic artist Daniel Bystedt's addon designed to streamline the process.  On paper this is one of the more interesting ones, it has full bake group support, works right in the DCC most people reading this are already probably using and has an array of interesting bake options that are more designed for converting procedural shading networks into baked versions (alongside the usual high to low mesh bakes).  

Bystedt's Blender Baker results:

Setting up the bake groups and assigning meshes from High to Low is actually quite breezy with some cool options like matching by bounding box, so let's press bake and see what the results look like!

Oh, uhh...hmmm.

I left it there for about 15 minutes to see if something eventually happened and nope, it really just didn't work. I tried again with two default cubes in a fresh scene and it got stuck in the same place. A look at the terminal reveals that it seemingly cannot select the bake groups once entering the bake scene, loses track of what it's doing and so therefore does nothing.  I'm using 3.6.2 LTS of blender and version 1.25 of Bystedt's baker (intended for the 3.6 release). My guess is at some point during 3.6's journey the python API changed and that was the end of that. 

This is almost certainly not the fault of the developer, blender addons are just finnicky like that and break frequently with updates. I suppose I could try installing 2.93 and an older version of the baker, or maybe installing an earlier build of 3.6 to see if that fixes it,  hey I could even get into the source python, see which commands no longer work and start modifying it... 

...But the easiest solution from my perspective is just to bake in handplane unless you need the ability to convert blender shading nodes to a baked texture. If the addon gets an update for the upcoming (as of writing) blender 4.0 I'll reassess it and update this article. 

Conclusion:

The basic summary is, download handplane! It gives you all the tools you need to make perfect bakes with reasonable speed and decent user friendliness. 

I do still recommend Marmoset Toolbag or Substance Painter over it, they're superior bakers in every way. But using free software is not at all an inhibitor to achieving excellent results here. 

Tricking substance painter's baker into converting height maps into curvature and ambient occlusion:

This guide is based heavily on the workflow described by Jonas Ronnegard (one of the GOATs) in this tutorial here:

Painting in height map details is one of the most useful features of Substance Painter, however it's often annoying to connect the details from height maps into generators and smart masks due to the lack of curvature and AO representation (usually requiring fiddly anchor point setups). This article will show you how to trick the painter baker into converting height map details into proper AO and curvature maps. 

For this demonstration we're going to start with a basic techno-cube with some minor greebling, but this process works equally well on organic models. Bring it into painter, bake the mesh maps in the regular way and add some height map details using alpha stamps. 

This looks pretty cool, however the problem reveals itself when we go and have a look at our bake maps, none of these details are represented in the AO or curvature. 

This means they won't be automatically included in any curvature or AO based masks and generators, such as metal edge wear or dirt. To fix this we first need to run out the tangent space normal map as a texture by itself, name it something distinct and put it in a different place to where you'll be exporting your main textures later.  Then import that map you just baked as the new "Normal" mesh map. Once you've imported that, open the baker again and bake only the "World space normal" map, this will take the tangent space normal we baked and push it to the World space normal channel. (If this doesn't work try turning "Use low poly mesh as high poly mesh" on, in this example I have no high poly mesh to begin with). 

The World space normal is what painter will use to determine the AO and curvature, so once that's all good we can simply tick it off, then tick on only curvature and AO and hit bake again.

You can see the ambient occlusion is unfortunately nowhere near as good as an actual 3d bake from geometry would be, but the curvature bake is pretty much perfect.

Soon as you have now imported the height map again in the form of the normal map it's important that you now turn off your height channels or you will get a double transform on your normals and they'll appear too strong. 

So let's now throw on some smart masks and generators to show you the difference. 

Summary:

To put the workflow succinctly:

Export normal map - > Reimport normal map to mesh map - > Bake World space normal only - > Bake curvature and AO only - > Turn off Height channels 

and if you have Substance Designer as well:

Export height map as 32 bit exr - > multiply HBAO/RTAO on top of original AO bake - > reimport as mesh map 

Happy baking!